The CAPP Oil Sands Survey - Is it Informative or Misleading?

Yesterday the Canadian Association of Petroleum Producers (CAPP) released two opinion polls on the Alberta oil sands and said the results gave “…the oil sands industry a candid perspective on the views of Canadians on the development of the oil sands.”


The CAPP new release quotes Imperial Oil CEO Bruce March saying “Canadians are telling us that we need to do better. We have received a clear message: the economic and energy security benefits of the oil sands cannot come at the expense of the environment.” Here! Here!

HAS THE OIL INDUSTRY HAD A CHANGE OF HEART?
I applaud the effort of CAPP to better understand what sustainable and responsible oil sands development means. The hearts and minds of the project developers may be coming around and the attitudes may be changing from the arrogant and threatening approaches they assumed in response to the Royalty Review Panel proposals. They even when over the line so far as to sponsor an Astroturf website on the royalty issues back in the day.

There is good work being done by some enlightened oil sands developers that on mitigation and some even on prevention of environmental impacts but it is late coming and sporadic, to say the best.

PUBLIC OPINION POLLING IS AS MUCH ART AS IT IS SCIENCE
I am not a pollster or a statistician but I am a student of public opinion and public policy. But boy-oh-boy the process, content and presentation of the CAPP surveys shows that they still have lots to learn about being clear, transparent and accountable when they do public opinion research reporting.

We sponsor a lot of research for clients of value drivers on public policy issues and we know how much of an art it is. Read the great piece in the Globe and Mail over the unreliability and huge “margin of error” in the monthly Stats Can unemployment report. Heather Scoffield’s piece entitled “Extremely Influential, Notoriously Unreliable” says it all but read the column for an in depth review of this reality.

IS THIS JUST A "PR" EXERCISE?
Is this poll and its release just a communications exercise? The language of the news release indicates it is more than just PR but the process, results and the lack of rigour (to put it politely) in the presentation makes CAPP's intentions suspect.


Here are some of the difficulties I have. The methodology shows that we do not actually have a survey of “Canadians” as the news release touts. We have a barely adequate same of 425 Edmontonians in one instance and 429 Torontonians in the second survey. While Torontonians may presume they speak for Canadians. As an Edmontonian, I can assure you we share no such presumptions. CAPP knows better.

The small sample size means the margin of error is very large, 4.8% and there were two different time frames for each survey. Edmonton was surveyed in the first half of June and Toronto was done later in the month and into July. This difference in timing could make them two very different and non-comparable survey results.

What if 500 ducks drowned in oil sands tailing ponds mid June and not in April? That would impact and change the opinions of one survey to the next. Obama was campaigning on dirty oil and Dion was into his Green Shift in June of 2008 and there was lots of media happening. Why didn't they survey both cities at the same time? And why did they only do two cities and not a national survey? beats me.


PRESENTATION OF RESULTS LEAVES MUCH TO BE DESIRED
The presentation of the pie charts doesn’t show the data used to calculate the charts. They also say the results are the “mean of the first and second represented.” Why did they do that and why use means? Providing the actual numbers would be so helpful to reassure that this presentation is accurate and representative.

The Public Policy Priorities question is framed around the “next federal election” and the “next provincial election.” Why would they tie the question to a federal election that is not contemplated or even wanted and provincial elections are a long way off in Alberta and Ontario?


Why not just ask what are your public policy priorities and not politicize the question if you are look for authentic data? Tying it to and hypothetical election means the dominant answer is going to likely be “Other/No Opinion.” That skews the results and its usefulness. Take out the “Other /No Opinion and the top three are consistent as Economy, Healthcare and Environment. Note Climate Change is separate from Environment and #4.

The findings on the “…greatest environmental concerns about the oil sands activity in Alberta” are essentially the same as we found in our Discrete Choice Modeling survey done in November 2006. The top 2 value drivers in our survey were Habitat Protection and Carbon Emissions. They were followed closely by water use and reclamation concerns. Only about 25% has no concerns or no opinion. CAPP finds does not even ask about tailing ponds and reclamation in its survey which was done 2 months after the drowning of 500 migrating ducks in a tailing pond. Why not, given the timing? It was news all around the world.

The Bar Graphs in the CAPP presentation uses a typical technique that can mislead and even go so far as to misrepresent the data. The “Y” (vertical) axis usually is presented as a 100% scale. That way there is a sense of relative opinions between alternative answers and the overall impact of the results. CAPP never uses 100% in its “Y” axis presentation and that gives a skewed appearance to the data.

They top out their graph presentations at 60%, 50%, 45% and40%. On the very last question, the biggie about if people think it is possible to balance economic benefits and protect the environment they use 70%top scale. They had to because the results showed 60% of Edmontonians and 50%+ (we are not sure of the exact number based on the presentation) agree this is possible.

Not using 100% on the “Y” axis can be seen as a “slick” presentation technique. Not doing the “Y” axis consistently in the presentation is even worse. This does nothing to help provide clarity and consistence and meaningful representation of the data. Again the lack of the actual numbers used to calculate the graphs is an omission that is irritating at least.

I know of some of the great science-based environmental work some individual oil sand developers are doing. I want to give CAPP and the oil sands industry generally the benefit of the doubt but they don’t make it easy.


I will deal with the implications of the findings of the survey in more detail in a later post.