Climate Audit

I have a lengthy op ed in today’s National Post go here summarizing some of the debate since publication of our 2005 articles. The article is on page FP19. Update: The link is now offline. Here is the text as I submitted it to National Post; it may differ a little, but not much.

The much-vaunted “hockey stick diagram” became famous a few years ago when the Intergovernmental Panel on Climate Change (IPCC) used it to argue that the “1990s were the warmest decade in the millennium and 1998 the warmest year”. These sound bites were used in speeches advocating Kyoto; the Government of Canada promoted the “hockey stick” interpretation of temperature history on its web site, sent it to schools across the country and quoted its conclusion in pamphlets mailed out to all Canadians.

The “hockey stick” theory overturned the view in the first IPCC report that there had been at least equivalent warmth in the “Medieval Warm Period”, when, for example, Vikings had settled in Greenland. Recently, it has been reported that one of the leading scientists in the field wanted to “get rid of the Medieval Warm Period”. The hockey stick was how they did it. Quite apart from the MWP, climate during most of geological history was warmer than at present.

In two peer-reviewed articles published this past winter, Ross McKitrick and I showed that there had been no due diligence on the hockey stick calculations by the IPCC and that there were serious problems in the calculations. Our main article was published in the same scientific journal that published the hockey stick graph used by the IPCC

What has been the public reaction? The story has been reported around the world. Coverage began in the National Post and the Dutch science magazine Natuurwetenschap & Techniek. Since then articles have appeared in, among others, Nature, Science, The Economist, and a front page feature (Feb 14) in The Wall Street Journal. The story has been reported on the BBC and Global, as well as German and Dutch television. My website http://www.climateaudit.org has received over 250,000 hits since mid-February.

What has been the reaction from climate scientists? It varies.

Richard Muller of Berkeley likened our contribution to removing a piece of a jigsaw puzzle that was in the wrong place so that investigation about climate history can resume with a clean slate. Hans von Storch, a famous German climate scientist, said that Mann’s error was material, that it was “good that debate about the temperature history of the last millenium can be resumed again without reservations”, and that we are entitled to “thanks” for this contribution. On the other hand, Andrew Weaver of the University of Victoria, a prominent Canadian climate scientist, said that our original paper should have been “rejected” and that he believed that giving equal space to both sides in a dispute can be dangerous, particularly when applied to scientific matters.

One claim has been pretty well universally accepted: we showed that an unreported step in the Mann calculations mines datasets for hockey-stick shaped series. We showed that we could produce hockey sticks even from random data. This observation has been verified by others and we would say that this claim is not argued by anyone other than perhaps the original authors.

What of our other main claims?

We showed that this obviously unsatisfactory statistical method had really bad results on actual data as well: we showed that the bad method mined for bad data creating a “perfect storm”. It turned out that their results depended on the inclusion of a controversial set of US bristlecone pine tree ring series, which had a hockey stick shape. However, the specialists who studied bristlecones had explicitly stated that the hockey stick shape was not due to temperature, but to fertilization. We showed that the original authors had known their results fell apart with the removal of these series and that they had not only failed to disclose these adverse results, but claimed the opposite in a later commentary on their own work.

We also showed that the original authors withheld vital data (certain verification statistics), which showed their conclusions were statistically insignificant, and that the procedure for benchmarking the one verification statistic that they did report was incorrect.

To date, none of these claims has been challenged. This is not to say that these claims have been accepted or that our work has not been challenged. There has been a concerted effort by climate scientists to show that the errors in the hockey stick calculations “do not matter”. In fact, there has been much more effort by climate scientists to try and disprove our results than ever went into checking the original hockey stick. We made the process easy by publishing all our computer code, unlike the hockey stick authors who still refuse to release theirs even seven years after the original publication. They told the Wall Street Journal that to show the code they used to produce their results would be “giving in to intimidation.”

We know of 5 submissions thus far to academic journals commenting on our most recent results (in addition to 2 submissions last year on our 2003 results). In the United States, the mere submission of two papers criticizing our results prompted the University Corporation for Atmospheric Research (UCAR), a prominent, federally-funded institution that receives hundreds of millions of dollars for climate research to issue a nation-wide press release declaring our work is without merit. One of the two papers was shortly thereafter rejected by the journal (the other is still undergoing review), but the original press release remains on the UCAR web site. Coincidentally, the journal Nature recently attacked the use of press releases to announce research results before they are accepted at journals. They denounced the practice in the case of stem cell research, but did not criticise the UCAR.

Without getting into particulars beyond what has been publicly disclosed, none of them actually contest any of our specific findings. None dispute the nature of the undisclosed computational step. None contest the unacceptable dependence of the results on the bristlecone pines; none try to argue that bristlecone series are a valid “proxy” for temperature history. None address the failure of the hockey stick to pass simple verification tests.

One style of comment does not test the impact of the erroneous method on the hockey stick itself but on completely different data sets or on unrelated computational problems. Our reply to these responses is more or less: So what?” As far as we can see, the only place to test the impact of errors in the hockey stick method is on the calculation of the actual hockey stick.

Another type of response is to show that a hockey stick can be produced even without the erroneous method by, for instance, increasing the number of principal components used to represent the North American tree ring network. But every such permutation that we have seen boils down to a back-door method of allowing the bristlecone series to dominate the final results. Once you know of the role of these defective proxies in the hockey stick, you can’t simply ignore them. But this is what is being attempted. Further, these salvage attempts fail common statistical verification tests. However, in every example that we have seen this information is withheld from the reader (as it was in the original article). This is the case for the papers cited in the UCAR press release, for example.

A third type of response has been to mischaracterize our work. As Muller and others have clearly understood and as we have explained on many occasions, our work to date has been entirely critical. We are not advocating our own reconstruction of climate. We are simply arguing against “flawed intelligence” which is not backed by the data. If this re-opens debate for other interpretations, including those held by the IPCC in the pre-Mann era, then that would be a welcome outcome.

What has been the reaction from the government and IPCC? Not once have we been contacted by Environment Canada or any other Canadian government ministry dealing with climate research to discuss our work. I had contacted Canada’s Chief Climate Science Advisor during Kyoto negotiations (Henry Hengeveld) last fall and took him to lunch to explain our work. He shrugged it off and never followed up. Environment Canada has a comment on their website dismissing our work, based only on a claim by the original authors that the errors did not matter. A reader from Manitoba forwarded to us an email from Environment Canada responding to his question about why they still promote the hockey stick. Apparently they have dismissed our research on the basis of some unpublished and fallacious commentary they found on the internet, without ever asking for our input. We have had no contact from IPCC either.

Our efforts to promote the concept of auditing important climate studies prior to usage in public policy is getting increased attention. We have learned that people have the wrong idea about journal peer review. Users of scientific research for policymaking generally assume that when an article is published in a peer-reviewed journal it means that someone checked the data, checked the calculations and checked that the stated conclusions are supported by the evidence presented. But peer review does not guarantee any of this. Influential papers in climate research can go for years without the data or methods even being disclosed, let alone independently checked, even as huge policy investments are made based on them. So we have urged policymakers to put in place formal processes to ensure complete disclosure of data and methods for any scientific work that is being used to drive policy debates. We urge the development of audit procedures to verify compliance with such requirements. We believe such innovations would be good for science and good for the policymaking process, even if a few more scientific icons get broken as a result.

One of the first places that we would recommend such procedures is the temperature data set used by the IPCC. Other researchers have tried without success to get access to the supporting data. One of them shared with us the response he received from the principal author of the dataset: “We have 25 years invested in this work. Why should we let you look at it, when your only objective is to find fault with it?”