An Op Ed by David Legates of the University of Delaware in today’s National Post, entitled
Where’s the data?: Holding science to prospectus standards would stop climate researchers from launching misrepresentations like the ‘Hockey Stick’ By: David Legates
Tuesday, September 20, 2005
In June, the energy and commerce committee of the U.S. House of Representatives opened an investigation of a prominent scientific study and the circumstances under which it became the centrepiece of a report by the United Nations Intergovernmental Panel on Climate Change.
The investigation has many observers, including climate scientists themselves, up in arms. The Washington Post called the committee action a “witch hunt,” while others have compared it to the Spanish Inquisition. The American Geophysical Union and the American Meteorological Society have written a joint letter of protest, accusing the House energy committee of undermining science and attempting to intimidate its authors. Editors of prominent journals like Nature and Science have weighed in on even stronger terms.
Although critics contend the issue is about scientific freedom, the questions actually pertain to disclosure, due diligence and the need for access to publicly funded scientific data when public policy is at stake. In reality, the investigation is not only entirely proper, but long overdue.
The saga begins in 1998, when Michael Mann and colleagues published a graph in Nature that they argued represents the air temperature history of the Northern Hemisphere for the last 1,000 years. Owing to its shape, the curve is called the “Hockey Stick.” It shows a relatively constant air temperature (with a slight decline) from A.D. 1000 until the late 1800s. But over the last century, the air temperature dramatically increases by about 0.6C, which, the authors and believers assert, proves that humans are indeed responsible for virtually all of the climate change of the past millennium. It was the Hockey Stick that originated the sound bite declaring 1998 to be the “warmest year” of the millennium and the 1990s the “warmest decade” — a sound bite used by the Canadian government in making the case for adopting the Kyoto Protocol.
The Hockey Stick stands in stark contrast to a long-held view, amply supported by work of other researchers, that the last 1,000 years were characterized by a warm beginning (the Medieval Warm Period), a rapid cooling around A.D. 1500 (the Little Ice Age), and a latter-day recovery from this cooler period. The Hockey Stick became entwined with energy policy when the IPCC replaced this traditional view and featured the Hockey Stick prominently in its 2001 assessment of climate science — in a section written by Mann himself. It surprises many to learn that the IPCC assessment often is written by scientists who dominate the debate about specific issues.
Clearly such scientists have axes to grind and, in Mann’s case, he used the IPCC as a forum to promote his own research. Other IPCC authors admonished Mann to include other, less Hockey Stick-like representations in his assessment. They were ignored in the final report, however, and, owing to the influence that the IPCC reports carry, the Hockey Stick became a public icon, enthusiastically promoted by supporters of the hypothesis of greenhouse warming.
The statistical methods used by Mann and his colleagues have been the subject of much recent scrutiny. Based on our own research and a detailed comparison with the published evidence, Willie Soon and I raised the spectre of flawed statistics in the Hockey Stick when we testified with Mann at a U.S. Senate committee hearing in 2003. Subsequently, two Canadians with strong statistical training — energy analyst Stephen McIntyre and economist Ross McKitrick — attempted to replicate Mann’s results using the data he had supplied them. They found a number of errors, improper calculations, and misrepresentations of methodology. In the refereed literature, other researchers have expressed concerns about and demonstrated problems with the Hockey Stick.
The McIntyre and McKitrick study led to a corrigendum in Nature, where Mann and his colleagues admitted to various inaccuracies in their original description of their data and analysis. Nature took the extremely unusual step of requiring Mann and co-authors to provide a new archive of data and a new verbal description of their methodology. But even with this revised release, key aspects of the Hockey Stick remain impossible to replicate — and replication is a hallmark of scientific inquiry. Mann continues to refuse requests for full disclosure, telling The Wall Street Journal that to do so would amount to “giving in to intimidation.”
Despite the importance of the Hockey Stick for climate policy and the repudiation of scientific ethics implicit in Mann’s statement, there was no reaction to The Wall Street Journal article by the U.S. National Research Council or any learned societies and virtually no shock or surprise from climate scientists themselves. However, these extraordinary and injudicious remarks by Mann did attract the attention of the U.S. House energy and commerce committee, an important committee with broad investigatory powers, which carried out hearings on Enron, for example.
But the issue here goes beyond data and methodological documentation. The energy and commerce committee asked Mann and colleagues about the withholding (from their analysis) of vital statistical information that was highly adverse to their claims. This amounts to selectively choosing data to support their position and ignoring data that refutes it. But the academic community has misconstrued the intent of the committee by largely assuming it is attempting to decide nuances of statistical interpretation. In fact, the committee is on much more familiar turf than the learned societies have appreciated: Their request regards issues of disclosure, framed in the language of securities legislation — terminology with which the House committee is completely familiar. If science were subject to prospectus standards, withholding of such information would not have been permissible.
“Informational hoarding” is being challenged. Some academic journals now require publication of all data and computer code along with the article itself. The U.S. National Institutes of Health, which funds many billions of dollars worth of medical research, has mandated that large grants are conditional on data sharing. Other federal agencies are now beginning to consider NIH’s lead to provide verification of important findings.
Since the House energy committee is responsible for energy policy, it has every right to demand additional scrutiny for studies upon which energy policy is being made. Failing to disclose data or methods is not an acceptable option when energy policy is at stake. Moreover, since Mann was the author of the section of the IPCC that touted his own research before others had the opportunity to critically re-examine his work, serious questions must be raised about conflicts of interest within the IPCC and how it came to promote speculative findings that had not been independently evaluated and which since have been shown to be flawed.
The outrage expressed by the AGU, AMS and other scientific societies is hypocritical. Funding for climate science amounts to several billion dollars a year, but these groups strongly protest the accountability that goes with it. Both the AGU and AMS have adopted statements calling on the United States to change its energy policies in light of the climate-change issue. Yet while they insist that this research be the basis for policy decisions, they object to its scrutiny by policymakers.
In this instance, the House energy committee has uncovered a real problem in science — one that extends far beyond the climate-change issue. Scientists must demand that results and conclusions stand up to independent verification. Yet since the climate-change community has failed to impose such standards on itself, it cannot be surprised if legislators have opted to do the job for them.
____________________________________________________________________
Dr. David Legates is an associate professor and director of the Center for Climatic Research at the University of Delaware in Newark, Del.
Climate Audit 
20 Comments
“No-one expects the Spanish Inquisition!” etc….
It seems largely a nice article, but seems to have a few unfortunate flubs, for example it conflates MBH98 with MBH99, and refers to you as an “energy analyst”.
Not to be picky; I just want perfection. 🙂
Here’s another flub:
“But over the last century, the air temperature dramatically increases by about 0.6C, which, the authors and believers assert, proves that humans are indeed responsible for virtually all of the climate change of the past millennium“.
Actually, I think what they claim is that humans are responsible for virtually all of the climate change of the past 100 years, or even the past 50 years.
No, Jaime, I think the article means what it says there. If, according to the proxy record, the past millenium was basically flat temperaturewise with a slight downward trend, and the recent warming is most all from human activity, then the imputed claim follows.
Today there is an article in the BBC News titled Mars ‘more active than suspected’ . The article includes the statements:
The agency mentioned in the first sentance is NASA.
The folks over at RealClimate seem to have missed today’s BBC article as well as David Legates Op-Ed piece.
It is unfortunate that David Legate’s Op Ed piece had a few errors. However, I do not believe that correcting these errors would effect his main points.
On the other hand, if we could correct the statistical errors in MBH98, the hockey stick might well vanish.
Does anyone else have a problem reconciling this comment from the op ed:
“the academic community has misconstrued the intent of the committee by largely assuming it is attempting to decide nuances of statistical interpretation.”
With this question in the letter from Barton to Mann:
a. Did you run calculations without the bristlecone pine series referenced in the article and, if so, what was the result?
b. Did you or your co-authors calculate temperature reconstructions using the referenced “archived Gaspe tree ring data,” and what were the results?
c. Did you calculate the R2 [sic] statistic for the temperature reconstruction, particularly for the 15th Century proxy record calculations and what were the results?
d. What validation statistics did you calculate for the reconstruction prior to 1820, and what were the results?
e. How did you choose particular proxies and proxy series?”
?
Let them get the information. This is an investigation. No reason for them not to dig. a real scientist does not care try to hide the details or refuse to answer questions by swtiching from witness answering Q&A mode to “advocate arguing a point mode”. Read some Feynmann.
Re #7: The intent of the committee, I conjecture (not being privy), extends to understanding how IPCC reports are produced, including whether and to what extent major factual assertions are independently checked. Over the years the IPCC has made some rather grand claims about the rigour of its assessment process, and these claims have been key in getting wide acceptance for the IPCC reports. In the TAR the hockey stick was a central exhibit. The TAR claimed the hockey stick had significant reconstructive skill; supporting literature by Mann and coauthors claimed their reconstruction was robust to the exclusion of dendrochronological indicators; its exclusive usage by the IPCC implied it was more reliable than the other graphs then available in the literature, and its visual prominence implied that readers ought to weigh it heavily in forming policy conclusions. By asking these questions of Mann, the Committee is, in effect, asking the IPCC (since Mann was the author of the relevant sections) to explain what they knew at the time they prepared the TAR.
If they were questioning the Bre-X geologists they might ask questions about whether certain types of verification tests had been done and what the results were. This wouldn’t imply they were going to weigh in on technical debates about the statistics, only that they want to know what the analysts knew at the time they were selling shares based on claims of a major gold deposit.
re:#7
Hello, Mark!
Let’s look at what the committee was faced with. Reports were floating around that there were various problems with MBH98 and other proxy reconciliations and that both the principals and their publishers / funding agencies were not providing the information available to people having a serious interest in determining the reproducibility of their findings.
Since policy decisions were being requested relying, at least in part, on these findings, the possibility that these results couldn’t be verified is a serious business. As is usual in investigations involving possible malfeasance, the committee requested all the information which might be required in a hearing so as to be prepared to address whatever line of investigation came up.
Since Steve has claimed that the Hockey team indeed ran a check of their methods without bristlecone pines, and didn’t report the result it was certainly important to know if this was the case or not. Likewise, what the result was would be important is seeing if there was a good reason for withholding the data.
Consider a plausible line of questioning.
Congressman: In earlier testimony it was stated you ran your methods on a dataset lacking the bristlecone pines but didn’t report it because the results were unfavorable. What do you have to sa about this?
Dr. Mann: Hmmm. To tell the truth I don’t know if such a run was made or not. You have to realize I was just the overseer, so to speak, and I couldn’t be bothered with each and every run and certainly wouldn’t remember the result at this late date in any case. And certainly if it was done and the results had been of any significance I would have been told.
Admittedly the committee could then require Dr. Mann to go back and find the data and reappear, but that’s just the point. A committee of Congress doesn’t want to get stuck in such time-wasting situations. Therefore they send off requests for information well ahead of time so that it’s available when needed. It’s disengenuous to say you can’t see what value this sort of data might have when trying to sort out how forthcoming the Hockey Team was in responding to legitimate requests for information during a public hearing.
Mark, the questions were factual and did not ask for a statistical interpretation or justification. Mann resumably they could have answered as follows:
a. Yes. Values of the temperature index in the 15th century were much higher than when the bristlecones were included.
b. Yes. Again, values of the temperature index in the 15th century were higher, when the archived version of the Gaspe series was used.
c. Yes. The value of the R2 statistic for the 15th century reconstruction was 0.02.
d. Values of the R2 [ CE, Sign and Product Mean Test] for the calculation steps commencing in 1400,1450, 1500,…,1800 are shown in Table 1 attached hereto.
e. The “clear a priori criteria” referred to in Mann et al. [2000 are …[I have no idea, but it would be interesting to find out.]
Re: #11
Yes, but that would be someone who had no doubts about the validity of the statistical treatment and the robustness of the claims made upon them. What kind of world are you living in Steve?
Re #11. Steve – surely the committee have made a statistical judgement by selecting those questions as opposed to all the others they might have asked. It makes no sense to ask whether Mann calculated the R2 statistic unless you believe that he ought to have done so and that the RE statistic by itself was inadequate – this is a technical judgement and quite a sophisticated one. The committee would presumably also expect a fairly detailed technical response from Mann (which they got) and are therefore committing themselves to further technical assessments and judgements. They can hardly have expected Mann to give the one-liner responses you suggest. It is analogous to an ID champion asking a molecular biologist “Did you ever calculate the probability of all the genes needed for the human eye arising by chance mutation”. The biologist can’t reply simply “no” or “yes and its a zillion to one”. He has to give an explanation or he will be misintepreted.
More broadly I don’t see how they can move this forward without effectively appointing themselves arbiters of the science. They may like to think that they are only assessing process and due diligence but in practice they will be drawn into a technical morass – just look at the complexity of the discussion here and on realclimate. But this isn’t like Bre-X because they cannot call on disinterested experts. The danger is that they will either just end up wasting a lot of everyone’s time or worse they will jump to a conclusion based on insufficient evidence – and that this will become a feature of all science with policy implications in the future.
PS I am not at all a supporter of Mann and co. Your points seem very credible to the extent I understand them. I just think this is a crap way of trying to resolve the issue.
Mark,
What “discussion” on RealClimate? They studiously avoid any discussion of the issues Barton’s committee is interested in.
Anyway, don’t be disengeneous. You know, I know and everyone knows that the point of the committee is to get to the bottom of why Steve (and others) can’t get the information which should be forthcoming if the data archiving policies being enforced were what they should be both according to the published policies of the Journals, Universities, etc. and what the the Government expects them to be.
Consequently the questions asked are ones necessary to see if Steve should have been able to obtain what he was looking for and whether or not the failure to do so is reflective of laziness (by Mann and others) or an attempt to hide something.
I’m sure you have preconceptions as to what Barton is looking for, but then why are you complaining about Barton himself having expectations? There’s a difference between an expectation and a procedure. AFAIK, the procedure followed by Barton is just what would be expected by a committee going about its normal business.
Mark, obviously someone connected with the Committee read our E&E article, where similar questions are posed (look at the end of the article). I can tell you – without speculating – exactly what was on my mind when we raised these questions in the first place. These are disclosure issues – not statistical issues – and explicitly raised as securities type issues in our article.
I have considerable experience in disclosure requirements for securities offerings. You are legally obliged to disclose relevant adverse information.
I’ve heard a lot of complaining from skeptics about supposedly biased IPCC reporting, a lot of which seemed like idle complaining unless you had an objective standard of disclosure. The objective standard that I’ve used is the disclosure standard applicable to mining promoters in a prospectus. If the IPCC process and Nature peer review are as wonderful as advertised, then climate scientists should easily surpass mining promoter standards of disclosure. Since disclosure standards for securities offerings have been tested through litigation and prosecutions, the obligations are relatively objective and relatively well understood. This is not to say that they are always met, but they are sufficiently objective that courts can enforce them.
The disclosure issues in our E&E article were ones that really stuck in my craw – and this is from someone who understands mining promotions.
1) If Mann calculated an R2 statistic and it was adversely insignificant, by prospectus standards, that was mandatory disclosure.
2) Likewise with bristlecones.
3) Likewise with Gaspe and the failure to disclose the unique “editing”
4) false claim about bristlecone robustness
These are essentially securities issues. It doesn’t matter what the statistic is. What matters is – was it calculated? was it disclosed?
These are really misconduct issues that should have been investigated by the universities or by the learned societies. They are related to, but distinct from, the policy issues e,g, conflict of interest by IPCC reviewers relative to promoting their own articles; adequacy of journal peer review as a foundation for engineering-equivalent calculations; adequacy of IPCC review process i.e. stadiums of peer reviewers “commenting”, but no one actually checking any calculations; NSF failures to ensure archiving and responsive behavior from their grantees (and even their being co-opted).
If there are no disclosure standards for climate science equivalent to those applying to mining promoters, and climate scientists do not always meet mining promoter standards, then there’s an issue that the Committee should investigate.
The answers from Mann et al. were not responsive but were extraordinarily cheeky. I’d never have been that cheeky to a senior committee. It’s like they’re asking for trouble.
I’m sure that the interest of the Committee is more in the policy issues. The Mann case comes into play because it is a practical example of a lot of problems and it’s better to work from practical examples to more general cases. However, M,B,H cheekiness may rile the Committee.
If I were them, I’d be inclined to ask one of the learned societies to investigate the misconduct aspects of the MBH failure to disclose. That would have one real advantage: it would isolate whether the failure to disclose was a defect in disclosure standards applicable to climate science or whether it was misconduct by the individual scientists – both of which lead in quite different directions.
Steve
There are many things I would like to say about this. However, I believe this forum is intended to be about climate science issues and the many things are not at all to do with the science. So I will hold fire unless someone says its OK.
Mark
Re #16, Mark Frank
Your previous comment #13, although I disagreed with it, was perfectly sensible and thoughtful. So I say, it’s OK, go ahead with your many things.
(Not that what I say particularly matters, ‘cos it isn’t my blog.)
Those perhaps not familiar with the mining and exploration industry might be interested to know that the industry around the world has for some years, and particularly following the Bre-X debacle in 1996, been developing codes of appropriate reporting for public documents that are very often adopted by the regulatory authorities.
In Canada, the underlying obligation on the discloser is to ensure that his disclosures are “True, Plain and Fair.”
In Australia, the relevant Code is the “JORC Code (2004) – Australasian Code for Reporting of Exploration Results, Mineral Resources and Ore Reserves” prepared by The Joint Ore Reserves Committee of The Australasian Institute of Mining and Metallurgy, Australian Institute of Geoscientists and Minerals Council of Australia (JORC). http://www.jorc.org/pdf/jorc2004web.pdf
A quote from SCOPE is instructive:
“The main principles governing the operation and application of the JORC Code are TRANSPARENCY, MATERIALITY and COMPETENCE.
TRANSPARENCY requires that the reader of a Public Report is provided with sufficient information, the presentation of which is clear and unambiguous, to understand the report and is not misled.
MATERIALITY requires that a Public Report contains all the relevant information which investors and their professional advisers would reasonably require, and reasonably expect to find in the report, for the purpose of making a reasoned and balanced judgement regarding the Exploration Results, Mineral Resources or Ore Reserves being reported.
COMPETENCE requires that the Public Report be based on work that is the responsibility of suitably qualified and experienced persons who are subject to an enforceable professional code of ethics.”
There is of course much more detailed information in the JORC Code, but it seems to me that there is nothing at all unusual on imposing those preparing public reports using public moneys the obligation to meet these basic standards of good practice. It is interesting to view the Hockey Stick debate, and the reports of the IPCC in the light of these requirements.
Mann and the IPCC are on a crusade. You can’t expect them to meet strict scientific criteria when they are on the moral high ground. This is moral science they are doing, which has to meet different standards than regular science. Someone must have forgotten to tell the House Committee.
Thanks.. Good article..