Climate Audit

Yes, Minister was a wonderfully funny British comedy of a generation ago, (some info here and here or google yes minister humphrey) which featured Sir Humphrey Appleby as a senior civil servant (deputy minister) "managing" the Minister, his nominal boss, but who was always outwitted by Sir Humphrey. Each episode would start off with the Minister having some bright idea of instituting change, often involving a cut to civil service budgets. Sir Humphrey would resist and no budget would ever be cut. In a way, Sir Humphrey is descended from PG Wodehouse’s Jeeves. Here are some typical bon mots:

“It is axiomatic in government that hornets’ nests should be left unstirred, cans of worms should remain unopened, and cats should be left firmly in bags and not set among the pigeons. Ministers should also leave boats unrocked, nettles ungrasped, refrain from taking bulls by the horns, and resolutely turn their backs to the music.”

“Almost anything can be attacked as a failure, but almost anything can be defended as not a significant failure.”

“If people don’t know what you’re doing, they don’t know what you’re doing wrong.”

Keep that thought in mind as we see how Sir Humphrey (Cicerone) dealt with the Boehlert questions in classic Yes, Minister style.

You’ll remember that, on June 23, 2005, the Barton Committee sent a letter to Mann (with similar letters to Bradley and Hughes) asking:

7c. Did you calculate the R2 statistic for the temperature reconstruction, particularly for the 15th Century proxy record calculations and what were the results?7d. What validation statistics did you calculate for the reconstruction prior to 1820, and what were the results?

On July 15, 2005, Cicerone, the President of the National Academy of Sciences, in best Sir Humphrey Appleby fashion, politely invited Barton to butt out, suggesting that NAS expert panels would be much better at resolving this sort of question, citing the r2 question in particular. On July 23, Mann himself told the Barton Committee:

My colleagues and I did not rely on this statistic in our assessments of “skill” (i.e., the reliability of a statistical model, based on the ability of a statistical model to match data not used in constructing the model) because, in our view, and in the view of other reputable scientists in the field, it is not an adequate measure of “skill.”

(Anticipating my report on Mann’s presentation a little, I’ll mention that one of the panelists asked him what the verification r2 statistic was for the 15th century. Mann said that they did not calculate the r2 statistic -" that would be silly.") Given that MBH98 mentioned the term “r2” on no fewer than 7 occasions and even has a map showing the verification r2 statistic for the AD1820 step, you’d expect his nose to grow when he said this, but that’s a story for another day.

Last July, the House Science Committee objected to the Energy and Commerce Committee intruding onto their turf and, later, with matters still unsettled, commissioned NAS to investigate the MBH controversy through a NAS expert panel. Their requisition included the following:

I am writing to ask you to empanel a balanced group of scientist to provide Congress with expert guidance on the current scientific consensus on the paleoclimate recod and particularly on the work of Drs. Michael Mann, Raymond Bradley and Malcolm Hughges (the so-called "hockey stick" thesis. The group should, in a clar and concise report issued in a relatively short period of time answer the following questions …

2) (b) What are the principal scientific criticisms of their [Mann, Bradley and Hughes] work and how significant are they? (c) Has the information needed to replicate their work been available? (d) Have other scientists been able to replicate their work?

Now let’s look at the actual terms of reference for the Panel posted in full here (this is a climateaudit link, I’ve seen the summary at other URLs, but I don’t have a third party link to the full Statement of Task handy):

SUMMARY
This study will describe and assess the state of scientific efforts to reconstruct surface temperature records for the Earth over approximately the past 1,000-2,000 years. The committee will summarize current scientific information on the temperature record for the past 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. The committee will address tasks such as identifying the variables for which proxy records have been employed, describing the proxy records that have been used to reconstruct surface temperature records for the pre-instrumental period, assessing the methods employed to combine multiple proxy data to develop surface temperature reconstructions, discussing the geographical regions over which proxy data can be reliably extrapolated, and evaluating the overall accuracy and precision of such reconstructions.

….
There has been significant controversy over the techniques used to make such projections. Thus, in late 2005 Chairman Sherwood Boehlert of the US House of Representatives Committee on Science requested that the National Academy of Sciences empanel a committee to provide Congress with expert guidance on the current scientific consensus on surface temperature reconstructions. His request highlighted the need to understand the current scientific consensus on the temperature record for the last 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used (including description of the scientific criticisms and how significant these criticisms are), and explain how central the debate over the paleoclimate temperature record is to the overall scientific consensus on global climate change.

Technical Context:
The most useful data for climate reconstructions are annually or seasonally resolved proxies, which are typically combined into “multiple proxy” assemblages that are then compared against long temporally overlapping instrumental records to calibrate and verify the climate signal in the data. Most of the controversy surrounding surface temperature reconstructions centers on the method by which these multiple proxy assemblages are created and calibrated. The reliability of certain proxy data sets has also been the subject of debate, as has the geographical extent over which a limited number of proxy data can be reliably extrapolated. An authoritative evaluation of the reliability of proxy data, the methodologies employed to create multiple proxy assemblages, and the overall uncertainties associated with surface temperature reconstructions would help resolve these controversies and delineate the appropriate role for paleoclimate indicators in evaluating global change.

PLAN OF ACTION

Statement of Task
The committee will describe and assess the state of scientific efforts to reconstruct surface temperature records for the Earth over approximately the past 1,000-2,000 years. The committee will summarize current scientific information on the temperature record for the past 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. As part of this effort, the committee will address tasks such as:

* Identify the variables for which proxy records have been employed (e.g., temperatures averaged over specific geographical areas and time periods, extreme temperatures).
* Describe the proxy records that have been used to reconstruct surface temperature records for the pre-instrumental period (e.g., tree rings, sediment cores, isotopes in water and ice, biological indicators, indicators from coral formations, geological boreholes, historical accounts) and evaluate their limitations.
* Assess the various methods employed to combine multiple proxy data to develop surface temperature reconstructions and the major assumptions associated with each approach.
* Discuss the geographical regions over which proxy data can be reliably extrapolated.
* Evaluate the overall accuracy and precision of such reconstructions.
….

Here is the wording in the summary at the NAS project webpage:

the committee will summarize current scientific information on the temperature record for the past 1,000-2,000 years, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. The committee will address tasks such as identifying the variables for which proxy records have been employed, describing the proxy records that have been used to reconstruct surface temperature records for the pre-instrumental period, assessing the methods employed to combine multiple proxy data to develop surface temperature reconstructions, discussing the geographical regions over which proxy data can be reliably extrapolated, and evaluating the overall accuracy and precision of such reconstructions.

Towards the end of von Storch’s presentation, he posted up the Boehlert questions and started giving answers (as we were about to do as well.) It turns out that most, if not all, of the panelists had not seen the Boehlert questions. Von Storch’s introduction of the Boehlert questions prompted a discussion about whether these questions were within the scope of the panel’s mandate. Von Storch criticized MBH replicability giving a very categorical answer to one of the Boehlert questions that was identical to ours. So the panel has testimony on the matter. We also pointed out that presenters D’Arrigo and Hegerl had not archived their data and had refused to make it available as part of the IPCC review process. This sparked responses from both D’Arrigo and Hegerl purporting to justify this and concerns by the chairman about whether replication and data archiving was going off topic and distracting from the questions that they were charged with answering, even though these were obviously questions specifically asked by Boehlert.

Goldston, representing the House Science Committee, closed off the first day’s proceedings by observing as a public comment that the Science Committee realized that there were many large questions associated with climate change and recognized that there were many big and contentious issues still to come in the future. However, the Science Committee had intentionally asked some finite questions associated with current controversies, since they wanted to take at least a few small issues off the table.

Document Comparisons
1. Let’s first compare the characterization of the Boehlert questions in the terms of reference document to see if NAS has accurately represented the scope of the Boehlert questions in its Statement of Task. NAS said that Boehlert had asked NAS to "describe the principal methodologies used (including description of the scientific criticisms and how significant these criticisms are)" Do you think that this is an accurate representation of the Boehlert question: " 2) (b) What are the principal scientific criticisms of their [Mann, Bradley and Hughes] work and how significant are they? (c) Has the information needed to replicate their work been available? (d) Have other scientists been able to replicate their work?"? Note that NAS completely left out any specific mention of MBH, which was front and centre of the Boehlert questions, and left out the specific concern over replication.

2. Next let’s compare the NAS characterization of the Boehlert questions to the Statement of Task in the running text shown above. As noted above, NAS characterized the Boehlert questions as asking them to "describe the principal methodologies used (including description of the scientific criticisms and how significant these criticisms are)". The Statement of Task in the corresponding section uses the term "describe the principal methodologies used and any problems with these approaches". Again no specific mention of MBH or replication. Is there a difference between the terms used in each case? Is the change in wording accidental or intentional? If it was intentional, what is intended by the change in wording? In the specific list of tasks, there is again no mention of specific tasks related to MBH or replication.

3. The Boehlert letter was not attached to the panel’s Statement of Task. Was this intentional or accidental? Both von Storch and ourselves, independently assumed that the Boehlert questions were being addressed by the panel, while the panel itself didn’t seem to be aware of the specific Boehlert questions.

The manoeuvring over terms of reference is really quite delicious in Yes, Minister terms. The Minister (Committee) has dared to ask to ask a question, has attempted to grasp a nettle, perhaps even to open up a can of worms. There are lots of possibilities.

Now a panel that was so minded could obviously take the position that replication is fundamental to the scientific method and that criticisms about replicability are "scientific criticisms". Indeed, I intend to urge the Committee to use the Boehlert questions as a guide to interpreting the scope of what is involved in a "description of the scientific criticisms" as the lack of replicability is obviously pretty fundamental. However, they might use the language of Cicerone’s Statement of Task to simply avoid doing any of the inevitably unpleasant work of sorting through MBH and MBH replicability.

If the latter, one can see a variety of outcomes. If the NAS panel doesn’t answer the Boehlert questions, then it seems to me (and I’m just a fascinated spectator to the process) that Cicerone will have totally sandbagged the Science Committee in front of the Energy and Commerce Committee. The Science Committee will have asked straightforward questions and got Sir Humphreyed by Cicerone. They will look like the gang that can’t shoot straight. No wonder Goldston asked the panel to pay attention to the Boehlert questions.

Similarly, if the NAS panel doesn’t answer the Boehlert questions (and the Barton questions before them), the Science Committee will hardly be in any position to object if the Energy and Commerce Committee starts hearings – after all, the Science Committee wouldn’t have been able to get a few simple questions answered. Nor would Cicerone or NAS be able to object to the Energy and Commerce Committee opening hearings having asked for the opportunity to answer the Barton questions (r2 etc.) and then failing to answer either them or the Boehlert questions.

The situation for the panel is pretty complicated. Should they go along with Sir Humphrey and just duck the issue or should they ask Cicerone for a clarification of their task (referring to the fact that both von Storch and ourselves brought the Boehlert questions to their attention and even presented them with answers to the Boehlert questions)? Or should they proceed to answer the Boehlert questions on their own? For what it’s worth, when Mann presented on Friday, no one even asked him for the record the Boehlert question: Has the information needed to replicate [your] work been available?