Phil Jones’ written answers to the Muir Russell panel shed interesting light on the insularity of IPCC authors, who see nothing odd about a system in which reviewers do not see either author responses to their review comments or the comments of other reviewers until long after the release of the final document. Jones’ comments were made in connection with questions from Boulton about Jones’ threat to keep McKitrick and Michaels 2004 out of IPCC and McKtrick’s allegation that Jones and other chapter 2 authors, having grudgingly agreed to refer to McKtrick and Michaels 2004, had “fabricated” IPCC’s editorial comment that its results had no statistical significance. This topic is revisited in AR5 (First Draft) where IPCC (with surprising candour) admitted that there was no “explicit” basis for the disparaging claim in AR4.
Jones was interviewed by the Muir Russell panel (actually just Boulton and Peter Clarke) on April 9, 2010.
On April 15, Boulton sent follow-up questions to Jones about his handling of McKitrick and Michaels 2004, an issue that had attracted notoriety as a result of IPCC Coordinating Lead Author Jones’ email saying:
The other paper by MM [McKitrick and Michaels] is just garbage. […] I can’t see either of these papers being in the next IPCC report. Kevin [Trenberth] and I will keep them out somehow — even if we have to redefine what the peer-review literature is!
Despite Muir Russell’s promises at his opening press conference and despite requests
from the Parliamentary Science and Technology Committee, for the most part, the Muir Russell panel failed to create a useful record of questions and answers. Boulton’s questions about McKitrick and Michaels 2004 are a rare exception.
In apparent compliance with Jones’ 2004 threat, the chapter of which Jones and Trenberth were Coordinating Lead Authors contained no mention of McKitrick and Michaels 2004 in the two drafts sent to reviewers (First Order Draft and Second Order Draft). In the final AR4 report, IPCC grudgingly mentioned McKitrick and Michaels 2004 (and the related de Laat and Maurellis 2006), adding the adverse editorial assertion that they ceased to have “statistical significance” when atmospheric circulation was considered – a comment which, according to McKitrick’s strongly worded submission to Muir Russell, was both untrue and which had no support in the peer reviewed literature that IPCC was supposed to draw on.
Boulton’s follow-up questions on April 15, 2010 asked Jones about the basis for the IPCC claim that the McKitrick and Michaels 2004 had no “statistical significance” as follows:
When the final IPCC FAR was published in May 2007, it included a new paragraph in Chapter 3, on page 244, that referred to the McKitrick and Michaels (2004) and De Laat and Maurellis (2006) papers, and that had not been included in either of the drafts shown to reviewers. It is assumed that this was either written by you, or in consultation with Trenberth, but in any case, the two of you, as Coordinating Lead Authors, bear responsibility for its inclusion. It reads:
“McKitrick and Michaels (2004) and De Laat and Maurellis (2006) attempted to demonstrate that geographical patterns of warming trends over land are strongly correlated with geographical patterns of industrial and socioeconomic development, implying that urbanisation and related land surface changes have caused much of the observed warming. However, the locations of greatest socioeconomic development are also those that have been most warmed by atmospheric circulation changes (Sections 184.108.40.206 and 3.6.4), which exhibit largescale coherence. Hence, the correlation of warming with industrial and socioeconomic development ceases to be statistically significant (highlighting added). In addition, observed warming has been, and transient greenhouse-induced warming is expected to be, greater over land than over the oceans (Chapter 10), owing to the smaller thermal capacity of the land”.
Q1: What is the justification for what appears as an ad hoc conclusion not based on published research that summarily dismisses an argument that is based on peer-reviewed research?
Q2: Why were these conclusions not shown to or discussed with expert reviewers during the IPCC Report preparation?
Q3: The references to sections 220.127.116.11 and 3.6.4 of the IPCC Report are misleading since neither section presents evidence that warming due to atmospheric
circulation changes occurs in the regions of greatest socioeconomic
development. Neither section even mentions industrialization, socioeconomic
development, urbanization or any related term. How can they therefore be
used to justify the stance of the above quotation?
Q4: No justification is given for the claim of statistical insignificance, which has a precise meaning. Do you have a p value that justifies this statement, and if not,
what does it mean?
I will not discuss all the answers in this note and urge interested readers to consult the original here.
Let me start this quick review with the fourth question of Issue 2, since “statistical significance” is a technical term and IPCC made very specific assertions on this point.
As McKitrick had alleged, Jones did not have a p-value that justified his claim, but argued that “there is no need to calculate a p value for a statement that is based on the laws of physics”.
The pattern of atmospheric-circulation-related warming appears similar to the geographical distribution of socioeconomic development. Such similarity makes it impossible to use purely statistical methods to ascribe patterns of warming trends to patterns of socioeconomic development. It remains possible, however, to ascribe patterns of warming trends to atmospheric circulation because its influence is in accord with the laws of physics and can be detected in day-to-day weather variations, on which timescales socioeconomic trends are infinitesimal. As stated, it is essential to extract the known and understood influences first and then look at the residuals. There is no need to calculate a p value for a statement that is based on the laws of physics.
The latter statement surely raises epistemological issues on which Phil Jones hardly stands as an authority. (For example, I am unaware of any publications by Jones in epistemological literature.) But even if Jones were a qualified epistemological authority (which he isn’t), the statement in question was not “based on the laws of physics”. The IPCC statement in question was “the correlation of warming with industrial and socioeconomic development ceases to be statistically significant” – this statement may be based on an argument from physics, but it is a statistical statement and was not itself based on the laws of physics. Boulton should not have accepted such flannel.
In response to the related first question, Jones denied that the IPCC assertion about the lack of statistical significance was “ad hoc”, but, instead of evidence, merely presented a whinging complaint about too many academic papers:
The fact that MM2004 is in the peer-review literature does not mean it is good science. There are examples of poor science across all areas of science in the peer-review literature. Occasionally scientists submit comments on poor or incorrect papers, but this sadly is something of a rarity. With the plethora of journals it is becoming harder and harder to read and respond to all the literature. One could make a full time job of publishing criticisms of poor or incorrect papers.
Any competent inquiry would have seen through Jones’ flannel, but not Muir Russell. Although Boulton asked supplementary questions, none addressed these issues. Nor did the final report contain anything but flannel on the topic.
By the way, there’s an interesting development on this front in IPCC AR5 (First Draft) which admitted (with surprising candour) that AR4 “provided no explicit evidence” for the claim that the McKitrick and Michaels 2004 (and de Laat and Maurellis 2006) results had no statistical significance as follows (more on this on another occasion):
McKitrick and Michaels (2004) and de Laat and Maurellis (2006) analysed surface air temperature trend fields and assessed potential for biases in terms of national socioeconomic and geographical indicators. Both studies concluded that urbanisation and related land surface changes have caused much of the observed warming. According to the AR4, the correlation of warming with industrial and socioeconomic development ceases to be statistically significant if one takes into account the fact that the locations of greatest socioeconomic development are also those that have been most warmed by atmospheric circulation changes. AR4 provided no explicit evidence for this overall assessment result.
Returning to Boulton’s (sensible) second question – a question that remains relevant with IPCC’s recent efforts to extinguish public discussion of draft documents in real time:
Why were these conclusions not shown to or discussed with expert reviewers during the IPCC Report preparation?
Jones explained to Boulton (and one can sense an almost Gavinesque sigh) that IPCC reviewers don’t get to see author responses to their comments at each stage (First Draft or Second Draft), that they only saw the author responses long after publication of the final report, that this was an IPCC system and that reviewers “were aware” of and accepted this system as a condition of submitting review comments. Jones:
The comment/response files for each stage were not released after each review, but only released together when the final report was published in May 2007. You seem to be under the impression that expert reviewers saw responses to their
comments at each stage. This has never been the case in any IPCC Report. This was an IPCC decision and all reviewers were aware of this when they made their reviews.
A system in which reviewers do not see author responses or review comments from other reviewers until after (and some months after) the final report surely warrants some introspection even if IPCC reviewers have acquiesced in such a system in the past. Nor does such a system appear to me to comply with (reasonable) public expectations that IPCC procedures be “open” and “transparent” – as IPCC officials like to proclaim.
In addition, I would be surprised if this sort of system is compliant with, for example, US federal standards for “influential scientific assessments”, a failure which may well create problems down the road for US agencies seeking to rely on IPCC reports without triggering fresh public comment and peer review.