I’m posting up our July 2005 review of Ammann and Wahl. The recently accepted A&W version is here. I’m posting this up for a variety of reasons. Mann relied heavily on Ammann and Wahl in his NAS panel testimony (which wasn’t even online as accepted last week) and so it’s hard to finish off the discussion of NAS without wading through Ammann and Wahl. Also the associated UCAR Media Advisory that all our claims were "unfounded" had been widely cited.
Comparing A&W against my previous comments in a peer review capacity is an interesting object lesson in what peer review is and isn’t. It’s also interesting to speculate as to whether IPCC considerations may have affected how it’s been handled.
Differences between Versions
First, the information about verification r2 statistics and the various arguments for r2 versus RE, referred to a couple of posts ago, was not in the original submission. The following is new in A& W Revised: Section 2.3 entitled “Calibration and Verification Statistics — Validation; the first few paragraphs of Section 5. Summary and all four appendices, plus Tables 1S, 2S and Figure S1. Some sentences have been changed here and there, but otherwise the rest of the text looks pretty much word-for-word what we saw last June.
I suspect (but do not know) that these sections have been added since December 14, 2005. When I met Ammann in San Francisco, he indicated that he’d not had to make any significant changes. We discussed the disclosure of adverse verification r2 results. I strongly urged him to get in front of the issue or else he’d be in an even worse situation. He didn’t agree. Something’s changed his mind since December 14. Maybe it’s my pounding away at climateaudit. Or maybe there were strings attached to the “provisional acceptance” received on Dec. 12 , 2005 (which I was unaware of in San Francisco). It doesn’t really matter. The sections are new.
Obviously, the table of verification statistics completely vindicates our claims on this matter in McIntyre and McKitrick [GRL 2005] and makes the UCAR Media Advisory that all our claims were “unfounded” in need of retraction.
Background to our Climatic Change Review
This was the second article that I’d reviewed for Climatic Change. Stephen Schneider, who is prominent in the climate debate and a critic of ours, had particularly criticized E&E back in 2003 for not allowing Mann et al. to review our 2003 article. Whether that particular criticism was merited, Schneider then had the virtue of consistency and, when MBH submitted a supposed rebuttal of MM03 to Climatic Change, I was invited to be a reviewer. I’ve found Schneider to be an engaging correspondent, although I’m not very happy about what he’s done here.
Obviously, I approach things from a different angle. In my capacity as a reviewer of the MBH 2004 submission, I asked for the residual series, cross-validation statistics and source code that Mann had refused to provide to me as an opponent in controversy.
This provoked some soul searching at Climatic Change. Schneider said that no one had asked for such things in 28 years of his editing the magazine. He was worried that, if reviewers were expected to examine source code, no one would ever review for his journal. I said that I didn’t expect everyone to examine source code, but that I wanted to and needed to.
Their editorial board met and decided that they would not even ask Mann for source code, but agreed that the journal would have a policy requiring authors to provide supporting data.
OK, I asked for supporting data such as cross-validation statistics and residual series under the newly established data policy. Mann refused. I sent in a review that Mann had flouted the newly established data policy and therefore the article could not be accepted. I never heard anything more about this article and it never appeared.
Of course, by this time, Jones and Mann  had cited the article (in review) as trashing us and did not withdraw the comment in Jones and Mann  when the underlying article was rejected. Jones and Mann  in turn has been quoted on a number of occasions.
On May 12, 2005, I was asked to review the Ammann and Wahl submission. Schneider knew what to expect from me. Equally, I had been given a surprisingly fair shake by Schneider in 2004. We had much correspondence while Clim Chg was deciding on a data policy, much of which was amiable fencing. You run into some skilled defenders of indefensible positions (Karl Ziemelis of Nature is like that), whose casuistry has a certain flair that appeals to me.
The other part of the background pertains to the IPCC 4AR process. While realclimate had issued several denunciations of us and these denunciations assuaged most climate scientists, nothing of this type from the Hockey Team had appeared in the peer reviewed literature (the prior MBH submission not making into print).
So from an IPCC 4AR point of view, as Luboà’¦à⟠Motl has pointed out, we were still “state of the art” — hardly where IPCC WG1 wanted things. While they may have said that they had “moved on”, it would be hard for IPCC 4AR to say nothing about MBH98 and simply pretend that it had never happened.
By this time, the deadline for IPCC 4AR to be in press was looming up — December 2005 was the cut-off. It’s not far-fetched to think that IPCC WG1 was pretty anxious that Ammann and Wahl make the cut-off for inclusion in IPCC 4AR.
For some time before this, Ammann had said that they had “exactly” replicated MBH98 and, on May 11, 2005, the date of one of my rare public appearances, UCAR issued a national Media Advisory saying that Ammann and Wahl had shown that all our claims were “unfounded”.
This “Media Advisory” was quickly inhaled into IPCC processes. Sir John Houghton cited it to a Senate Committee to show that we had been refuted; Mann mentioned it about 6 times in his letter to the Barton Committee; even the European Geophysical Union cited this to the Barton Committee. Although the article was unpublished, Ammann and Wahl sent it to IPCC 4AR (described there as Wahl and Ammann 2004, J. Clim) in the same version as we were asked to review. IPCC 4AR in turn relied on Ammann and Wahl to launder MBH98.
A bunch of things strike me on re-reading our review from last June.
First, I had been able to reconcile our code to A&W code almost immediately (I reported in real time on climateaudit), so I knew that they had got the same adverse verification results as we had obtained. As far as I was concerned, it was completely unacceptable both for Ammann and for UCAR to issue a press release that all our claims were “unfounded” knowing that our claims about verification statistics were true. Accordingly, in my capacity as a reviewer, I requested this information.
I’ve posted up the correspondence to prove that the information was requested and then refused. In our review, I made very severe criticisms of this refusal. At a minimum, it was a flouting of Climatic Change data policies that had been established during a similar process with Mann. But it also evidenced that the authors knew that the results were bad and were withholding the data. Since the revised version responded to these criticisms, obviously my criticisms were not taken as being unwarranted. I don’t think that the arguments in the new sections are any good, but that’s a story for a different day.
As far as IPCC 4AR goes, there’s a huge difference between the revised version and the unpublished version sent to IPCC 4AR — the revised version, however unwillingly, vindicates our claims that MBH98 verification r2 and other statistics fail and has longwinded excuses that look like special pleading. There was not a whiff of that available to IPCC 4AR.
Second, I objected strenuously to the many misrepresentations, especially concerning “our” reconstruction. Here nothing has changed in the Revised A&W. I made point after point in our review about misrepresentations. Notably, we categorically denied that we had ever “presented” an alternative reconstruction, showing explicit statements as far back as the time of MM03. Even realclimate has recognized this. Nonetheless, they spend pages arguing over a straw man, arguing that “MM” reconstructions (which are just demonstrations of MBH98 without bristlecones or with centered PCs or such) fail a verification RE statistic. For us the salient point was that MBH98 failed verification r2 (as did the non-bristlecone variation). We were merely illustrating non-robustness; statistical verification of a variation merely illustrating non-robustness is pointless (a position also implicitly taken in BàÆà⻲ger and Cubasch). We objected vociferously to the misrepresentations. I further objected to them when I met with Ammann in San Francisco. But hardly a comma has changed in response.
Third, their Response Letter of June 10, 2005, the one in which they refused to disclose the adverse verification r2 statistic, concluded as follows:
Concerning whether RE is "at issue" The requester mentions that the RE statistic is at issue, a claim that Dr. Ammann and I have shown is made moot by the results of our indirect tests in ms #3321. In addition, Dr. Ammann and I have shown in other material referenced in mss. #3321 that the analysis of McIntrye and McKitrick in GRL (2005)–which claims RE significance levels are improperly determined by Mann, Bradley, Hughes–is itself deeply flawed. Thus, the argument in the request is incorrectly put in this regard, and it also ignores that we do use an entirely separate statistic–the deviation from verification period mean.
The “other material [by Ammann and Wahl] referenced in mss #3221” was their concurrent submission to GRL. We pointed out that we had been informed on June 6, 2005 that this article had been rejected by GRL and that, in any case, their GRL submission contained no discussion of RE significance levels, much less a demonstration that our results were “deeply flawed”. As you will see, I called a spade a spade in the review.
In correspondence in June 2005, Schneider mentioned that sometimes an impasse arose between reviewers and authors, in which case, Climatic Change might issue a “springboard editorial”. While our review was severe, it was, for the most part, expressed in objective terms and in copious detail. Although this presumably created an impasse, I heard no more from Climatic Change. Later I noticed that A&W was “provisionally accepted” in December 2005 and recently we heard that it was finally accepted. While some comments were implemented in the revision, most comments were simply ignored.
Someone has pointed out that Ammann and Wahl were “provisionally accepted” by Climatic Change about 1 day before the IPCC deadline expired — was that a coincidence? Seems unlikely to me, but you never know. An interesting issue arises from the big differences between the version accepted by Climatic Change and the version on file at IPCC WG1. IPCC WG1 reviewers would be unaware that MBH98 failed verification statistics, as we had alleged. Did the A&W version as it existed in December 2005 meet IPCC policies for being finalized? An interesting question.
I’m as aware as the next person that there’s a controversial aspect to all of this. But, for example, our relations with non-Hockey Team authors, Huybers and von Storch and Zorita are completely cordial, even though we disagree with them. I had pleasant dinners in San Francisco with Huybers and with Zorita; von Storch congratulated me when we met in Washington for what we were doing. I’d rather have comments than no comments. It’s the sludge that bugs me.
So when we come to responding to Ammann and Wahl , we have to still wade
through all the misrepresentations contained in this “peer reviewed” article. The r2-RE arguments were not there before, are a form of response to our review and I’ll discuss them separately.
As to the rest, while we have a controversial interest, I think that every point about mischaracterization and misrepresentation is objectively verifiable and could have and should have been sorted out last summer in editing rather than simply adding to more controversy. (In my opinion, one of the reasons that Saiers rejected their GRL submission was a similar difficulty in wading through misrepresentations, every one of which we were contesting.)
Why weren’t these matters ironed out? Did Climatic Change make any attempt whatever to resolve these issues? What were those attempts? It looks to me like they did nothing. I really can’t think of a whole lot to add in respect to the rest of the article to what we said in our review of July 2005. I wish that at least a little of it had been dealt with back then.
The new material about r2 and RE, probably added since December after the "provisional acceptance" , looks very weak to me. There are no third-party statistical sources. Their only statistical reference is Wahl  and Lytle and Wahl , articles on pollen, not time series – hardly imposing foundations to build their theory of verification statistics. in our review, we listed many statistical references that needed to be considered. None of them are referred to. Who reviewed this stuff? Did anyone review it? (Look I’m cynical – the MBH Corrigendum was never peer reviewed.)
Ammann and Wahl  should have been cleaned up last summer to extract whatever points did not apply to straw men. I gave some concrete suggestions on how this could be done. So what do Climatic Change peer reviewers do?