Julien Emile-Geay from Judith Curry’s university – who, together with Kim Cobb, is teaching a course on the Hockey Stick – has joined our debate with a forceful criticism of Craig Loehle’s recent paper. While Emile-Geay seems to be a lively young man with some very cordial comments about CA here and his comments are very welcome here, his initial skate around the hockey rink seems surprisingly unreflective about the defects of the canonical Hockey Team studies. Whatever validity his points have against Loehle, all too often they apply even more forcefully against Team articles, none of which are criticized. A little more attention, shall we say, to the “beam in his own eye” or at least of his teammates.
Emile-Geay asks of Loehle:
Where are the CE, RE, and most importantly R-squared statistics that are so dear to ClimateAuditers ? How are we supposed to guess whether the reconstruction has any skill ?
I agree with this 100%. These statistics are part of the game and should be provided. While I think that these statistics have to be very carefully assessed and that the risk of spurious RE statistics is not understood by climate scientists at all, I agree that readers are entitled to such information about any proposed reconstruction presented as a positive alternative.
But the more interesting issue in this demand is surely not the performance of the Loehle reconstruction, but the dissonance between Emile-Geay’s demand for a verification r2 statistic from Loehle as compared to past contortions by Mann (and Ammann) in trying to cover up the MBH verification r2 failure.
As we speak in November 2007, Mann has never reported the verification r2 (or CE) statistics for any MBH98-99 steps prior to the AD1820 splice. You can confirm this by examining the original MBH98 SI where the RE is reported but not the verification r2. On the other hand, MBH98 Figure 3 shows a map reporting the AD1820 verification r2 results – a step for which results were favorable, unlike the earlier AD1400 and AD1000 steps. MBH98 also explicitly says that verification r2 results were considered and IPCC TAR states that the Mann reconstruction had “skill” in verification statistics without limiting this to the RE statistic.
Mann has never actually reported the reconstructions for the individual steps, including the AD1400 or AD1000 steps, forcing any interested readers to run the gauntlet of trying to replicate his study from scratch in order to obtain the elementary statistics said here by Emile-Geay to be a necessity for any study (and a demand with which I agree.) I have been trying to obtain Mann’s actual result (or equivalently the residual series) for the AD1400 step since 2003 without any success in order to carry out the verification tests demanded here by Emile-Geay. In 2003, Mann refused. I asked the National Science Foundation and they refused. I filed a Materials Complaint to Nature and they refused, saying that Mann was not required to produce the results of his “experiments”; it was up to him.
In 2004, I acted as a reviewer for a submission by Mann to Climatic Change, supposedly rebutting our EE 2003 article. In my capacity as a reviewer, I again asked for this information and again Mann refused. Ultiumately Mann withdrew the article rather than providing the data (although the rejected article is check kited in Jones and Mann 2004.)
In MM(2005 GRL and 2005 EE), we observed that the verification r2 for the AD1400 step under consideration there was ~0. In MM (2005 EE), we observed that it seemed inconceivable that Mann had not calculated the verification r2 statistic (as indeed he had as evidence by his source code).
This issue has prompted much subsequent controversy. The House Energy and Commerce Committee asked Mann, Bradley and Hughes whether they had calculated a verification r2 statistic and asked what it was. Even in response to a congressional inquiry, Mann refused to provide the verification r2 statistic. He did provide code and the code shows for certain that he calculated a verification r2 statistic in the same source code step as the RE statistic – observed in summer 2005 at CA.
The National Academy of Sciences specifically referred to the verification r2 issue when they wrote their complaint letter to the House Energy and Commerce Committee. In our presentation to the NAS panel, we summarized the verification r2 issue as it then stood. In the NAS panel presentations, a NAS panelist asked Mann whether he had calculated a verification r2 statistic for the AD1400 step and what was it. Mann famously denied calculating it, saying that that would be a “foolish and incorrect thing” to do – notwithstanding the fact that his own source code, produced for the House Energy and Commerce Committee and the MBH98 figure for the AD1820 step, showed that he had calculated the statistic: he merely didn’t report it for the adverse steps.
As an anonymous reviewer of Wahl and Ammann, I asked the authors to report the verification r2 statistic for their calculations (fully knowing that the results were zero). They refused, while, at they same time, they issued a press release stating that all our claims were “unfounded”. I’ve reported previously on my proposal to Ammann in San Francisco in Dec 2005, proposing a joint paper itemizing points that we agreed on, points that we disagreed on, knowing that our codes fully reconciled (while neither fully reconciled to Mann’s); he said that this would interfere with his career advancement. I also urged Ammann to report the verification r2 results telling him that he seemed like a nice young man but that I would not stand idly by if he failed to report the adverse results; he still said that he would not report the verification r2 statistiis. I filed an academic misconduct complaint and, in late February 2006, the adverse verification r2 results were disclosed in an Appendix to the revised Wahl and Ammann, fully confirming our previous results (although Wahl and Ammann did not credit us with priority or acknowledge that they had confirmed our earlier findings).
The Wahl and Ammann preprint came online a couple of days after the NAS panel hearings. In a supplementary letter to the NAS panel, we alerted the NAS panel that the revised Wahl and Ammann had confirmed the adverse verification r2 scores, and the NAS panel noted these failed verification r2 results in their report.
While I fully agree that Loehle should have reported the verification r2 statistics for his reconstruction (and I would be surprised if they were any better than the results for MBH or other Team studies), it is extremely hypocritical (and all too characteristic of Team climate science) for Emile-Geay to criticize Loehle for this omission given the history of obstruction on this matter by Mann and Ammann. If Mann wouldn’t provide this information to the NAS panel even when asked directly, shouldn’t that (and related ) refusals have occasioned Emile-Geay’s disapproval long before his opprobrium against Loehle’s omission of this statistic (an omission which should be corrected).
BTW I’ve calculated the Loehle performance statistic relied on exclusively in Juckes et al 2007 – the 1856-1980 r. Loehle’s recon has an 1856-1980 r of 0.594, which matches or exceeds the 1856-1980 r reported by Juckes for several CVM variations (MBH .535, Esper 0.599, Jones et al 1998 0.367). It is my view (expressed in my review of Juckes et al ) that his claim that these correlations were 99.99% significant was absurd, because trivial variations with different medieval-modern relations also had 99.99% significant 1856-1980 correlation (now including the Loehle reconstruction). Juckes was not required to even respond to that criticism (in breach of Climate of the Past policies), but the issue is surely back on the table.