In a recent post, I showed that MBH had calculated cross-validation R2 statistics, but this information had been excluded from their summary of cross-validation statistics in their Supplementary Information here. We had surmised this in our original article, but had previously been unable to absolutely prove it. The House Committee requested information from M,B and H about whether they calculated the R2 and other cross-validation statistics. It’s interesting to look at their responses with this information in mind.
The question to all three was as follows: (question 7 to Mann, question 6 to the others):
7. Provide a detailed narrative explanation of these alleged errors and how these may affect the underlying conclusions of the work, including, but not limited to answers to the following questions:
c. Did you calculate the R2 statistic for the temperature reconstruction, particularly for the 15th Century proxy record calculations and what were the results?
d. What validation statistics did you calculate for the reconstruction prior to 1820, and what were the results?
Hughes does not provide a sequential answer to the questions sent to him by the House Committee. Instead, he writes a discursive letter, selectively replying to questions, but nowhere answering these questions.
Bradley does provide a sequential letter, but his response to his question 6 likewise does not answer either of these questions. Instead, he editorializes on journal review procedures and cites a realclimate posting. The cited realclimate post does not contain any information about whether MBH calculated the R2 and other cross-validation statistics. I’ve responded to the issues in this realclimate posting in posts Errors Matter #1, #2 , #3 and here, as well as elsewhere on this blog
So now to Mann. As usual, Mann is not short of verbiage. The answer to 7C essentially re-states a position previously put forward at realclimate here , in which Cook et al  as well as Luterbacher et al  were cited. I provided a reply to this here, noting ironically that Cook et al  used an R2 statistic . Cook et al is not cited to the House Committee. So Mann’s down to one citation.
The Wilks’ citation is also interesting. If you look at equation 7.20 in Wilks, the RE statistic is necessarily lower than the R2 statistic if the series meet stationarity conditions required to establish a linear relationship. At most Wilks says that an R2 statistic is not sufficient (not that it is not necessary). If you watch Mann’s argument, he switches from sufficient conditions to necessary conditions with a conjurer’s skill. For example, he’ll find a text saying that an R2 statistic is not sufficient for statistical significance (a point with which I agree) and then say that this shows that an R2 statistic is not necessary for statistical significance — a point with which I disagree and which he’ll have a hard time finding any reputable statistical authority outside the climate science community to endorse. I’ll re-visit these issues on another occasion.
Here I want to focus on his answer to 7D (which is avoided by Bradley and Hughes), Mann says:
A(7D): The Committee asks “[w]hat validation statistics did you calculate for the reconstruction prior to 1820, and what were the results?” Our validation statistics were described in detail in a table provided in the supplementary information on Nature’s website accompanying our original nature article, Mann, M.E., Bradley, R.S., Hughes, M.K., Global-Scale Temperature Patterns and Climate Forcing Over the Past Six Centuries, Nature, 392, 779-787 (1998). These statistics remain on Nature’s website (see http://www.nature.com/nature/journal/v392/n6678/suppinfo/392779a0.html) and on our own website. See ftp:holocene.evsc.virginia.edu/pub/Mannetal98.
First if you go to the Nature website http://www.nature.com/nature/journal/v392/n6678/suppinfo/392779a0.html and follow the links, you will find that the statistics are deleted (I pointed this out a few months ago.) I continue to be blocked from Mann’s FTP site at UVA, but my recollection is that these statistics are not at this website. (I’ll verify and edit this point.) However, the statistics do still maintain a precarious existence at the original Umass FTP site : ftp://eclogite.geo.umass.edu/pub/mann/ONLINE-PREPRINTS/MultiProxy/stats-supp.html.
As I pointed out yesterday, the cross-validation R2 statistic is not included in this table, but was definitely calculated in Mann’s recently released source code, as was the Nino RE statistic (for which only the R2 statistic was reported). So Mann has given an incorrect and highly misleading answer to the House Commitee on a point that is not incidental, but at the heart of the matter. Mann sure seems to be playing with fire.