Mann et al. have responded to our criticism by claiming that the errors which we have identified “don’t matter” because they can “get” MBH-type results under several different methods, one of which is through not using any PCs. Ross and I previewed an initial reply to these arguments here and plan to issue a pdf version of our reply. I’ll amplify our earlier discussion here, starting with Mann’s no-PC salvage proposal. For a variety of reasons – abandonment of any pretence at even spatial sampling, non-robustness to bristlecone pines and lack of statistical skill over a range of verification statistics – the no-PC reconstruction fails to salvage MBH98.
Here is a typical statement of how the Hockey Team presents this argument:
We quickly recap the points for readers who do not want to wade through the details: i) the MBH98 results do not depend on what kind of PCA is used, as long as all significant PCs are included, ii) the results are insensitive to whether PCA is used at all (or whether all proxies are included directly), and iii) the results are replicated using a completely different methodology (Rutherford et al, 2005).
This hyperlink restates arguments made at realclimate on Dec. 4, 2004 even before our papers were released and underpins public statements by realclimate coauthors Gavin Schmidt and William Connolley.
Representations and Warranties: Now that the errors of their PC methodology are being understood, it is my view (and it seems self-evident to me) that it is insufficient for Mann et al. to merely "get" a hockey stick shape some other way – they have to do so while continuing to achieve the representations and warranties of MBH98, which led to the widespread acceptance of this study. Among the most important such representations are the following:
1. “a reasonably homogeneous spatial sampling in the multiproxy network was achieved by representation of ‘densely sampled regional dendroclimatic data sets’ through principal components analysis ensured” (p. 779);
2. “the long-term trend in NH [temperature] is relatively robust to the inclusion of dendroclimatic indicators in the network” (p. 783 and re-stated in Mann et al. , A Further Note)
3. the MBH98 network attained a “high level of skill …in large-scale reconstruction back to 1400″ (p. 785). Recently, Mann has amplified this by saying that studies which do not satisfy a number of “statistical verification exercises” should not be considered in climate studies .[….]
4. the proxies were selected according to “objective criteria” [Mann et al., 2000]
5. the proxies are linearly related to large-scale climatic patterns, and are neither local, non-linear nor non-climatic. (p.780)
Let’s look here at the consistency of the no-PC alternative with the first three MBH98 representations.
Even Spatial Sampling: : In the 1400-1450 period of the MBH98 Northern Hemisphere temperature reconstruction, “reasonably homogeneous spatial sampling” was achieved by representing 76 North American tree ring chronologies by 3 PC series. Without PC (or other representation method), North American tree ring series would account for 80 of 95 series in this step (of which 20 were bristlecone pine ring width chronologies) – which would have been an obvious failure of "reasonably even spatial sampling". Instead, through PC representation, MBH98 temperature calculations in the regression steps were done with 22 proxy series, of which 7 were still based on North American tree ring networks (2 PC series from the North American network, 1 PC series from the Stahle/SWM network, the extrapolated Gaspé series and 3 Stahle precipitation reconstructions from tree ring chronologies.) Even after PC representation, 7 of 22 series (32%) used in temperature calculations were based on North American tree ring data – period, which we believe already strains the limits of “reasonably homogeneous spatial sampling”. A calculation with 80 of 95 proxies being North American tree ring series would be a total abandonment of the premise of reasonably even spatial sampling. Had such a reconstruction been proposed in the first instance, we do not believe that it would have been taken seriously or could have been used to ground major policy initiatives. We see no reason why it should be taken seriously now.
Robustness: this salvage reconstruction does not deal with the issue of non-robustness to bristlecone pines. If the bristlecone pines are removed from this network, there is no hockey stick even under a no-PC method. Robustness was an important representation for the acceptance of MBH98; the no-PC salvage alternative needs to meet this representation and it does not.
Statistical Skill: Mann claimed great statistical skill for MBH98, a claim which we dispute in MM05(GRL) but which was essential in MBH98 aceptance. Mann has recommended the following for a series which lacks "skill":
[it] fails statistical verification exercises, rendering it statistically meaningless and unworthy of discussion in the legitimate scientific literature
Mann does not provide a suite of verification statistics or digital information about the no-PC reconstruction, but only the RE statistic for two different versions of the no-PC reconstruction (0.39; 0.33). From related calculations, our surmise is that the R2 statistic for the no-PC reconstruction (like for MBH98 itself in this period) will be approximately 0.0 and not statistically significant. Contrary to claims by Mann et al., we do not argue that analysts should exclusively look at the R2 statistic and not look at the RE statistic; we’ve argued that they should look at both of these statistics (and others) — hardly a controversial point. In MM05 (GRL), we provided a benchmark distribution of RE statistics under the data mining methods used in MBH98. The RE statistics of the no-PC salvage reconstruction are only slightly above the median. Taken together with the statistically insignificant values of other verification statistics, it is obvious that the salvage reconstruction fails “statistical verification exercises” of realclimate. Mann has stated in arguments against us that a reconstruction which fails "statistical verification exercises" should not be considered in climate science. What’s sauce for the goose is sauce for the gander.
Thus, for a variety of reasons – abandonment of even spatial sampling, non-robustness to bristlecone pines and lack of statistical skill over a range of verification statistics – the no-PC reconstruction fails to salvage MBH98. I will discuss the consistency of the other salvage alternatives with MBH98 representations in the next few days.