Climate Audit

McShane and Wyner is being published as a “discussion paper” and has attracted numerous submissions so far, including a discussion by Ross and I which has been accepted. As readers have noticed, discussions by Schmidt, Mann and Rutherford and by Tingley are online. Other submissions have been made by Wahl and Ammann and by Nychka et al.

Too much discussion so far has focused on whether or not Mc-W’s short history of the dispute has captured all the nuances of the dispute. I presumed that they were trying to merely trying to set the table in a few pages, but, needless to say, this short history has attracted controversy (which I’m not going to dwell on today.) It’s too bad that climate scientists have focused so much on this minor aspect of the paper. Most of the rest of the attention has been paid to their “own” reconstruction later in the paper, about which I will comment on another occasion.

Today I’m going to comment about the analyses in their sections 3.2 and 3.3, which are, in effect, an extended commentary on benchmarking the RE statistic – though few, if any, climate scientists seem to have grasped this point thus far, in part, because MW do not clearly link their analysis to this issue, though the relationship is quite clear.

MW observe that the climate science practice of assuming that the “proxies” are a “signal” plus low-order AR1 noise is not one that is supported by observed AR1 coefficients – see their Figure 4. This is a point familiar to CA readers. They observe that pseudoproxy tests limited to low-order AR1 noise insufficiently replicate observed proxies and conduct more testing simulations using “empirical AR1” coefficients and brownian motion (random walk) pseudoproxies.

Their test template is to calculate holdout (i.e. verification) RMSE (root mean square error) statistics using the lasso multivariate methodology, summarizing their results in their Figures 9 and 10. The denominator in an RE calculation is proportional to the square of the RMSE of the “in-sample mean”, which they also call the “intercept” model. Thus, the RE of a proxy reconstruction is directly linked to the RMSE of the proxy reconstruction and the in-sample mean RMSE_{intercept} as follows:

RE_{proxy}= 1- holdout_RMSE_{proxy}^2/ holdout_RMSE_{intercept}^2

The benchmarking against a class of pseudoproxies is calculated from the distribution obtained from simulations:

RE_{pseudoproxy}= 1- holdout_RMSE_{pseudoproxy}^2/ holdout_RMSE_{intercept}^2

In their Figure 9, if the RMSE boxplot for a class of pseudoproxy is better(smaller) than the RMSE boxplot for the in-sample mean (intercept), then the RE statistic for the proxy reconstruction will not prove “significant” against that class of pseudoproxy.

They make the surprising observation that white noise pseudoproxies outperform low order red noise (the usual climate science benchmark) under their test. More controversial claims arise from their observation that “empirical AR1” pseudoproxies and Brownian motion pseudoproxies( random walks) outperform actual proxies under their test setup. (This can be seen in the smaller RMSEs in the figure shown below.)

The benchmarking of RE statistics was an issue that was put into play in McIntyre and McKitrick 2005a, where we observed that you could get high RE statistics from pseudoproxies with autocorrelation coefficients mimicking the autocorrelation coefficients of actual proxies, rather than the very low-order AR1 coefficients assumed (without proof) by climate scientists. While MW cite us favorably on the concept of “empirical AR1” pseudoproxies, they unfortunately don’t directly link their figures to the issue of RE benchmarking as clearly as they might have – confusingly tracing the issue back to von Storch et al 2004 (which addressed a different issue), rather than McIntyre and McKitrick 2005a,c.

They did their simulations using a multivariate method ( the “lasso”), familiar in their community, but not actually used in paleoclimate. They faced a tradeoff here – between analysing the properties of the proxy network using a methodology known to the outside world (as opposed to today’s version of RegEM or CPS). I think that they’d have been better off doing a form of CPS, as it would have removed an obvious criticism. Having said that, my own surmise is that the relative performance of Mann08 proxies versus pseudoproxies will not be particularly sensitive to the use of CPS versus lasso. I just don’t see that the weighting patterns from the lasso are going to be different enough to radically change the results.

They accurately note that the Team responded vehemently against the concept of “empirical AR1” coefficients in Ammann and Wahl 2007. The battleground issue here is whether the high AR1 coefficients observed empirically are properties imparted to the “proxies” by the climate system or whether they are because of gross problems with the proxies. Ammann and Wahl (2007) has argued that using autocorrelation coefficients estimated from actual proxies results in

“train[ing] the stochastic engine with significant (if not dominant) low frequency climate signal rather than purely non-climatic noise and its persistence”.

This was asserted rather than demonstrated and, as we know, assertion suffices far too often in climate science. It seems evident that if the proxy networks contained a “dominant” or even “significant” “low frequency climate signal” (as Ammann and Wahl assert), then the graphs of the proxy series would have a consistent low frequency appearance (as opposed to the visually inconsistent appearance shown in MW Figure 6 and elsewhere.) Ammann and Wahl do not explain this. The very inconsistency of the series within proxy networks such as Mann et al 2008 argues forcefully against the interpretation of high empirical autocorrelation coefficients as being imported from a climate “signal”, as opposed to being an inherent feature of the proxies themselves.

So while I expect that climate scientists will argue against “empirical AR1” coefficients as too severe a pseudoproxy test, I, for one, do not think that “empirical AR1” coefficients are too severe a test – if anything, they are probably not severe enough. (Note that empirical AR1 coefficients place less structure on the autocorrelation than the hosking.sim simulations used with the NOAMER tree ring network in our 2005 simulations and simplify this aspect of this analysis.) MW makes the following additional and reasonable observation in dismissing Ammann and Wahl’s objection to empirical AR1 coefficients:

it is hard to argue that a procedure is truly skillful if it cannot consistently outperform noise– no matter how artfully structured.

I’ll discuss their “own” reconstructions on another occasion.