Climate Audit

A couple of weeks ago, the New York Times and other institutional media proclaimed that “tree rings” (sometimes “ancient tree rings”) had “shown” that 2023 was the warmest summer in 2000 years (link link) Almost 25 years to the day since they had similarly proclaimed that 1998 was the warmest year in 1000 years.   The recent proclamation, as in 1998, was based on an article in Nature (Esper et al, 2024); the proclamation in 1998 was similarly based on an article in Nature (Mann et al, 1998), together with its companion article in Geophysical Research Letter (link).

A “Confidence Interval” Trick

In addition to the similarity of the conclusions to the two article, the structure of the money diagram in both cases is almost identical, as shown in the comparison below.

Figure 1. Left – On the left is the money diagram from Mann et al (1999)  showing confidence intervals from the Mann reconstruction in light grey, the reconstruction in black. The reconstruction itself only went to 1980. The instrumental temperature (as an “anomaly”) for 1998 was shown as a point. A horizontal line was then drawn across the diagram to show that the 1998 instrumental temperature exceeded the confidence interval for all prior dates of the reconstruction – hence the “warmest year in 1000 years”.  Right – the Esper et al 2024 reconstruction only went to 2010.  The point estimate for 2023 exceeds the confidence intervals for all prior years.

Each diagram shows purported confidence intervals around the reconstruction in light/medium gray. Each diagram also denotes a the recent instrumental estimate featured in the headline as a highlighted point.  In each case, the highlighted instrumental point is more than 10 years after the end of the reconstruction. In each case, the conclusion is obtained by observing that the instrumental point is higher than any of the upper confidence limits of the corresponding reconstruction.  In other words, both Esper et al (2024) and Mann et al (1998-99) used the same technique.

In Climategate controversy, there was voluminous discussion of the term “Mike’s Nature trick”, but, to my recollection, there was little to discussion of the term in relation to the use of confidence intervals to arrive at a “warmest year” conclusion.

In his notorious Climategate email, Phil Jones described “Mike’s Nature trick” as the splicing of proxy and instrumental temperatures that he had carried out for the World Meteorological Organization.  Mann had indeed spliced proxy and instrumental temperatures for the calculation of the smoothed reconstruction illustrated in the article, but had cut the smoothed version back to 1980 (the end of the proxy data.)  Mann vehemently denied that the splicing of proxy and instrumental data was “Mike’s Nature trick” and instead claimed that Mike’s Nature trick was nothing more than showing an estimate (reconstruction) and actual (observed temperature) on the same figure, clearly marked.  But this is such a benign and commonplace statistical practice that it cannot be reasonably described – even by the statistically and mathematically challenged Jones – as a “trick” in the mathematical sense.  A mathematical “trick” implies ingenuity or novelty, but showing estimate vs actual is trivial and commonplace.

However, the technique shown above – comparing a point estimate in a year without proxy data to the confidence envelope of the proxy reconstruction – was a novelty introduced by Mann and, in that sense, an actual candidate for the term “trick” (sensu mathematics) as opposed to the commonplace and trivial comparison of estimate and observed on the same figure.

The validity of this comparison (for Mann et al 1998-99 and Esper et al 2024) depends not just on the validity of the reconstruction, but the validity of the confidence intervals.  

Some Comments on Mann et al Confidence Intervals

Mann et al went to considerable lengths to obfuscate analysis of their confidence intervals and, indeed, key information related to that calculation was discovered within the last year – see link.  Mann’s reconstruction was done in 11 steps, with each step having a different reconstruction. The purported confidence interval for each step was twice the standard error in calibration period for that step.  However, to this day, Mann never archived the results of each step, and, to its shame, Nature refused a request that Mann be required to archive the results for each step.  Mann had also refused to provide us with the residuals that had been used to calculate the standard errors. (However, he did provide residuals to Tim Osborn in a Climategate email and this data became available as a result of Climategate.)  Further complicating analysis, while Wahl and Ammann and ourselves could replicate one another’s results, neither of us could replicate Mann’s results.  In the last couple of years, more than 20 years after the original study, Swedish engineer Hampus Soderqvist, by reverse engineering, finally accomplished an exact replication of Mann’s reconstruction – it turned out that Mann’s list of proxies used in early steps was inaccurate: several listed proxies were not actually used, and several unlisted proxies were used.

One of the long standing (and, unfortunately, under-appreciated) issues with the Mann et al 1998-99 reconstruction is overfitting in the calibration. Due to a sort of inverse regression of instrumental temperature on large networks of proxies (a variation of Partial Least Squares). If there is overfitting in the calibration period, the standard error in the calibration period will be artificially small.  To get a more realistic estimate of confidence intervals, one needs to use the standard error in the verification period.  There is a direct relationship between the calibration period standard error and calibration period r2: a high r2 statistic necessarily means a small calibration standard error and vice versa.

If one is concerned about calibration period overfitting, the verification period r2 statistic is a simple and effective test: if there is a high calibration r2 and dismal verification r2, then there has almost certainly been statistical overfitting and a flawed model.  In addition to withholding the results of the individual steps, Mann et al 1998-99 concealed extremely low verification r2 statistics.

In the recent Mann-Steyn libel trial, the DC judge blocked both McKitrick and myself from presenting any technical evidence on Mann’s failed verification statistics or Mike’s Nature trick, even refusing to allow the defense to show a table of failed verification statistics that had been published in Geophysical Research Letters.

These controversies have been known for some time, but they connect directly to the confidence interval trick. Had the “money diagram” been calculated with standard errors from verification period residuals, the confidence interval envelope would almost certainly have been at least double the size of the confidence envelope shown in MBH98 and MBH99.  (Late last year, Soderqvist made exact stepwise MBH results available for the first time. Re-doing the diagram with verification period standard errors would be a useful exercise.)

Some Comments on Esper et al Confidence Intervals

Needless to say, there are many issues with both the Esper et al 2024 and with its associated “confidence” intervals.  But before getting into the details, I recommend that interested readers re-examine a truly excellent 2012 article by Esper, Buentgen and coauthors also in Nature (link), an article on which I favorably commented many years ago (link).  Esper et al (2012) observed that tree ring chronologies (black below) dismally failed to record huge millennial-scale change in the high-latitude Northern Hemisphere summer insolation (and temperature), showing the following diagram in their SI.

Figure S1. Temperature trends recorded over the past 4000-7000 years in high latitude proxy and CGCM data. Multi-millennial TRW records from Sweden 1 , Finland2 , and Russia 3 (all in grey) together with reconstructions of the glacier equilibrium line in Norway4,5 (blue), northern treeline in Russia 3,6 (green), and JJA temperatures in the 60-70°N European/Siberian sector from orbitally forced ECHO-G 7,8 (red) and ECHAM5/MPIOM9 (orange) CGCM runs10. All records, except for the treeline data (in km) were normalized relative to the AD 1500-2000 period. Resolution of model and TRW data were reduced (to ~ 30 years) to match the glacier data.

Esper et al (2012) observed that summer insolation at 50N decreased by more than 35 wm-2 (!!!) since the early Holocene  and 6 wm-2 since Roman times – vastly more than the forcing of ~1.5 wm-2 associated with increased CO2 since pre-industrial times. They identified changes in the northern treeline at Yamal (green) and changes in the equilibrium altitude of a small ice cap in Norway (blue) as two proxies that were responsive to large-scale millennial-scale changes in insolation and high-latitude summer temperatures.

Conclusion

Whether or not the comparison of an observed temperature point to the confidence envelope of a reconstruction to draw conclusions about “warmest year in 1000 years” was precisely what either Mann or Jones defined as “Mike’s Nature trick”, it can be fairly described as a trick (sensu mathematics), whereas plotting an estimate and observed on same figure is so commonplace and trivial that it cannot reasonably be described as a trick (sensu mathematics.)

In that spirit, I think that it is fair to describe “Mike’s Nature trick” (and the similar trick employed by Esper et al 2024) as a confidence trick.  In the mathematical sense, of course.

As a caveat, readers should note that the question of whether tree rings (ancient or otherwise) show that 2023 (1998) was the warmest summer (year) in 1000 or 2000 years is a different question than whether 2023 was the warmest summer in 1000 years.  My elevator take is

1. that 20th and 21st century warming are both very real, but that the 19th century was probably the coldest century since the Last Glacial Maximum and that the warming since the 19th century has been highly beneficial for our societies – a view that was postulated in the 1930s by Guy Callendar, one of the canonical climate heroes;
2. per Esper et al 2012, given the failure of tree ring chronologies to reflect major millennial-scale changes in summer insolation and temperature, what possible reliance can be attached to pseudo-confidence intervals attached to 2000-year tree ring chronologies in Esper et al 2024 (or any other tree ring chronologies)
3. in addition, we know that there is global-scale “greening” of the planet over the past 30-40 years that has been convincingly attributed to enhanced growth due to fertilization by higher CO2 levels. So, in addition to all other issues related to tree ring chronologies, it is necessary to disaggregate the contribution of CO2 fertilization from the contribution of increased warming – an effort not made by Esper et al 2024 (or its references.)

In a follow-up article, I will examine details of the Esper et al 2024 reconstruction, which, among other interesting features, connect back to Graybill bristlecone sites and the Briffa sites under discussion in the period leading up to the Climategate emails.