Climate Audit

Take a look at today’s puzzle.

On the left, I’ve plotted the three EIV reconstructions for the NH hemisphere using the infilled CRU series as a “target”. The data is straight from Mann’s website. One shows the “full” global network, one shows the “full” NH network (both of which I take to really be series selected as discussed in my Full Network post, but it’s always hard to be sure) and the third one is the “screened” NH network. As I understand it, SH proxies are used in the “GL full” network to teleconnect with NH reconstruction and this is the difference with the NH “full”, but again I’m not 100% sure.

The right panel shows the differences between the series, showing the differences in a cycle.

Take a careful look at both panels because lots of questions jump out at me.

First, all three reconstructions go to 2006 even though there aren’t any proxies that go to 2006. How’d they do that? (Of course, we can’t tell by consulting the EIV code, because, contrary to the representaton by PNAS, working EIV code isn’t provided.) Did they splice the instrumental record? Could be. It’s hard to tell.

Second, all three reconstructions are identical in the calibration period. How’d they do that? Is this from splicing or overfitting?

Third, look at the the year 600 in the right hand panel. The black and red series essentially flip orientation. How’d they do that? It looks like there’s a reconstruction step and the regression sign for a series flips over when the composition of the network changes. Whatever it is, it’s not a good sign,

Fourth, what about the green series in the left hand panel. What’s going on in the early portion? How’d they do that?

Just to show that I’m not making this up. Here’s a graphic from the SI in which I think that I discern some of these patterns:

Stranger and stranger.

The “full” network isn’t.

Mann et al 2008 describe a “full” network consisting of 1209 proxies:

We made use of a multiple proxy (‘‘multiproxy’) database consisting of a diverse (1,209) set of annually (1,158) and decadally (51) resolved proxy series … Reconstructions were performed based on both the ‘‘full’ proxy data network and on a ‘‘screened’ network (Table S1)

The locations of the “full” network are illustrated in their Figure 1, captioned “spatial distribution of full (A) and screened (B)” network. Their Figure S7 compares “long-term CPS NH land (a) and EIV NH land plus ocean (full global proxy network) (b) with and without using tree-ring data”. Their Figure S8 compares “long-term CPS NH land (a) and EIV NH land plus ocean (b) reconstructions (full global proxy network)”. Their Figure S11 shows “plots of the NH CPS Land reconstruction (A) and EIV Land plus ocean (full global proxy network)”.

Table S1 shows breakdown counts by dendro and non-dendro, by annual-resolved and total, by NH, SH and Global. The top portion of Table S1 is excerpted here:

The problem is that their actual calculations appear almost certainly not to have used the network with the count statistics shown in Table S1, but a truncated version of that network, with their “full” network using only 663 (560 – NH) of the 1209 sites (1026- NH). Jean S has run the gridproxy.m program using the FULL option and so we have been able to “prove” this for this part of the program (CPS) through an actual run of the program. (I’m not sure that the EIV portion of the program can be run with present code.)

First here is a table showing counts that compare to the top right counts (NH all) for Table S1. The column headed Matlab shows the counts obtained from the gridproxy.m program, while the column headed “Reported” shows the corresponding column from Table S1. I’ll explain the calculation of the other 3 columns below (which I calculated from first principles. For now, you can see that my NH total in the third column matches the counts from gridproxy.m exactly in nearly every case and, in no case, differs by more than 1.)

An Unreported Screening Procedure on the Full Network
The difference appears to result from an unreported screening procedure on the “full” network.

Mann’s SI does describe a screening procedure on his “screened” network:

Screening Procedure. To pass screening, a series was required to exhibit a statistically significant (P 0.10) correlation with either one of the two closest instrumental surface temperature grid points over the calibration interval (although see discussion below about the influence of temporal autocorrelation, which reduces the effective criterion to roughly P 0.13 in the case of correlations at the annual time scales)

The implementation of a procedure along these lines can be discerned in gridproxy.m in the case (confusingly) entitled “WHOLE”, but which means screened. At one stage of the program, correlation hurdles are defined for various proxy classes (using the proxy class codes to set the hurdle.) “Annual” and “decadally” resolved proxies have different benchmarks. Documentary and speleothems (5000,6000 series) have somewhat different benchmarks (corals – 7000 check for negative correlation). The relevant lines are:

case ‘WHOLE’
ia=4; %% annually resolved
ib=5; %% decadally resolved
corra=0.106; %% 9000,8000,7500,4000,3000,2000
corra2=-0.106; %% 7000
corra3=0.136; %% 6000, 5000
corrb=0.337; %% same as above +1
corrb2=-0.337;
corrb3=0.425;

Later in the program, they test the observed correlation for each proxy against the relevant benchmark (their code is very inelegant; they speak Matlab with a heavy Fortran accent). For the “screened” network, corra is 0.106.

for i=1:m1-1 % This is for searching annually-resolved proxies
if (z(3,i)==9000 | z(3,i)==8000 | z(3,i)==7500 | z(3,i)==4000 | z(3,i)==3000 | z(3,i)==2000) &…
x(kk,i+1)>-99999 & x(kkk,i+1)>-99999 &…
z(1,i)>=ilon1 & z(1,i)=ilat1 & z(2,i)=corra
n=n+1;
…
end

For corals and speleothems, they do it a little differently, testing the absolute value of the correlation:

… z(1,i)>=ilon1 & z(1,i)=ilat1 & z(2,i)=corra3

Now watch what happens in the “FULL” case. Here’s how they set their hurdles. Can you see what happens?

case ‘FULL_’
ia=4;
ib=5;
corra=0;
corra2=0;
corra3=0;
corrb=0;
corrb2=0;
corrb3=0;

If a series has a negative correlation to temperature (as about half of the tree ring series do), the test shown above will lead to the exclusion of this series. So this test either intentionally or unintentionally eliminates all the series with negative correlations to gridcell temperature – thus the reduction in NH count from 1036 to 560. Jean S got this count from running gridproxy.m and I got this number by independently comparing reported correlations to a benchmark of 0. So while it’s not impossible that I’ve misinterpreted something here, until we see some alternate explanation, I think that this interpretation should be regarded as valid.

Now this creates problems – both with the statistical significance of what was reported and with the accuracy of the methodological description in the PNAS article, both familiar themes in this corpus.

We’ve discussed on many occasions that you can “get” a HS merely from picking upward-trending series from networks of red noise (David Stockwell had a good note on this phenomenon on his blog a couple of years ago. My first experiments of this type were on the cherry picks in the original Jacoby network.) Since gridcell temperature series from the mid-19th century to late 20th century by and large have an upward trend, this test is equivalent to a simple pick operation. Whatever the merits of effectively screening the data set for upward trending data, this affects statistical benchmarks, as the null is applying the same sort of operation to red noise networks.

Secondly, whatever one may think of the Mannian PC algorithm (and defenders are reduced to a pretty hard core right now), few people defend the non-reporting of their modification of the PC algorithm and their failure to assess the statistical significance and properties of these changes. (Quite aside from whether they can “get” a HS using some other method.) We’re into something pretty similar here. Again there seems to be an unreported screening operation on the “full” network. The value of archiving code is once again demonstrated as the unreported operation can only be discerned through parsing code. Because Mann et al commendably archived code concurrent with publication, it hasn’t taken years to identify the issue. (I note that the precise identification of the problem with Mannian PCs also was identified when a little bit of code was inadvertently left in a directory identified for the public after MM03.) So code really does help pin down problems.

Let’s say that you were a partner of the firm North, Hegerl and Cicerone and charged with issuing an opinion on the financial statements of Team Capital Management Inc.(TCM) And let’s say that you were doing so in heady pre-crash days when markets were going up and mark-to-market accounting was something that the companies wanted to do.

The footnotes to the TCM statements said that mark-to-market accounting had not been used for non-arms-length investments in securities issued by Briffa MXD Inc. which had shown a lack of “sensitivity” to rising world stock market prices, while mark-to-market accounting had been used for a high-flying Finnish penny stock (Kortajarvi Lake Gold). And while TCM statements did not mention their holdings in controversial Bristlecone Estates, a project that was much in the news, it turned out that mark-to-market accounting had been used on this speculation as well.

In the MD&A (Management Discussion and Analysis), the company said that they were profitable even without their investments in U.S. mortgages. Elsewhere in the footnotes, they said that TCM was profitable even without mark-to-market accounting in the Finnish penny gold stock. However, the combined effect wasn’t discussed.

Can the partners in North, Hegerl and Cicerone certify that these statements meet GAAP? Of course they can’t.

If you’re using mark-to-market accounting on the penny gold stock that went up, then you have to use mark-to-market accounting on the shares in Briffa MXD Inc that went down. While disclosure is important, you can’t disclose your away around this sort of inconsistent non-GAAP accounting. You can’t use mark-to-market accounting on stocks that went up and not do so for stocks that went down.

In case this parable seems too harsh, here are the exact statements from Mann et al 2008. Obviously the precise issue of “mark-to-market” accounting doesn’t arise: I use this as a parable for inconsistent accounting for data that goes up as opposed to data that goes down,

Accounting for MXD
Mann et al say of their handling of the Briffa MXD series (which “diverge” from rising world markets):

Because of the evidence for loss of temperature sensitivity after ~1960, MXD data were eliminated for the post-1960 interval.

In 1998, Briffa postulated an unknown anthropogenic cause for the post-1960 downturn, but nothing has turned up 10 years later. An open alternative is that there is a non-linear upside-down U-shaped temperature response, an alternative referred to even in IPCC AR4 as follows:

Others, however, argue for a breakdown in the assumed linear tree growth response to continued warming, invoking a possible threshold exceedance beyond which moisture stress now limits further growth (D’Arrigo et al., 2004). If true, this would imply a similar limit on the potential to reconstruct possible warm periods in earlier times at such sites. At this time there is no consensus on these issues (for further references see NRC, 2006)

An opinion that you’d think that the auditing firm of North, Hegerl and Cicerone would be aware of.

In respect to the accounting of Mann et al, they truncated this data merely because of a potential non-climate anthropogenic impact on the data, an impact which is postulated, but not proven.

Accounting for Finnish Speculative Stocks
The Mann et al 2008 footnotes say of the speculative Finnish sediments:

These records include the four Tijander et al. (12) series used (see Fig. S9) for which the original authors note that human effects over the past few centuries unrelated to climate might impact records (the original paper states ‘‘Natural variability in the sediment record was disrupted by increased human impact in the catchment area at A.D. 1720.’ and later, ‘‘In the case of Lake Korttajarvi it is a demanding task to calibrate the physical varve data we have collected against meteorological data, because human impacts have distorted the natural signal to varying extents’)…

We therefore … compaired [sic] the reconstructions both with and without the above seven potentially problematic series, as shown in Fig. S8.

Unlike their handling of Briffa MXD data which was truncated merely because of a potential non-climate anthropogenic impact (without reporting on the impact of this truncation), Mann et al included the Finnish sediment (which went up), purporting to justify this by arguing that they were still profitable without the data. A policy which, regardless of its individual merit or lack of merit, is inconsistent with the handling of the Briffa MXD data which went down.

It’s actually worse here, because the original authors do not merely say that human effects unrelated to climate “might” impact the record. They categorically say that they were so caused. No “might” about it. For example:

This recent increase in thickness is due to the clay-rich varves caused by intensive cultivation in the late 20th century. …

There are two exceptionally thick clay-silt layers caused by man. The thick layer of AD 1930 resulted from peat ditching and forest clearance (information from a local farmer in 1999) and the thick layer of AD 1967 originated due to the rebuilding of the bridge in the vicinity of the lake’s southern corner (information from the Finnish Road Administration).

There is another shoe to drop in the handling of the Finnish sediments, which I will discuss in my next post. It appears to me that Mann et al actually have a short position on the Finnish speculative stocks, which they have inadvertently accounted for as a long position. But more on this in another post.

Accounting for Subprime Mortgages on Bristlecone Estates
A consulting report by North and associates, also sponsored by Cicerone, said that subprime mortgages on Bristlecone Estates, in particular, “strip bark” derivatives, should be “avoided”. However, Mann et al 2008 nowhere mention that strip bark derivatives remain on their books.

GAAP Accounting
It’s one thing to object individually to (1) the truncation of post-1960 Briffa MXD data due to a potential anthropogenic disturbance (which may actually be a non-linear response); (2) the use of the 20th century Graybill bristlecone chronologies despite (a) potential non-climate anthropogenic fertilization or (b) recommendations by the NAS panel not to use strip bark derivatives; (3) the use of 20th century Finnish sediment data despite an established non-climate anthropogenic disturbance.

However, the combination of the three is much worse than any one of these individually. The inconsistency of (1) and (3) is particularly opportunistic.

I make no imputation as to intent on the part of the authors, as intent is irrelevant (and, as is blog policy, I ask readers not to speculate on intent either). All that matters is what is presented. Auditors confronted with opportunistic accounting policies of this type cannot certify that the statements comply with GAAP. If North, Hegerl and Cicerone were chartered accountants and certified that such inconsistent and opportunistic practices complied with GAAP, they would be treated very severely by any supervisor of accounting practices.

if Team methodological descriptions were correct?

Mann says:

All ITRDB tree-ring proxy series were required to pass a series of minimum standards to be included in the network: (i) series must cover at least the interval 1750 to 1970, (ii) correlation between individual cores for a given site must be 0.50 for this period, (iii) there must be at least eight samples during the screened period 1800–1960 and for every year used.

I checked two series familiar to CA readers – Gaspe (cana036) and Sheep Mountain (ca534). Aside from any other issues with these series, the early portion of the Gaspe chronology has only a couple of samples as does the post-1983 portion of the Sheep Mountain chronology. I checked these series in both the “original” and “infilled” versions. The “original” version transcribed ITRDB crn series, including portions with fewer than 8 samples. The “infilled” version was identical to the “original” version where the original version had values. Thus, contrary to the statement in the SI, the “infilled” versions used data with only one sample.

Does it “matter”? Well, why say it if it’s untrue?