Mann and Rutherford [2002]: the CE Statistic

Barton’s question 7d asked MBH about other verification statistics. We’ve discussed their withholding of the R2 statistic here, here and here. In our GRL article, we also pointed out that their 15th century reconstruction also failed the CE statistic, another verification statistic used in dendroclimatic studies [Cook et al, 1994].

Mann’s retort was that climatologists "prefer" the RE statistic. I recently noticed an interesting discussion of the CE statistic in Mann and Rutherford [2002].

Mann and Rutherford state:

We made use of two distinct diagnostics of the data variance resolved by the reconstructions [see e.g. Cook et al., 1994] including the reduction of error (“ÅRE’) which uses the calibration period climatology as a baseline (this is called “ÅàÅ½àⰧ by Mann et al., [1999]), and the coefficient of efficiency (“ÅCE’) which instead employs the verification period climatology as a baseline [equations reformatted here because I don’t know how to do math terminology on WordPress]:
RE = 1- àÅ½à⡠(à’¡”¬?[i] — u[i])^ 2/ àÅ½à⡠(à’¡”¬?[i] — à’¦à«[cal])^2
CE = 1- àÅ½à⡠(à’¡”¬?[i] — u[i])^ 2/ àÅ½à⡠(à’¡”¬? [i] — à’¦à«[ver])^2

RE = 0 and CE = 0 represent the scores for a reconstruction which simply specifies the climatological mean, and thus only positive values of the statistics indicate statistically significant skill. CE = 0 is a more challenging threshold since, unlike RE, CE does not reward reconstructing an observed change in mean relative to the calibration period.

Their Table 1 shows reconstructions from pseudoproxies constructed to have a relationship to temperature. Of the 29 reconstructions from pseudoproxies, 13 have RE values over 0.9 and a cumulative total of 24 of 29 have a value exceeding 0.8. Thus, very RE values seem to be obtained ffrom pseudoproxies with an actual relationship to temperature. 13 of 18 had CE values exceeding 0.5 and 20 of 29 had values exceeding 0.3. So again, reconstructions with pseudoproxies yield high CE values.

Our GRL article reported that cross-verification statistics for the 15th century MBH98 reconstruction were (inter alia): R2: 0.02; CE: -0.35.

So Mann and Rutherford [2002] demonstrate use by two people holding themselves out as climatologists, and not limiting their statistical discussion to the problematic RE statistic

Here’s another sentence in Mann and Rutherford [2002] that’s fraught with significance:

Uncertainties in the reconstructions can be estimated from the residual uncalibrated verification period variance. .

This is a point that has bothered me in the MBH98 confidence interval calculations, which I may have mentioned before. If you have calibration and verification periods, then confidence intervals should be calculated from the verification period, as stated here by Mann and Rutherford [2002]. Obviously I don’t rely on them as authoritative on statistical issues ; the point can be confirmed elsewhere. However, in MBH98, they used the calibration period residuals.

Here’s the rub: for the 15th century reconstruction, there was a decently high calibration R2 (from sure curve fitting), while the verification period R2 was 0. The verification R2 of 0 means that the confidence intervals are not reduced below natural variation. There is NO reduction of uncertainty. The use of calibration residuals is not acceptable and should not have been permitted by alert referees.

Mann and Rutherford [2002] may explain another old question, that has re-surfaced recently (Crowley in EOS). After we published our first study, Mann said that the data set which we had used was not the correct data set. Mann said that we had asked for data in an Excel spreadsheet, that errors had been introduced in preparation of this spreadsheet for us and that we’d failed to notice the errors: "garbage in, garbage out". These were all provably false: I’d asked for an FTP location and had emails to prove it. I had no interest in data in a new form – I was interested in the data as used, which any reader of this blog will recognize as the way that I work. We posted up the correspondence to prove the point, but the lie still continues – it was recently published in EOS by Crowley.

The data set was not prepared for us. When we checked the date on the file at Mann’s FTP site, it was dated on August 8, 2002, shortly after the creation of Mann’s FTP site in Virginia on July 31, 2002 . So the claim that they had created it for us was a lie as well. (This does not mean that this file was necessarily used in MBH98. There are third alternatives, one of which is mentioned below in connection with Mann and Rutherford, 2002.) Shortly after publication of MM03, Mann deleted the "garbage" file from his FTP site (where it was on Rutherford’s section) together with its incriminating date evidence.

As to our supposedly failing to notice the errors, we spent over 20 pages itemizing errors in MBH98 data, one of which was collation errors. We described the re-collation and re-calculation of principal component series. For Mann to claim that we failed to notice the errors was one more lie. The disinformation continues and has been re-cycled on this board recently by "Mitch". Unfortunately, I have to respond to Crowley’s EOS article some time soon.

At the time,as I’ve mentioned, I was intrigued by a famous scientist making statements that were provably untrue. It sure caught my interest. Also a new directory of proxy data suddenly materialized in November 2003 – the "real" books, rather than the ones that we’d been shown before. There was lots of hair on this as well. However, I’ve gotten very tired of these canards continuing to circulate and the credence that they continue to receive.

As to the deleted file, it had the name pcproxy.txt. At the time, I found that there was a diagram on Rutherford’s website containing a graphic with the expression "pcproxy", but didn’t save it. A few weeks later when I went back, it had been vacuumed as well. In this case, fortunately, there was a record of Rutherford’s graphic at the Wayback Machine archive and the graphic with the phrase pcproxy is reproduced below (the date of this graphic was 2001, as I recall):

Figure 1. From Rutherford’s webpages as archived by Wayback Machine. Note the legend in which the phrase pcproxy appears.

Contemporary with MM03, we archived the re-collated data, which did not have the collation errors of Mann’s "garbage" data – in Mann’s data, the 1980 values for as many 9 series were identical to 7 decimal places. This was not easy to notice in a big data set, but caught my attention when I noticed it. Ultimately this doesn’t "affect" the results other than showing a lack of care and suggesting that a comprehensive examination of their results was required.

So we provided evidence that we didn’t use the "garbage" data. Mann has never provided evidence that he didn’t. Our surmise for some time (since the November 2003 disclosure of tte "real" books) was that pcproxy.txt was probably the product of work that Rutherford did, between MBH98 and our inquiry. Mann and Rutherford [2002 ] does not explicitly report on proxy results (as opposed to pseudoproxy results), but certainly evidences Rutherford working on this topic aronud the date of pcproxy.txt being archived at Mann’s then new Virginia FTP site.

Reference:Michael E. Mann and Scott Rutherford, 2002, Climate reconstruction using “ÅPseudoproxies’, GRL 29(10), 1501, doi: 0.1029/2001GL014554.

1. TCO
Posted Sep 12, 2005 at 8:52 PM | Permalink

They are really jokes, Steve. But isn’t it time to use your brain for something new?

2. TCO
Posted Sep 12, 2005 at 8:53 PM | Permalink

And the discussion of CE and the discussion of data set editions seems like different topics.

3. Steve McIntyre
Posted Sep 12, 2005 at 10:58 PM | Permalink

I’ve wondered about the provenance of pxproxy.txt for some time and the connection with Mann and Rutherford [2002] seems strong. That’s a link that I have that is not there for most people. It’s not a wonderfully constructed post, I’ll agree. I have off days.

I’ve been drafting a response to Crowley’s article in EOS, which is why some of the older history is on my mind. It bores the hell out of me to write, but I have to keep picking spitballs off the wall. Unfortunately, it seems that if you leave the spitball alone, it becomes received wisdom.

4. Brooks Hurd
Posted Sep 12, 2005 at 11:02 PM | Permalink

TCO, They may be different topics, but they are have the same source.

I have been in Steve’s position, with other people spreading lies about me. When I learned of the situation, I wasted no time letting others know about the lies. Whenever possible, I droped a few hints that the other person’s work was a bit less than flawless.

5. ET Sid Viscous
Posted Sep 13, 2005 at 12:05 AM | Permalink

And to Steve’s credit, Unlike others I won’t mention, when people come on here and slag him with blatant mis-truths, and unsubstantiated attacks he let’s them stand, addressing what few points that are made that are coherent enough to be addressed, though it may fall on deaf ears.

I know few people (Myself not included) who would allow that on their own site, to the degree that he does, since some are really nasty. Thankfully most are so lacking of, for lack of a better term, quality, that they only detract from the posters.

The sarcasm discussions aside, I think this is what makes his position stronger. It also helps the lack of substance of those that come on here and attack him.

He has skin thicker than ten men, and he’s good at the ciphering to boot. the only thing I wish, and I realize it’s difficult as he’s in a furball, for a more organized laying out of the facts (The he-said/she-said and when data was available, when it disapeared etc etc etc). The math I’m sure is top notch It’s far over my head, but I can grasp enough to see the points he’s making, and more importantly that no one has offered any substantive argument against.

6. Bruce
Posted Sep 14, 2005 at 1:56 AM | Permalink

Steve, in case it case has escaped readers’ attention, a new “hockey stick” or “hurricane club” is being road tested over at Mann’s RealClimate. Seems a similar pedigree of splice spiced data (and shape) to the plot.

One hates to think that a predicted \$200b fubar might be used to fuel a multi trillion tyranny…

7. TCO
Posted Sep 24, 2005 at 1:31 PM | Permalink

1. I agree with Sid’s points (good mostly, little bad) about Steve’s performance.

2. Re: 4. fine, but then the title should be different and the intro para or sentence should mention that the theme is broader then CE (that it is “issues with a particular paper”. But to title it CE, prattle straight into CE. and then seque to another topic is confusing. note that this goes to sid’s point about organization.