Willis writes: A couple of things.
First, I’ve digitized all of the Hegerl proxy data, and placed it here. I sampled it at ~three year intervals, and interpolated the actual years.
Second, I took a look at their reconstruction method. They say:
The first step of the reconstruction technique is to scale
the individual proxy records to unit standard deviation, weigh them by their correlation
with decadal NH 30-90°N temperature (land or land and ocean, depending on the target
of reconstruction) during the period 1880 to 1960, and then average them.
Now, except for one proxy, they are using decadally smoothed data. But for the reconstruction procedure, they have averaged out their data to the decadal level. Thus, they are basing their entire reconstruction on how well it fits vs. eight data points … it seems like this alone should put their error levels from “floor to ceiling”. Let me go see …
Yes, there is not a single correlation coefficient (r^2) in the lot that is significant. In fact, the series are so short (only 8 data points = 6 degrees of freedom) and the autocorrelation is so strong that the p-value for the r^2 can only be calculated for four of them. The best of these four is w. Greenland, p = 0.23. Statistically meaningless. The other ten, once they are adjusted for autocorrelation, have less than one degree of freedom, so their p value cannot even be calculated.
Thus, none of the r^2 values is statistically different from zero, and their method falls apart.
Here’s a spaghetti graph of the contestants …
Only 4 of the 14 have an r^2 that is better than a straight line with regards to the Jones data …
The result of the correlation weighting procedure is a dataset that it is more autocorrelated than most of the individual datasets … so we can’t calculate the p value of it either.
A bozo test of the value of their method, I suppose, would be to compare individual correlations with the first four decades of the Jones data, and then see how well they do in the next four decades. A little bit of “out-of-sample” test … I’ll do this using the smoothed data, rather than the decadally averaged data as they have done, to get a more accurate result. Hang on a few minutes … OK, thanks for waiting. Here’s the results …
YIKES … they have almost no correlation at all with the earlier four decades of Jones data, only with the later decades. The overall correlations dont have any relationship with either half, and the two halves have no correlation with each other … and these are the proxies that we’re going to depend on for temperatures a thousand years ago?!?
Can you say “fails the out-of-sample test”? … I knew you could.
w.
PS – How can the correlations be so different for the different periods? Easy. Here’s Jones versus some selected datasets. You can see why the correlations are so radically different during different time frames.
Climate Audit 
15 Comments
Willis, can you give a URL for the 30-90N series that you used (or did you do the calculation yourself?)
Shouldn’t Hergerl et al. be weighing them with the (signed) square of the correlation? (That’s how much of the variance is explained; also, variances are linear quantitites.)
I’m reposting this here, from the “Hegerl Proxies #1:” thread.
Steve M., I used the Jones database available here for the temperature data. Don’t know if that’s what she used, but it gives correlations very close to those reported in the the Hegerl paper.
w.
—————————————- repost —————————-
Man, this sucker is bogus. Where do I start?
They say:
They go on to explain the 12 sites, kinda, with sites that they don’t even know where they’re from, such as Boniface … they say:
Since they got a whole bunch of their data from Esper … couldn’t they have just asked him where the Boniface site was? But I digress …
Then they show this graphic:
Now, I see 14 records in this graphic … which they have used. The reason there are fourteen is that they use a short and long series from western Siberia, and a short and long series from the western US (“w. US. Hughes” and “w. US composite” respectively). The U.S. short series is described as:
Generated by Lloyd and Graumich … analyzed by Esper … provided by Cook … man, we’re a ways down the food chain. In any case, I’m sure this will be seen as an “independent” verification of the Hockeystick …
Then, there’s Mongolia, described as:
Not available? What’s up with that? Then we have:
Gotta love how they mess with the data … “obvious growth overprint”?
Of course, you can’t have Siberia without Yamal …
Then we have the mysterious:
which doesn’t tell us a lot. But further research reveals, you’re gonna love this, folks, that the Luterbacher “European Historical” proxy list includes … wait for it … Yamal in Siberia. Which means that Yamal is in this paper, not once, but twice.
I also suspect that this European composite may contain a common core with the Greenland series described next. Luterbacher says that he uses “1st PC of winter àŽàⲱ8O from Greenland” as a proxy. The west Greenland series, on the other hand, is described as:
A curious thing about this Greenland composite is that it comes from:
The curious thing is, Fisher is using the cores, not as a temperature proxies, but as precipitation proxies … wonder how Hegerl/Luterbacher removed the confounding variable …
Of course, we have to have the obligatory no-show:
Since the correlation is so low, 0.04, this proxy disappears in the “blend”, with a weighting of 0.2% of the total … but they still put it in. Why?
And how about the US long series? Well, that’s described as … as … well, in fact, it’s not described at all. Nowhere. Nothing. Not a single thing about it … searched the paper for “composite”, “US” … nada.
Next, we have the errors. Curiosities abound here. One expects error bars to get larger the further back we go, since there are less series (not even counting the disappearing Mackensie River series). But the wierd part is, the 95% confidence intervals in the middle section (~1000-1600AD) is smaller than the errors in the oldest section (~600-1000AD), and these, in turn, are smaller than the errors in the newest section (~1600 – 1890 or so). Here’s the data:
Era______ minus_______ plus_________
Recent___ 1.6 – 2.2___ 0.6 – 1.2 ___
Middle___ 0.8 – 1.4___ 0.4 – 1.0____
Oldest___ 1.2 – 1.5___ 0.6 – 1.0____
How many proxies are we talking about over time? They say:
I have verified this by looking at the data. As a first approximation, the errors should be proportional to one over the square root of the number of proxies. Thus, these errors should increase as 1 : 1.3 : 1.6 … but they don’t.
They also say:
Hmmm … why would the red noise be “filtered” by correlation weighted averaging?
In any case, I can’t make heads or tails of how they claim to “scale” the proxy average. I’ve tried and tried, but I just can’t see how they can estimate the average in the paleo reconstruction from this as they claim. They’re using the eight points of the weighted reconstruction, and the eight points of the data, to estimate the paleo error … how? As you can see, and as I showed before, the correlation with the first four decades is horrible … how can they get useful information out of that? Here’s the data they’re using …
w.
Re #2, Sara, thank you for your question. That’s what I thought too … but I did it the way they said, and came up with their results. Signing the r^2 isn’t necessary, by the way, as all of the correlations are positive.
Doesn’t make much difference in any case, since the average of the unweighted proxies is very close to the average of the weighted proxies.
w.
Re #4, Willis, Yes, I’d understood that the inaccuracy was the authors’, not yours.
This shows, I think, that the authors’ lack of statistical competence should be evident to anyone with an appropriate statistical background without even seeing the data. So this is further evidence that the peer reviewers didn’t have an appropriate statistical background to review the paper, or they didn’t read the paper, or they didn’t care that it was wrong.
Willis, if you look at my post on Mann’s PC1 in Hegerl, you’ll see that there is a description of their western US long series, albeit incorrect. The shorter US series is the average of two cherry-picked Graumlich foxtail sites, incorrectly attributed to Llloyd and Graumlich 1997; they are referred to in Bunn et al 2005 about which I’ve posted on this site and were archived in May 2006 after much fighting with Science.
The longer US series is Mann’s PC1 from Mann and Jones 2003, incorrectly described as an RCS-processed result "kindly provided by Hughes". So the long series is actually an old friend, even if Hegerl et al are too embarrassed to admit that they used Mann’s PC1.
Hey you guys. Can’t you just layoff of all of this statistical bs! Clearly you just don’t understand climate science. Come on over to RC and discuss these issues there if you really want to address these climate science issues!
What is really p…..g me off is that these silly questions you guys are asking are making it harder for me to get my grants. How would you feel if the income you need to feed your wife and kids is threatened? Hey?
#7 – is this a wind up? I suspect so.
Self-Concerned Climate Scientist: where exactly is the “statistical bs” that we should lay off of?
Re #7
My place or yours?
Jokes aside. I’ve posted two good questions in the past at RC (on the statistics of hurricane trends and on the constrainedness of GCM parametrization) and gotten next to nothing in reply. What little I got indicated a heads-in-the-sand/maintain-the-consensus attitude towards climate science. At CA you can actually generate a discussion and learn something. Example: Dr. Judith Curry and Dr. Isaac Held at CA provided answers to my questions that could not be gotten over at Consensus-Keepers.
Consensus-keeping is inherently anti-scientific. But as you point out: it’s good for Climate Science Inc.
Stick around and ask a question, or comment on a post. You might actually learn something. Or maybe you have something to teach us? What’s your take on bcp’s, Yamal, extrapolative RegEM, confidence interval estimation, and the like?
I think that post was intentionally facetious.
Mark
Willis E thanks much for your comments on the quoted parts of the paper. When I first read Steve M’s quotes from this paper, I am thinking is it me or does the approach in this paper just seem a bit loosey/goosey. When I read the above excerpt I was getting confused thinking Steve M had put some of his own comments in quotes.
That one really bothered me.
#5: It appears this is another example where the 43 Wegman identified as co-authoring together are also peer reviewing each others papers. Sadly we will not know because reviewers are not identified – a practice that has to change. Good work Willis.
I know this has been talked about before – the idea of having a “statistics reviewer” on every paper. If you guys had to design a way for that to work for any given journal, how would you suggest?
Also, a very well known scientist in the mesoscale meteorology world (at least, I respect him and his science very much) has written his own “review” of the peer review process here, I think some of you might be interested to read it.
RE: #7 – Hey, who says you are the only one who merits a grant? I also have granny, auntie and several … um …. shall we say … undocumented children who don’t reside with me to look after… a person’s got to do what they’ve got to do … 😉
One Trackback
[…] Problems with Mann-Holes :: Problems with the use of boreholes in the earth as temperature proxies. Willis on Hegerl :: Review of a paleoclimate reconstruction. When Good Proxies Go Bad :: Analysis of the proxies in […]