Just when you think that the MBH98 Little Shop of Horrors has been fully explored, something new turns up. I’ve never spent any time on the last part of MBH98, where he does a "detection and attribution" study linking his temperature reconstruction to solar, volcanic and CO2. All these detection and attribution studies (e.g. Hegerl) richly deserve close analysis. Anyway a blogger in New Zealand has written some highly provocative analyses, which includes a close analysis of first differences, which one of our readers was wondering about. For readers who will only read articles listed in the Jesuit Index or Nature/Science, these analyses may not be for you. For readers who are interested in what’s going on with these studies, go there immediately. It’s at the felicitously named Sir Humphreys, go to Chefen. I’ve made a few snarks about signal processing approaches to things, but Chefen shows the strength of a signal processing approach in knowledgeable hands.
Here are a few teasers, but please don’t merely accept my short characterization of the results, check it out.
Chefen examined the Fourier spectra and Allen variances of the first differences in the Mann temperature reconstruction, obtaining
"a nearly perfectly straight line sloping downwards … with a slope of nearly -1. To anyone who works with signal processing that means… almost a pure noise source…. It looks whiteish with the low frequency removed, or as if some noise has been high-pass filtered.
He states:
If you do want to create your own Mann-method temperature noise signal, then just use a N(0,0.0115) distribution and add it up. It’s a fairly good approximation.)
Now for the next interesting find. Chefen did a simple check on the correlation of the temperature reconstruction to solar and to CO2, reporting:
Note at this stage that Mann outright claimed that the correlation for the CO2 data in the 20th century was much much higher than that for the solar data. Now straight away we are suspicious of that, because just looking at the graph above it is hard to say which is a better match. Indeed, if we do an actual correlation then the coefficient for the solar data is 0.804 while that for CO2 data is 0.815. That is, marginally higher for CO2 but the difference is utterly insignificant. We really have no cause for prefering one over the other and CO2 certainly does NOT dominate.
Now for the topper. Chefen gets exactly the same coefficients from random data processed according to the sum of filtered white noise described above:
Let’s take that idea a bit further, now I’ll generate a purely random data set in the following way…1. Create a normally distributed data set with the same mean and standard deviation as the difference temperature data. This is pretty much like white-noise and distributed as N(0.045,0.201).
2. High pass filter this with a cutoff frequency of 0.1/year to make the spectrum match that of the difference temperature data, which lacks power below 0.1/year.
3. Sum this data to create a noise data set that has the same characteristics as the real temperature data.
4. Calculate the correlation of this noise with the temperature data, just as we did for the solar and CO2 data.
5. Repeat this 100,000 times to get a good idea of what is going on.The correlations for with purely RANDOM data sets are tightly bunched in the range 0.80 to 0.83! The correlations for the CO2 and solar data lie completely WITHIN this range. What does this say?
It’s quite remarkable. Will his observations become more true if they are published in Nature or more formally in a journal? I don’t think so. They are either right or wrong as they stand. The journal is simply a form of endorsement and an effective means of disseminating the information to non-specialists. You have some qualified specialists, who may or may not have their own agendas, telling you that the article is worth reading. In this instance, I view myself as a qualified specialist and I’m telling you that this is worth reading. I’m not telling you that it’s necesaarily true as I haven’t replicated the simulations. But Nature referees don’t do that for you either.
Climate Audit 
36 Comments
Stop right there, meester mathbook. You seem to be wandering down the road of dissing the journals and saying that its impossible to publish contrary papers in them (that some of your cheering section say when I tell you to write papers). Your failure to puiblish has more to do with writing ability or hesitancy to be put in the open for correction or not knowing which journals to submit to or just inability to finish work. You’ve said yourself that you can get published fine, when you want to.
There are enough different specialty journals (or even SPECIALTIES) that you (and this other dude) could publish in, that it would be perfectly feasible for you to make your comments in the archived literature (and yes, Science Citation Index and a journal printed on acid-free paper and held in libraries and with peer review is a hell of a lot better than blogs that come and go
On topic: How about finishing the thought. What is the eggregious inference to draw from the result? Does it prove that the work is meaningless from a significance standpoint? The part about .805 and .815 being close I get. But the other stuff?
P.s. Damn comment box is floating under the side bar again. WHere is JohnA? Blog needs some design work.
No, I don’t intend that as a slight to journals. I like journals. There are some things that I wish that they’d do better, but you can make those wishes only because there’s a decent framework. I was snarking here not because of cheerleaders but because of a couple of trolls who seem to measure truth on whether it’s in a journal. I think that journals are a good way of disseminating information.
If I was in a publish-or-perish mode of life, I’d be publishing, don’t kid yourself. However, given where I am, I need to be a bit selective in what I take on. Maybe I’m wrong in this, but it’s not timidity. You don’t need to keep nagging me on it; I know your point and agree with much but not all of it; we don’t need to keep talking about it.
TCO, I think that Chefen makes a good point. It is either true, or it is not true. Where and how it is published doesn’t make any difference to whether it is true or not. I must say to you that my reaction to your comments on this topic (is it 20-30 times you have made your point?) is that why don’t YOU put a paper together and publish in the “right” journals. It seems to me that you have a good enough handle on the issues to make a real fist of it.
I have wondered, since I became interested in this blog, where the statistical and signal processing specialists are who can offer credible analysis. Steve is doing a great job in my view, and there are numerous credible contributors, but it is refreshing to see a totally new person, clearly expert in his field, emerge and tell it like it is, without fear or favour.
And already the trolls are emerging on his blog, using their standard tactics of dissembling, not taking on the real points he is making, talking about their “beliefs” etc.
I wonder if RC has picked up this topic yet. It is crystal clear, even to a layman, that something is not quite right with their work!
I’m sick of making the point as well.
Re publishing Steve’s stuff:
Is there no low-level climate researcher willing to write up Steve’s stuff and submit it to journals in exchange for co-authorships? Steve generates so many ideas that he should really have a little beehive of these people turning this stuff out. This is where funding becomes important. If Steve actually got any real funding, he could do this.
UCAR should pay him a fortune to do QA.
======================
I have a longish post for chefen at his site:
http://www.sirhumphreys.com/node/5888#comment-25618
bunch of questions. (I am not a statistician so apologies if they are not correctly stated or are a bit hit and miss in impact.)
1. Is the “Mann temp data” that you analyze the instrumental data or the proxy data?
2. Assuming instrumental, how does this average compare to those made by other people? How would your analysis differ if you had looked at a different temp reconstruction? Is the issue with HIS temp reconstruction or with his interpretations of THE temp?
3. Is it significant (and if so how, simple explanation please) that the first differences are random? If we look at market index numbers over time, we would see random first differences (well known that you can’t predict tomorrows move in a stock from today’s move). However, it’s also well known/beleived from CAPM theory that stocks over time appreciate.
4. Could you (in simple terms) comment on the significance of Allan deviation and interpretation of it’s results and if it is an accepted tool generally. (I did look up a definition on web of the tool, but still don’t really have a feel for it.)
5. what does it mean that you “did a linear fit of CO2 and solar onto temperature. It’s not clear to me what you are plotting when you get the three onto the same plot.
6. Don’t the GWer generally say that temp effects are a result of log CO2? So should you be looking at that versus CO2 linearly in your (and Mann’s) correlation experiments?
7. Also, isn’t a lag posited based on thermal mass and all that, so how does one do experiments with that in mind as a likely confounder?
8. (Related to 2) Is your analysis only of interest wrt Mannian claim/debunking industry or is it of interest in terms of general discovery of the nature of things?
9. Will you publish your work (please)?
10. Is it a requirement for correlations that the differences be similar or just the values follow a relationship? What if you have a very noisy system and then have a smoothly varying forcing (with perhaps some lag issues as well). Is it so strange that results in the dependant variable would stay noisy?
11. What is with the 200 year versus 100 year thing here? Are you seeing bad things in Mann by restricting the analysis to the last 100 years and thus removing the “time when CO2 was not changing much” that would help the overall correlation? I’m reaching here. But if Mann has issues with the correlation during an up period (in the micro sense) but he tracks well over long scale peaks and values, aren’t you taking something from him that supports his case, by using a shorter window? And intuitively, shouldn’t one use longer windows to get ideas about influences in difficult systems? I trust a market risk premium of 5% (per Copeland etal) based on 100+ years of data more than I do the dotcommers from Goldman who said (based on the late 90s) that it must be 3%.
TCO, I responded to your questions over at Sir Humphreys. I also responded to your questions about the slant or motivations of the blog and its authors, which I did not appreciate. Would you care to explain why that is at all relevant? In general I’ll say that intuition only gets you so far and (flawed) analogies with the stock market only get you so far. This will not be published because (1) it is only an exploratory investigation after a couple of hours work, (2) I don’t have the time nor money to go through that process and (3) it needs a lot more work.
Chefen:
1. The question about theme or slant was out of general interest (for my surfing pleasure).
2. I gave a caveat about my level of knowledge. I’m not nescesarily making an assertion when I refer to things like the stock market but am saying why isn’t this like that. I agree that analogies can be inexact or off. That said, I’ve often found these simple physical intuition questions to flush out contradictions and that the best people are those who can explain the issue in a Feynmanesque manner.
3. Thanks for your reply on the limitations of the analysis wrt publication.
P.s. I still like my market risk premium comment…
Dude,
Let’s talk on one blog at a time. I should have just given the link iinstead of copying the post. You felt that you had to reply both places. This is not the standard TCO treatment. I’m just hashing things over on the internet. Sheesh, that’s what blog comments are for. Don’t be so damn defensive. I like to kick the tires a bit before I buy a new automobile and look under the hood. Doesn’t mean I’m calling the car salesman a theif. Just like to check things out a bit. Would think you would have no problem with that given your scrutiny of Mann and such. I just like to ask the appropriate questions to make sure that I’m getting the right intuitions and understand all the limitations.
OK, cleared up, carry on over at the other place.
Oh dear! The last part of MBH98 not touched so far turns out to be the same sh*t as the rest of the paper. What’s next.. they have a fake bibliography? 😉
Anyhow, tack Chefen! In retrospect, one should have found those things earlier… 200 year smoothing is just so … hmmm… “Mannian”. While you are on to it, it would be nice if you could do some time-frequency analysis on data 🙂 If you are using Matlab, see here and here; the tutorial of the TFR toolbox is good reading for everyone interested in this topic. I’m not using R, but there exists at least Rwave-package. It would be interesting to see what those signals reveal under proper analysis, e.g., what their Wigner-Ville spectrums look like.
Talking of Sir Humphrey reminds me, didn’t the timetable for the NAS panel report state a publishing date of mid-May? Given the changes to the brief, have they given a revised indication of when it would be published?
I’ve made a few snarks about signal processing approaches to things
Hmmmm… maybe after reading this you might have some more appreciation for the signal processing approach 😉
#16. I agree that it was wrong to have a bad impression of signal processing methods because of how Mann applies the methods. Oddly enough, I had a nice chat yesterday with a young professor at the University of Toronto who taught signal processing. I noticed that he was reading a paper bristling with sigmas and integrals at my local Starbucks so I said hello, as I was also doing the same.
#14. One of the many things about the Corrigendum which irritated me was that Nature said that they didn’t have enough space to mention all the errors. So they mentioned a citation error in the Corrigendum, but nothing to do with principal components centering – which aside from the merit or lack of merit of the Mannian methodlogy – the methodology was incorrectly described in the original publication and this should have been mentioned. Ironically, having taken the trouble to acknowledge that the original article was cited incorrectly, they then cited another incorrect article.
Actually, MBH is replete with little and bizarre problems, some of which were noted in our MM03 article and others in our Materials Complaint to Nature. The most amusing surely have to be Mann’s geographic mislocations of precipitation series. He ascribed two French precipitation series to North American gridcells – I’ve offered the following slogan: “The rain in Maine falls mainly in the Seine”. Given his excoriation of Soon and Baliunas over allegedly confounding precipitation and temperature proxies, it is remarkable to see precipitation series used in MBH98 in a formally identical way to temperature series.
I suspect that the precipitation series assigned by MBH98 to the gridcell containing Bombay actually comes from North America. It isn’t the Bombay series. Mann has refused to provide a source for this series other than “NOAA”. Even with a complaint in hand about misidentifying series, NAture took the position that this was adequate identification.
Better late then never! I hope this professor takes an interest in what your doing.
You guys might want to look at this about figure 7 of MBH98.
I finally got round to doing what I originally intended and for the life of me I can’t get his correlations to match after about 1870, but they match near perfectly up until then.
Take a look, I’d love the opinion.
re #20: Welcome to the Mannian world! I just had to check it… you are right. I’m able to reproduce your figure… so this is again one of those things we need to start guessing what’s been actually done. [BTW, as a small hint, it is illustrative to put the both graphs (Mann’s and your) in the same figure, it is easier to see the differencies.] I could not make the graphs match exactly anywhere, there is also a rather large differece for DVI index about the same time.
Steve, you know the Fortran code provided, does it include these calculations?
Now to the guessing. I think Mann did not actually calculate the correlation coefficient. I checked also against the “true construction” (fig 5), since in their Corrigendum it is stated (see also here) that
[Does the last sentence mean that there are “known differences” or “no differences” ;)]
However, this does not change the situation. After wondering a lot what is stated and what’s actually done in MBH98, I am guessing that the answer lies again in the Mannian lyrics. From the actual text:
They are talking about “normalized regression coefficients”, not exactly about correlation coefficients. So this might again be one of those “normalization convention”/detrending things: you could try to normalize the data to the instrumental mean/std for the verification period (1902-1980), or something like that (not to substract the true mean of the window anywhere). See also Mann et al (2000) which might give additional hints. Good luck!
Jean S, I was just working through this. I independently noticed that the NH reconstruction in the figure was different. I’ll post up a graphic showing the differences.
#21, Ah I was just wondering about this back on my post. I didn’t have the text you have there. Now I notice he DOES use partial correlations. The question is how is he controlling them? Because on a case-by-case basis, CO2 and solar seem pretty damn inseparable. It seems to me that using partial correlations is going to lead to all sorts of problems as he assumes a linear model and surely the dust index will screw that up, not to mention omitted variables. After all the divergence begins around 1850 on my plots and half a window away is 1950 where a very dust-free period just finishes. Assuming a linear depedence seems OK(ish) for CO2 and Solar but for a dust INDEX? I just don’t see it. I think that is probably screwing it up since both dust and solar have much finer behaviour compared to the rather long smooth rise of the CO2, could the dust be confounding the regression due its rather nonphysical measurement? What would it look like with dust omitted? I don’t trust Fig 7 at all.
that paper is a mess,
I’ve done an “evolving multiple regression” and sort of get the downturn in solar coeffient, but don’t get the other things. I’ll work on it some more. There’s an interesting interaction here, because his multivariate method in the proxy reconstructions, after you do the two maximizations, reduces to being partial least squares, which is just partial coefficients. But Mann doesn’t appear to have known that.
I wonder what “evolving multiple regression” will evolve into – maybe Neanderthal Mann. I’ve always thought their methods were primitive.
re #23 (and #25): That’s it! I’m able (closely enough) to reproduce the MBH98 Figure~7 with (sample) partial correlation coefficients (sorry could not find quickly a better link)! So that’s the answer!
Now, this gets interesting when you choose other “reasonable window widths” :)… so here is another addition to the MBH98 classics:
I’ll leave you (Chefen, Steve) to take the fun out of this 🙂 I also invite Tim Lambert to reproduce the results, and then explain with your superior understanding how the following conclusions (with emphasis below for your convinience) are “robust” to the other “reasonable window widths” (try, e.g., 100-year as suggested):
With this figure 7 part, it sounds more and more like an argument of advocacy rather than a scientific paper. I wonder if these guys even know the difference. Most of them are washouts from hard science.
#26. Jean S, you’re pretty good at decoding Mann-speak. Based on what you said, I pretty much replicated Mann’s graphic by:
1) Subtracting the mean and dividing by the standard deviation for each 200-year window;
2) doing a multiple linear regression of the “old” NH reconstruction against CO2, solar and volcanic.
3) Taking the regression coefficients.
(I don’t entirly understood your link.)
As you suggested, I tried this with a 100-year window and, as you said, the results were completely different. The solar coefficient was much stronger than the CO2 coefficient.
This exercise is rather interesting statistically. There’s a lot of collinearity between the CO2 and solar in the 20th century – so if the regression were regularized through ridge regression, the coefficients would be moved towards the partial correlation coefficients (as plotted by Chefen originally.)
Also everyone remember the “detection and attribution” studies that are supposed to be the next line of argument after the HS studies and which are supposedly “independent” of the HS. This is a detection and attribution study. This is perhaps even worse than the HS part of the study. If it were possible to get hold of Hegerl’s data, I suspect that her results are a totally trivial exercise like Mann’s.
As a statistical exercise, this little multiple regression is presented with about the level of sophistication of a university freshman or high school senior. Maybe not even that. What separates it from an undergraduate essay is that the undergraduate would probably have reported statistical confidence intervals, might have done some testing. I’ll write this up soon, but it’s both amusing and pathetic.
I have been having a lot of fun with the data supplied by Hegerl in the Supplementary Info and simple regressions. Like comparing their short reconstruction CH.blend.zn with the long CH.long.zon. They are the same except for the long one goes to the 13th instead of the 16th Century. The shorter record gives Solar and GHGs as significant factors, but in the longer the GHG drops out and only the Solar is significant. If you plotted GHG correlation against length of record, starting at the present day, you would get a steady decline to insignificance. Interesting, I think it means something but not sure what.
I was very surprised that the Solar record record they use is very strong in the reconstructions I have. Even more so if you allow for a little temporal shift in the peaks. Another thing, if you smooth the volcanics you get a much higher correlation that wipes out GHG significance. Sounds a bit like the window dependence above.
Typical of multiple regression to be all over the shop, but interesting none-the-less.
I sent yesterday my code to Steve, but it somehow didn’t reach him. I resent that today, but before he can write up the results, here is the quick preview for all those interested.
First, the replication of the results. The solid lines are mine, and the dash-dot lines are from MBH98. Good enough match?!?
And now here is the same for the 100-year window! Compare and reread #26 🙂
Tim Lambert, any progress in reproducing these results?
#30 – Jean S
Don’t hold your breath.
I’m sure that Lambert is spending too much time forming opinions to do any actual study.
I can’t read the legend on the 100 year window Jean S.
Anyhow what I want to see is what happens when you use
consecutive 11 year means ( to eliminate the sunspot cycle)
and stop correlating moving averages which is really suspect
in my view.
See my post on Sir Humphrey / Cheffen
John Reid
re #32: blue: CO2, green: solar, red: volcanic. There is no moving averages (smoothing) used here. Those are simply partial correlation coefficients in moving windows plotted such that the year corresponds to center of the window. I (or Steve?) check later today what happens if you smooth the series before calculating the correlations. Although I think the remaining difference has to do something with the “normalization” (compare #28). Anyhow, doing any processing on the series is rather meaningless in my opinion, since I really don’t think Mann’s contruction represents the temperature history.
[BTW, if you are using Mozilla, try clicking the image :)]
Jean S, I get much the same plot trying partial correlations and yes the window does have an effect under that analysis. Where exactly did the choice of 200 years come from for the window? It is stated to get a good SNR, but it’d be interesting to see the criteria for that particular measure. Was it a real measurement or just “long enough to make it look smooth”? Has anyone tried it omitting a variable at a time and comparing the results? You must be able to do some sort of comparison (Bayesian?) to see which is the better model, eg is co2+solar better than co2+solar+dust and if so what are the results.
I’ve done a new post on this. Check it out.
The blog site is down. The work was never published. Journals are BETTER than blog posts.
On the other hand, if you were referencing TCO, it is definitely appropriate.
Pity the Chefen, Sir Humphreys link is not longer working. I hope someone made a copy.
One Trackback
[…] week, through Chefen, Jean S and myself, here here here and here , we showed that MBH98 contained questionable statistical methodology for assessing the […]