Industrial Strength Voodoo Correlations

This is a very pretty example, though the problem is endemic as Mann et al 2008 uses Mannomatic methods for industrial strength voodoo correlations.

This one came up from trying to replicate Mannian confidence intervals from original data – an effort which promptly foundered because my CPS emulation, which, after much effort, finally worked on the 1850-1995 period, immediately foundered when I tried with the “late-miss” (1850-1949) and “early-miss” (1896-1995) calibrations. So back to step by step reconciliation, obviously a lot easier since a lot of modules are working. UC saved the Matlab intermediates so we had something to work with.

Here’s what happened.

In making my “late-miss” emulation for AD1000, I got all the “right” proxies and the calculations all worked (with a little editing.) But the orientation of the Socotra dO18 speleothem was inverted in the late-miss version (this is the speleothem series that we looked at in some detail a month ago, but that’s just a coincidence.)

Huh?? Why would the orientation be right in one emulation and wrong in another emulation? The answer was very timely in terms of “voodoo correlations”.

The “low-frequency” correlation was 0.476 for the 1850-1995 period: a “significant” correlation. The “low-frequency” correlation for the subperiod 1850-1949 was also “significant”: but it was -0.580.

In my initial attempt to emulate Mann’s “late-miss” recon, I had oriented the series using the sign of the correlation for the 1850-1995 period. Silly me. It appears that Mann’s alignment is so opportunistic that the same series is used in different orientations depending on whether it is a “late miss” or “early miss” version. Industrial strength Mannomatic.

Ooga in:

ooga=load(strcat(‘cps_results/’,upper(‘NH’),’_ea100′,expe,’_newgrid_’,iseries,’_gbeachyear.mat’));

Chaka out.

Ooga chaka.


54 Comments

  1. Posted Jan 16, 2009 at 2:32 PM | Permalink | Reply

    Wow. Too bad that didn’t fit into 250 words.

  2. Michael Whitty
    Posted Jan 16, 2009 at 2:43 PM | Permalink | Reply

    Interesting. Just to clarify – you’re trying to reverse-engineer Mann’s GW chart and projections, because he won’t release the model himself?

  3. Mike B
    Posted Jan 16, 2009 at 2:43 PM | Permalink | Reply

    Where is Julien Emile-Geay when you need him? Perhaps he can explain all these “sophisticated” Mannian methodologies.

  4. Steve McIntyre
    Posted Jan 16, 2009 at 2:48 PM | Permalink | Reply

    #2. NO. Mann has done a semi-decent job of archiving code. That’s how UC got the intermediates to see what was happening. But the code is horrendously written and lots of pointless to-and-fro, so it takes time to put into sensible terms and then to figure out what it all means statistically.

  5. Steve McIntyre
    Posted Jan 16, 2009 at 2:51 PM | Permalink | Reply

    #1. Jeff, it’s worse than I thought. Out of 1209 proxies, 308 have opposite “low frequency” orientations between early-miss and late-miss. Presumably many are “significant” in both directions. What a mess. YEah, I wish I’d noticed this earlier. But as you well know, it’s a laboratory of horrors and hard to know where to even bengin.

    • Tony
      Posted Jan 17, 2009 at 2:57 PM | Permalink | Reply

      Re: Steve McIntyre (#5)

      Out of 1209 proxies, 308 have opposite “low frequency” orientations between early-miss and late-miss.

      Oh my.

      “He must come through this hollow way, no other path leads to Küssnach.”
      Willhelm Tell

      Tell knew that there was no other path for Gessler to take – Mann simply diverted all paths to lead through the hollow way he had choosen to camp out at a long time ago.

    • MikeU
      Posted Jan 18, 2009 at 12:14 PM | Permalink | Reply

      Re: Steve McIntyre (#5), “Out of 1209 proxies, 308 have opposite “low frequency” orientations between early-miss and late-miss.”

      Ouch! Even RC should have a difficult time defending that study. Having Tiljander used upside down was a great example of confirmation bias. But having over 300 series used with differing orientations goes way beyond “sloppy”. Whether it’s incompetence or laziness (or both), it’s difficult to say. It cannot be deliberate, because if it were, the researcher would know how profoundly inept that would make him look were it to be exposed. It is indeed too bad this wasn’t discovered before Steve’s comments to PNAS were accepted.

      • Posted Jan 19, 2009 at 12:03 AM | Permalink | Reply

        Re: MikeU (#40),

        RC’s reaction should be interesting if they say anything. Since I can’t even comment there after asking them politely to rationalize removing data by correlation — RC gave a big SNIP to that question. I had no understanding of how over the top those guys were at the time or the question would never have been asked. It’s just data folks!

        Maybe a post on some of the early RC comments and some from the paper contrasted with reality would be fun on The Air Vent. I’m still waiting patiently for the detail from Steve’s latest quantum multi-state anti-proxy discovery though.

    • Deep Climate
      Posted Jan 20, 2009 at 2:31 PM | Permalink | Reply

      Re: Steve McIntyre (#5),

      Out of 1209 proxies, 308 have opposite “low frequency” orientations between early-miss and late-miss. Presumably many are “significant” in both directions.

      Strictly speaking, the Socotra d018 is the only proxy passing the significance test for all three periods and having an opposing sign between the “late miss” and “early miss” periods. But most of the proxies (tree ring, coral etc.) are evaluated with a one-sided test, so this is not a surprising result.

      Of the 484 proxies passing the 1850-1995 significance test, 342 also passed both sub-period tests (with 341 having r values with matching sign). 111 passed only one of the sub-period tests, and 31 failed both sub-periods.

  6. Clark
    Posted Jan 16, 2009 at 3:10 PM | Permalink | Reply

    Just wow.

  7. Kusigrosz
    Posted Jan 16, 2009 at 3:26 PM | Permalink | Reply

    The T rise during the last 150 years is some 0.3 – 0.4 percent in the absolute scale (which is the way T enters most physics equations). During that same time some other (especially local) factors that affect the “proxies” may have changed far more, due to human activity. Doesn’t that increase the chance of getting spurious correlations with T beyond that expected from pre-1850 noise?

    Suppose some effect is sensitive to two factors:
    Y = a * (T / T0) + b * (X / X0) + other_noise,
    and T changes by 0.4 percent while X changes by 4. If the changes in T and X are well correlated, and a isn’t much greater than b, I think one may get a good correlation with T due mostly to b.

  8. Steve McIntyre
    Posted Jan 16, 2009 at 3:42 PM | Permalink | Reply

    Just because the Team calls things “proxies” doesn’t mean that they are. You’re reading me into this than is there.

  9. Steve McIntyre
    Posted Jan 16, 2009 at 3:44 PM | Permalink | Reply

    :)

    I tried to emulate Mannian flipping. What happens if the sign of “low frequency” correlation changes but not the sign of the “high frequency” correlation. It all depends.

  10. Steve McIntyre
    Posted Jan 16, 2009 at 3:50 PM | Permalink | Reply

    OK, I’ve pretty much emulated the late-miss AD1000 step. It’s really a bit pathetic how bad this study is.

    • Mike B
      Posted Jan 16, 2009 at 4:24 PM | Permalink | Reply

      Re: Steve McIntyre (#10),

      And this is the study Gerry North wants you to “move on” from?

      What good is it to have statisticians like North and Nychka involved in this stuff if they won’t “represent the ‘hood” when needed?

      Sorry to vent Steve. I know it’s innappropriate, but I’m really, really disappointed by the work the statisticians that should be improving climate science.

    • Posted Jan 16, 2009 at 4:45 PM | Permalink | Reply

      Re: Steve McIntyre (#10),

      I would never have even guessed when this paper first came out. I can’t wait to see the writeup.

      Its the flapjack proxy or a new form of quantum data which exists simultaneously in two states. No I know, it’s [self snip] – jetlag.

  11. Thor
    Posted Jan 16, 2009 at 4:03 PM | Permalink | Reply

    Industrial strength voodo correlations? Now that’s funny :-) lol

    Anyway, I think I prefer this Ooga Chaka

  12. Steve McIntyre
    Posted Jan 16, 2009 at 4:12 PM | Permalink | Reply

    Here’s an Ooga Chaka complete with garbage cans – GIGO ? This seems to be the most appropriate Ooga Chaka so far.

    • Ron Cram
      Posted Jan 16, 2009 at 8:30 PM | Permalink | Reply

      Re: Steve McIntyre (#12),

      The first OogaChaka was hysterical the trash can one is even better.

      Regarding the post, is it possible to have a second comment on Mann08? Or is that only possible if Mann attempts to rebut you?

  13. Wansbeck
    Posted Jan 16, 2009 at 4:32 PM | Permalink | Reply

    #12 Looks like Al Gore with the blue bin.

  14. anonymous
    Posted Jan 16, 2009 at 5:08 PM | Permalink | Reply

    Is there an explanation somewhere of what late-miss/early-miss corresponds to ?

  15. Shallow Climate
    Posted Jan 16, 2009 at 7:52 PM | Permalink | Reply

    But, poor old man(n), thou prun’st a rotten tree
    That cannot so much as a blossom yield,
    In lieu of all thy pains and husbandry.

    –AS YOU LIKE IT

  16. jae
    Posted Jan 16, 2009 at 9:31 PM | Permalink | Reply

    LOL. Are we witnessing the ultimate cherry harvest ever? A ROBUST vintage cherry harvest for a perfect cherry pie? I’m still struggling for a metaphor for an upside-down cherry, though.

  17. jae
    Posted Jan 16, 2009 at 9:55 PM | Permalink | Reply

    Seriously, Some of the Team’s works may turn out to be the most egregious examples of bastardized science in history. If they had just backed down a little bit, years ago, they could have pleaded ignorance and “we just made a mistake.” But now, they are toast. Ego.
    snip

  18. D Johnson
    Posted Jan 16, 2009 at 10:07 PM | Permalink | Reply

    jae

    “Reminds me of Nixon, somehow”

    Funny. It reminds me of Clinton. Guess it’s a matter of perspective.

  19. mugwump
    Posted Jan 17, 2009 at 8:23 AM | Permalink | Reply

    Wonder why he changed the sign for just one part of the series? Why not allow arbitrary rescaling of the data at every point. Presto! Perfect proxies.

  20. Craig Loehle
    Posted Jan 17, 2009 at 8:39 AM | Permalink | Reply

    “oh what a tangled web we weave….”

    gee, do you think this affects the stationarity assumption? wow

  21. Carrick
    Posted Jan 17, 2009 at 9:49 AM | Permalink | Reply

    Actually the first version of Ooga Chaka made my hair stand up on end. It was either that or the sign flipping in the so-called proxies. Maybe both.

    Wow.

  22. Andy
    Posted Jan 17, 2009 at 11:38 AM | Permalink | Reply

    Could someone tell me what this means in simple language?

    • mugwump
      Posted Jan 18, 2009 at 10:49 AM | Permalink | Reply

      Re: Andy (#25), in simple language, Mann has a bunch of proxies that fall into three buckets: temperature predictors, temperature anti-predictors (ie, the negative of the proxy is a “predictor”), or junk.

      Now, a proxy should not be in two buckets. It especially cannot be both a predictor ant an anti-predictor. But that, apparently, is exactly how they are used: depending on the time period Mann is predicting, he changes whether a proxy is a predictor or an anti-predictor.

  23. anonymous
    Posted Jan 17, 2009 at 12:38 PM | Permalink | Reply

    Mugwump, as i understand it the sign change is done automatically by the algorithm… any anomaly is taken to be a “connected” anomaly and it’s sign adjusted accordingly. This particular proxy had its anomaly over one period inverted because over that period it was inversely proportional to the signal it was being calibrated against, whereas against another period it was proportional to the signal and so was not inverted despite being the same proxy.

    Could someone confirm this understanding is correct ?

    • mugwump
      Posted Jan 18, 2009 at 10:41 AM | Permalink | Reply

      Re: anonymous (#26),

      Mugwump, as i understand it the sign change is done automatically by the algorithm… any anomaly is taken to be a “connected” anomaly and it’s sign adjusted accordingly. This particular proxy had its anomaly over one period inverted because over that period it was inversely proportional to the signal it was being calibrated against, whereas against another period it was proportional to the signal and so was not inverted despite being the same proxy.

      That’s my understanding too. But why stop there? Why not adjust the magnitude of the proxy while you’re at it, not just the sign. And why change all data points uniformly? You’ll get a much better fit applying independent scaling factors to every data point.

      [Of course I am being sarcastic. The point is any manipulation that adjusts the data to match the signal is non-kosher. If he's willing to go down that road, why not go the whole way?]

  24. Tony
    Posted Jan 17, 2009 at 3:13 PM | Permalink | Reply

    While not a tree, I need to point out that Willhelm Tell as well does hide behind some sort of woody plant, an elder bush to be precise.

  25. Scott Brim
    Posted Jan 17, 2009 at 7:04 PM | Permalink | Reply

    The title of this thread is “Industrial Strength Voodoo Correlations” but perhaps “Mannufactured Correlations” might have been an alternative and otherwise appropriate title.

    At any rate, it is becoming clear, as other CA readers have suggested, that a comprehensive paper dealing with the full spectrum of statistical issues to be found in Mann 2008 Et Al would be a most useful and informative addition to the body of peer reviewed climate science literature.

    • Craig Loehle
      Posted Jan 18, 2009 at 8:57 AM | Permalink | Reply

      Re: Scott Brim (#29), I am afraid it would need to be a monograph. This is kind of like watching a magic show–what comes out of the hat next, ladies and gentlemen? You simply won’t believe your eyes.

  26. Ryan O
    Posted Jan 18, 2009 at 7:01 AM | Permalink | Reply

    At any rate, it is becoming clear, as other CA readers have suggested, that a comprehensive paper dealing with the full spectrum of statistical issues to be found in Mann 2008 Et Al would be a most useful and informative addition to the body of peer reviewed climate science literature.

    .
    Amen.

  27. Steve McIntyre
    Posted Jan 18, 2009 at 10:42 AM | Permalink | Reply

    #31. It crossed my mind a couple of years ago that you could make a very stimulating statistics course simply on the MBH laboratory of horrors. It gives lively examples of how not to do things. If you add M08 into the mix, you could have two courses,.

    • Joe Crawford
      Posted Jan 18, 2009 at 12:17 PM | Permalink | Reply

      Re: Steve McIntyre (#33),
      They would make a great course or lab in practical applications for statistics. But why limit it just statistics? When you consider all of the errors (e.g., statistical, procedural, coding, archiving, etc.) in Mann’s work that have been revealed on this blog over the last several years, they don’t point as much to a misuse or misunderstanding of statistics as they do to some combination of carelessness, lack of attention to detail and/or of ability or desire to thoroughly analyze a problem and select proper in depth solutions. Add to that, the fact that all of the referenced papers have passed peer review and been published in major journals, and this doesn’t bode well for climate science, or in fact, science in general. Maybe an introductory course in the scientific method would be more appropriate. It does seem to have fallen out of favor of late.

      Joe

  28. Steve McIntyre
    Posted Jan 18, 2009 at 10:44 AM | Permalink | Reply

    #32. What made you think that Mann didn’t also adjust the scale of the proxy differently in every case? :)

    • mugwump
      Posted Jan 18, 2009 at 11:11 AM | Permalink | Reply

      Re: Steve McIntyre (#34),

      oh dear

    • Gregor
      Posted Jan 20, 2009 at 5:25 AM | Permalink | Reply

      Re: Steve McIntyre (#34), I agree with Jeff (post #47 above) to the extent that I find a lot more fascinating and illuminating content in the subsequent discussions, particularly the #34 I chose to address this comment to…

  29. PaulM
    Posted Jan 18, 2009 at 11:01 AM | Permalink | Reply

    Re #25, #16 steve, I wonder how many readers understand what you are talking about – what is meant by ‘voodoo correleation’, ‘Mannomatic methods’ and ‘ex post selection of proxies by correlation’. If your blog is to have impact beyond a small clique, it is necessary to explain this clearly. Forgive me if I missed it, but I don’t recall you doing this.
    As I understand it (please correct me if I’m wrong) The Mann et al process is essentially:
    a. Choose a large number of ‘proxies’ (though they might as well be random number sequences) covering say 1000 yrs.
    b. ‘Screen’ the proxies, ie only pick those that correlate with the instrumental record over the last 100 years.
    c. Plot the average of those ‘proxies’ that pass the screening test – this guarantees you a hockey stick, because they will average out roughly to the instrumental record over the last 100 years and zero before then.
    This is obvious of course (except to Michael Mann and his co-authors and the referees of the paper) and can be shown by a simple code using random walks, as shown by Lubos.

    • Posted Jan 18, 2009 at 12:07 PM | Permalink | Reply

      Re: PaulM (#35),

      Plot the average of those ‘proxies’ that pass the screening test – this guarantees you a hockey stick, because they will average out roughly to the instrumental record over the last 100 years and zero before then.

      ..and if you want cooling trend, you need to fix PC1. Interesting story as well.

  30. jae
    Posted Jan 18, 2009 at 11:05 AM | Permalink | Reply

    Paul M: I think you also need a “few good proxies” to get a good HS.

  31. Gerald Machnee
    Posted Jan 18, 2009 at 11:57 AM | Permalink | Reply

    Let’s see if I got this right. You need a Ph.D. to write this. Acceptance guaranteed.
    What do you need do get a sound critique accepted?

  32. Steve McIntyre
    Posted Jan 18, 2009 at 12:34 PM | Permalink | Reply

    #41. The sociology is interesting. Psrt of the sociology is surely that specialist climate scientists stand by like bumps on logs (the Silence of the Lambs in an earlier post). For example, Peter Brown knows (because we discussed it here with him) that Mann used Brown’s drought proxies upside down – combining this with cherry picking. But Peter Brown doesn’t have the faintest interest in picking a fight with Mann.

    Or Dominic Fleitmann, who turned up here for a comment or two. Ironically it’s Fleitmann’s series that is opportunistically used in two different orientations.

    • Joe Crawford
      Posted Jan 18, 2009 at 2:07 PM | Permalink | Reply

      Re: Steve McIntyre (#42),
      It’s just that cavalier attitude, not caring whether ones peers get it right or wrong, that could eventually put a lot scientists out of business. I agree with most on this blog in that we desperately need to get the science right.

      snip – one of your words refers to a topic strictly not allowed here

  33. braddles
    Posted Jan 18, 2009 at 3:34 PM | Permalink | Reply

    Don’t get me wrong, I am a great admirer of this blog and read it every day, but this post is a great example of why this blog has less influence than it should. The opening sentence, as PaulM mentions, would be baffling to an outside reader, who may not read on, and the main finding, which seems quite dramatic to me, is found in the comments (#5) not in the blog entry itself.

    I suspect that the principals on the other side of the debate will be happy to ignore this unless something serious is done to publicise it, and nothing will come of it.

  34. Joe Crawford
    Posted Jan 18, 2009 at 4:05 PM | Permalink | Reply

    “snip – one of your words refers to a topic strictly not allowed here”
    Oops, sorry Steve. I fully understand your position on that one. My bad!

    Joe

  35. Gregor
    Posted Jan 20, 2009 at 5:26 AM | Permalink | Reply

    Oops, it was #45, not #47… my bad…

  36. Mark T.
    Posted Jan 20, 2009 at 10:10 AM | Permalink | Reply

    Isn’t the point of a blog, or at least one of the points, to elicit relevant comments from readers? Maybe a thought is stuck and needs some prodding from the masses to complete it?

    Mark

  37. Deep Climate
    Posted Jan 20, 2009 at 2:15 PM | Permalink | Reply

    There seems to be some confusion among readers about this post. So I believe the following graph is a useful aid (and could even be usefully incorporated into the original post).


    Essentially Steve is discussing the validation procedure in Mann et al 2008, presumably the NH CPS reconstruction for period 1950-1995 (last part of graph), which in turn is based on proxy calibration to instrumental temperature in the period 1850-1949 (“late miss” in Steve’s terminology).

    In case anyone wants to refer to the SI or supplementary data, they are available here.

    SD1.xls contains the “passing” values of r of each proxy for the three periods (1850-1995, 1896-1995, 1850-1949). The 1950-1995 reconstruction is based on the proxies found to be significant in the 1850-1949 period.

  38. Deep Climate
    Posted Jan 20, 2009 at 2:46 PM | Permalink | Reply

    Here are some initial thoughts about the “opportunistic” use of the Socotra dO18 proxy. It does seem to me that the greatly varying correlation of this proxy, depending on the period chosen, could argue against its inclusion in paleoclimate reconstruction.

    But what would be the implication of its exclusion for validation? To me, it seems that removal of the proxy would improve the validation scores, at least for the 1950-1995 recon. Why? Well, since the overall correlation is strongly positive, it is likely that the correlation in the 1949-1995 period was also positive. But as Steve has noted, in that reconstruction, the proxy was used in a negative orientation based on 1850-1949 calibration.

  39. Deep Climate
    Posted Jan 20, 2009 at 9:26 PM | Permalink | Reply

    More generally, perhaps the CPS reconstruction would benefit from a screening procedure that would weed out the proxies with correlations that fluctuate wildly over the instrumental period (e.g. specify a maximum difference in r between the first half and second half of the calibration period as an additional screening criterion).

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 2,895 other followers

%d bloggers like this: