More on von Storch !!

It looks like there is a huge cock-up in the von Storch and Zorita Comment, which completely screws up their simulations. We alluded to a defect in their methodology in our Reply, but I didn’t realize just how big a defect it was.

Von Storch and Zorita said the following:

MM05 performed a Monte Carlo study with a series of independent red-noise series; they centered their 1000 year-series relative to the mean of the last 100 years, and calculated the PCs based on the correlation matrix. It turned out that very often the leading PCs show a hockey stick pattern, even if the data field was by construction free of such structures. This finding was recently confirmed by others (F. Zwiers, personal communication).

We observed the following about this in our Reply.

VZ said that the MM05a Monte Carlo study calculated PCs on the correlation matrix; in fact, following MBH98, we used the decentered data matrix in our simulations. From this, we surmise that VZ calculations were not done by SVD on the decentered data matrix; this will have a material impact on their yield of hockey sticks.

I just realized how "material" the impact of this defect in their implementation of the hockey stick algorithm: it eliminates the effect totally!! Think about it for a moment – if you change the centering of the data set, you don’t change the correlation matrix. So if you do principal components on the correlation matrix of the short-centered data, you get exactly the same eigenvectors (weighting factors) as correctly centered data. You only get different eigenvectors if you do SVD on the short-centered data directly and take the left matrix, as Mann did. When you re-read the above VZ quotation narrowly, they have not shown that they have replicated the algorithm, only that Zweiers had confirmed the effect. Because we were happy that von Storch and Zorita seemed to confirm the Artificial Hockey Stick effect, we assumed that they had somewhat implemented the hockey stick algorithm and didn’t fully worry through this issue in our Reply, beyond the above sentence.

It would have been nice to have fully realized this at the time of our Reply, because von Storch has been nice to us and I would have given him a heads-up on the issue if I’d realized it. On the other hand, we’d asked to see their code to see what they were doing and didn’t get anywhere. We even had to argue with GRL to get the above sentence in. It is possible that the VZ description of their algorithm is inconsistent with their actual algorithm, in which case, the problem with their pseudoproxies is the more germane issue. But, if the VZ source code is consistent with the description in their article, the entire von Storch and Zorita simulation is totally screwed up. This won’t amuse realclimate.

Update (Friday): The following question needs to be explained as well: if the correlation matrices are the same, then isn’t this proving too much – i.e. wouldn’t VZ then get identical results either way? Presumably, in their implementation, they standardized on the short segment, not just short-centering, but also short-scaling. The short-segment standard deviations for each series will differ from the full-series standard deviation. So the correlation matrices are the same, but the weights from the eigenvectors are applied to versions which have been divided by slightly differing standard deviations. It is the difference between these standard deviations that accounts for the very slight difference in the two VZ runs, rather than the impact of the MBH98 method. This slight difference in standard deviations would be a minor glitch, if that were the problem. Indeed, short-segment standardizing in this form would, on the face of it, slightly discriminate against hockey stick shaped series, although the effect is slight. (This is doubtless why MBH98 used something called detrended standard deviations, which do not downweight hockey stick shaped series. The Hockey Team is serious about hockey sticks and every little edge is used.)

Update (Saturday): von Storch has sent me an email saying that they used covariance matrices in their PC calculations and defended his calculations. I’m considering the email and will post a more definitive response after this consideration.


12 Comments

  1. TCO
    Posted Oct 26, 2005 at 1:00 PM | Permalink

    I hope you’ve sent a note to this effect to them, warning them of the potential problem. If they are real scientists they will want to know. (Don’t rely on them reading the site.)

  2. TCO
    Posted Oct 26, 2005 at 1:03 PM | Permalink

    Your questions about their methods (wether right or wrong) points to the inadequacy of relying on written descriptions of algorithms when the situation is this complicated and the math is the critical component of the argument. They should publish the code (preferabbly) or if not, an explicit mathematicl formula, that is the algorithm.

  3. Steve McIntyre
    Posted Oct 26, 2005 at 1:22 PM | Permalink

    I will send von Storch a note.

  4. TCO
    Posted Oct 26, 2005 at 1:54 PM | Permalink

    hang in there, slugger.

  5. Posted Oct 26, 2005 at 7:36 PM | Permalink

    Seems like fun – and perhaps your reply to VZ was faster than it should have been. ;-)

  6. Steve McIntyre
    Posted Oct 26, 2005 at 11:14 PM | Permalink

    Lubos, it wasn’t written quickly. The first draft was in April and it went through multiple iterations.

    I don’t know why I didn’t think of it before – it’s a totally obvious point. AS I mentioned, because VZ endorsed the existence of the Artificial Hockey Stick effect, it was easy to assume that their methods had replicated it and miss the attribution to Zweiers. In fairness to my not noticing the point before, VS and Z obviously didn’t think of it, not did the referees.

    I don’t know what made me think of it now, after not noticing it before.

  7. Posted Oct 27, 2005 at 12:19 AM | Permalink

    Is a cock-up worse than a screw up?

  8. fFreddy
    Posted Oct 27, 2005 at 3:51 AM | Permalink

    Hello, Tim Lambert. Do you have any thoughts on Steve’s analysis and its implications ?

  9. Steve McIntyre
    Posted Oct 27, 2005 at 10:46 AM | Permalink

    Hi, Tim. I was going to include a sentence – saying: where is Tim Lambert when we need him? Not entirely ironically. What do you think of this?

  10. ET SidViscous
    Posted Oct 27, 2005 at 4:03 PM | Permalink

    Not climate but I thought people here would find it interesting.

    http://www.boston.com/news/local/massachusetts/articles/2005/10/27/mit_fires_professor_alleging_falsified_research/

  11. TCO
    Posted Oct 27, 2005 at 11:39 PM | Permalink

    Tim, your blog is more fair than RC. I’ve never had stuff (well except for that one pugnacious drunk one…that you still printed exact same comment when I stated it more stadely) blocked on your sight, but have on RC despite being more adult on the RC sight. b

  12. Steve McIntyre
    Posted Oct 29, 2005 at 9:31 AM | Permalink

    I’ve posted the following update just now: von Storch has sent me an email saying that they used covariance matrices in their PC calculations and defended his calculations. I’m considering the email and will post a more definitive response after this consideration.

    Their Comment indicated that they used “correlation” matrices. If they used covariance matrices, some aspects of the above comments will beed to be modified.

    I might add that we requested source code in order to be able to reconcile results and were unsuccessful. As all of you know, I disagree with obstacles to reconciliation.

Follow

Get every new post delivered to your Inbox.

Join 3,382 other followers

%d bloggers like this: