An Example of MBH "Robustness"

In the MBH source code, they apply steps that purport to weight the temperature PCs in their regression calculations proportional to their eigenvalues. Comments on their code say:

c set specified weights on data

c weights on PCs are proportional to their singular values

This is one of two weighting procedures in MBH for the regression, the other being the weighting of the proxy series. Wahl and Ammann tout one of the key results of their study as showing that the MBH algorithm is “robust” against “important” simplifications and modifications. They claimed:

A second simplification eliminated the differential weights assigned to individual proxy series in MBH, after testing showed that the results are insensitive to such linear scale factors.

This claim is repeated in slightly different terms on several occasions:

Additionally, the reconstruction is robust in relation to two significant methodological simplifications-the calculation of instrumental PC series on the annual instrumental data, and the omission of weights imputed to the proxy series.

Our results show that the MBH climate reconstruction method applied to the original proxy data is not only reproducible, but also proves robust against important simplifications and modifications.

Indeed, our analyses act as an overall indication of the robustness of the MBH reconstruction to a variety of issues raised concerning its methods of assimilating proxy data, and also to two significant simplifications of the MBH method that we have introduced.

I’ll discuss this claim in respect to weights on proxies in detail on another occasion, but today will note that it can trivially be seen to be untrue, merely through considering the acknowledged impact of differing weights on the NOAMER PC4. Obviously results are not “insensitive” to the “linear scale factors” (weight) of the PC4. Low weights remove the HS-ness of the result and yield a low RE statistic. So this particular claim is patently false.

However, it is true that the MBH result is “robust” to the use or non-use of weight factors on the temperature PCs. This can be seen through some trivial (though long-winded) linear algebra as follows.

As noted in the previous post, the matrix of unrescaled reconstructed RPCs \tilde{U} can be expressed as follows (using notation from prior post):

\tilde{U}=YP^2 C_{uy}^T   (C_{uy} P^2 C_{uy}^T)^{-1} C_{uu}L

Mann re-scaled each such series \tilde{U} so that its standard deviation (in the calibration period) individually matched the standard deviation of the corresponding temperature PC U . Denoting the matrix of rescaled RPCs by \hat{U} , we then have:

\hat{U}_{,i}= \tilde{U}_{,i}  ||U_{,i}}|| /  ||\tilde{U}_{,i}||  for i=1,…,k

Or stacking:

\hat{U}=\tilde{U} ||U|| / ||\tilde{U}||

where ||.|| applied to a matrix X is the square root of the diagonal of X^TX – which, when divided by \sqrt{N-1} yields the standard deviation and, for these ratio calculations, equivalent results.

The expression for \tilde{U} can be used to expand \hat{U} as follows:

\hat{U}=YP^2C_{uy}^T  (C_{uy} P^2 C_{uy}^T)^{-1}C_{uu}L * diag^{1/2}(U^TU)/ diag^{1/2} (\tilde{U}^T \tilde{U})

We obtain the following long (but easily calculable expression) for \hat{U} by substituting for \tilde{U} :

\hat{U}=YP^2C_{uy}^T  AC_{uu}L * diag^{1/2}(C_{uu})  / diag^{1/2}(LC_{uu}ABAC_{uu}L)

where A= (C_{uy} P^2 C_{uy}^T)^{-1} and B= (C_{uy}P^2 C_{yy}P^2 C_{uy}^T)

In the 1-dimensional case (one reconstructed temperature PC), these long matrix expressions reduce to a scalar and a very simple expression – as I discussed a long time ago. Even in the multiple PC cases, because the normalization is being done one by one and L is a diagonal matrix, one can apply the simple identity:

||LXL|| = ||L|| ||X|| ||L|| = L ||X||   where ||.|| is the diagonal matrix from the square root of the diagonal.

Thus all uses of the weight matrix L cancel out in the above expression yielding:

\hat{U}= YP^2 C_{uy}^T   AC_{uu}  diag^{1/2}(C_{uu})/ diag^{1/2} (C_{uu} ABA C_{uu}  )

Because the matrix L cancels out because of the underlying linear algebra, one would expect that any Wahl and Ammann “experiments” with varying this procedure would be “robust” to this particular variation.

29 Comments

  1. Stan Palmer
    Posted Apr 5, 2008 at 2:59 PM | Permalink

    Am I correct in assuming that Mann provides a procedure that does nothing and that his colleagues do not understand this.

  2. Vic Sage
    Posted Apr 5, 2008 at 4:03 PM | Permalink

    I’m also curious about how likely it is that Mann’s hockey team doesn’t even realize that this is a problem.

    But I’m sure it doesn’t matter and everybody should just move on.

    BTW, nice catch.

  3. henry
    Posted Apr 5, 2008 at 4:36 PM | Permalink

    Closing parentheses on last equation?

  4. Michael Jankowski
    Posted Apr 5, 2008 at 4:44 PM | Permalink

    Macbeth comes to mind:

    “..it is a tale
    Told by an idiot, full of sound and fury,
    Signifying nothing.”

  5. Pat Keating
    Posted Apr 5, 2008 at 5:55 PM | Permalink

    ..a poor player, who struts and frets his hour upon the stage, and then is heard no more.
    (From the same soliloquy, just before your quotation.)

  6. Anthony Watts
    Posted Apr 5, 2008 at 10:14 PM | Permalink

    Steve,

    check your email

    Anthony

  7. Robinedwards
    Posted Apr 6, 2008 at 3:56 PM | Permalink

    Please, what must I do to be able to submit a comment?

    Have tried with two longish contributions and one short one, but all failed :-((

    Robin

  8. Robinedwards
    Posted Apr 6, 2008 at 4:00 PM | Permalink

    However, a short attempt has just loaded successfully. Mysterious – must be that my others were too long.

    Robin

  9. Pedro S.
    Posted Apr 6, 2008 at 4:20 PM | Permalink

    OT: The Shakespeare quotation by Michael Jankowski and Pat Keating is from King Lear ;-)

  10. Pat Keating
    Posted Apr 6, 2008 at 4:25 PM | Permalink

    9 Pedro
    Wrong king. Macbeth, but he wasn’t yet king at the time he said it.

  11. Phil.
    Posted Apr 6, 2008 at 5:24 PM | Permalink

    Re #10

    Yes he was king, it was said when he received the news of his wife’s death as the English backed army attacked his castle at Dunsinane.

  12. Pat Keating
    Posted Apr 6, 2008 at 6:51 PM | Permalink

    11
    After I posted, I realized that I had confused that soliloquy with the “If ’tis done,..” soliloquy, so your post was not unexpected.

    However, wasn’t it “My life has fallen into the sere, the yellow leaf” at the point you are talking about? I thought “Tomorrow” was earlier than Dunsinane.

  13. steven mosher
    Posted Apr 6, 2008 at 6:58 PM | Permalink

    at some point between the matrix algebra and the transition to the bard I got lost.
    And I’m tangent man!

  14. Pat Keating
    Posted Apr 6, 2008 at 7:05 PM | Permalink

    13 Steve M
    Yes, you are. But we can try to compete….
    I think the tele-connection is “tale told be an idiot”.

  15. Posted Apr 6, 2008 at 8:50 PM | Permalink

    Re Robin, #7, your longish post is under the related “Squared Weights” thread: http://www.climateaudit.org/?p=2962#comment-232458“>.

  16. DKH
    Posted Apr 6, 2008 at 8:55 PM | Permalink

    It might be that I’m not reading the notation correctly, but it looks like the second to last equation equates a scalar and a matrix. It seems to me that diag^1/2^(LXL) a number, while L*diag^1/2^(X) is a matrix. I can’t claim to be well-versed in the terminology, so I’m not sure how this affects the math logic.

    Steve: EVerything is a matrix in this expression. The notation is a little idiosyncratic as diag^{1/2}(X) is the diagonal matrix consisting of the square roots of diag(X). EVerything in this line is a diagonal matrix.

  17. Phil.
    Posted Apr 6, 2008 at 9:04 PM | Permalink

    Re #12

    However, wasn’t it “My life has fallen into the sere, the yellow leaf” at the point you are talking about? I thought “Tomorrow” was earlier than Dunsinane.

    No, it’s Act V, scene v in Dunsinane:

    ” Seyton The Queen, my lord, is dead.
    Macbeth She should have died hereafter;
    There would have been a time for such a word.
    To-morrow, and to-morrow, and to-morrow,
    Creeps in this petty pace from day to day
    To the last syllable of recorded time;
    And all our yesterdays have lighted fools
    The way to dusty death. Out, out, brief candle!
    Life’s but a walking shadow, a poor player
    That struts and frets his hour upon the stage
    And then is heard no more. It is a tale
    Told by an idiot, full of sound and fury,
    Signifying nothing.”

    The line about the “sere, the yellow leaf” comes 2 scenes earlier.

  18. DKH
    Posted Apr 6, 2008 at 9:22 PM | Permalink

    Re #16:

    Didn’t notice the update. It’s not the second to last equation anymore; it is the second to last before the note.

    Steve: I’m tidying the nomenclature a little offline.

  19. DKH
    Posted Apr 6, 2008 at 11:26 PM | Permalink

    Thanks for the notes; I understand now (I think).

  20. Joe Soap
    Posted Apr 7, 2008 at 10:28 AM | Permalink

    Steve, you seem to be implying that the product of symmetric matrices must be symmetric. Not so, I’m afraid – but I haven’t delved far enough into this piece to see if it affects the result.
    e.g. A = {{1,3},{3,2}}, B={{7,4},{4,9}} AB = {{19,29},{31,30}}

    Steve:
    Thanks. Brain cramp on my part. I was using diagonal matrices here and the results rely on properties of diagonal matrices. I’ve corrected the text accordingly.

  21. wkkruse
    Posted Apr 7, 2008 at 12:19 PM | Permalink

    Anybody, When GISS does their monthly update to the historical reports, do they update the entire history or do the just tack on the latest month’s result to the previous month’s record?

    Thanks

  22. Posted Apr 7, 2008 at 12:47 PM | Permalink

    wkkruse:
    When I visited the monthly data in march, several previous monthly values had changed. So, at least occasionally, they do more than just tack on the new month’s value. I think some HadCrut numbers also change. But I’ve only checked once, so it’s possible that the change at HadCrut was me looking at different files.

    In any case, if you are using this data, it seems wise to check for the latest data fairly regularly. Don’t assume you can just go in, add the most recent value, and use that only.

  23. Stan Palmer
    Posted Apr 7, 2008 at 1:39 PM | Permalink

    re 22,21

    In any professionally managed engineering program, no previous version of data would just be discarded or overwritten. The older version would be archived and a change notice issued detailing the reason(s) for the update. This change notice would have to be approved by an authorized official before any modification to the current data could be done.

    Perhaps GISS is doing this and has the older versions safely archived with their change notices.

  24. bobby B
    Posted Apr 7, 2008 at 2:39 PM | Permalink

    It might be that I’m not reading the notation correctly, but it looks like the second to last equation equates a scalar and a matrix. It seems to me that diag^1/2^(LXL) a number, while L*diag^1/2^(X) is a matrix.

    I have absolutely NO idea what you just said. You are taking this entire controversy to a place outside of my areas of expertise, and thereby marginalizing the worth of my contributions to the discussion. As I am a person of worth and intellect and moral correctness, to exclude my input in favor of a technocrat’s formulas is entirely wrong, and works to show only that you have valued and weighted the wrong inputs and influences in your results-driven propogandizing.

    Clearly, your ad hominum, math-centric attacks on Mr. Mann’s work cannot detract from the essential TRUTH of what he tells us. Mann shows us our evil, and your attempt to use these outlandish, inscrutable MATH problems to discredit him fails to address, much less impeach, the core of his work.Mann is post-math!

  25. Sam Urbinto
    Posted Apr 7, 2008 at 4:46 PM | Permalink

    I have the Jan GHCN+ERSST numbers, and the ones now up for Apr. The anomaly for a number of months in 2007 are different.

    Jan version: 2007 87 63 59 66 55 53 51 56 50 55 46 39 Year 57
    Apr version: 2007 86 63 60 64 55 53 51 56 50 55 49 40 Year 57

    I didn’t check other years.

  26. Sam Urbinto
    Posted Apr 7, 2008 at 5:30 PM | Permalink

    Rather odd; the first quarter is unaffected, the second quarter .02 lower (average 0 for the quarter) the third quarter unaffected, and the fourth quarter .04 warmer (average .1 for the quarter) and the year unaffected.

    Why bother; The climate only matters on 30 year time scales.

    Somebody ought to tell the people working with the monthly and yearly anomalies to not bother with this weather stuff.

  27. Sam Urbinto
    Posted Apr 7, 2008 at 5:33 PM | Permalink

    OOOps, I meant .01 for the quarter 3rd.

  28. steven mosher
    Posted Apr 7, 2008 at 8:10 PM | Permalink

    re 27. you made a mistake. you made a mistake. you made a mistake.

    Its my tamino impression.

  29. Steve McIntyre
    Posted Apr 8, 2008 at 10:11 PM | Permalink

    A reader posted this nice characterization of this particular MBH “robustness” at his blog.

    In his analysis of his hockey stick temperature reconstruction, Michel Mann claimed that his results were robust to changes in certain weighting factors. Humorously, Steven McIntyre demonstrates that it is robust because when you do the math, the weighting factors actually cancel out of all the equations. In effect, Mann was saying that y =3x/x gives the answer “3” robustly for all values of x (well, except zero). True, but scientifically meaningless. But worrisome when a scientist has to run numerous simulations to discover the fact. I presume he thought his weighting factors were actually doing something in his model.

    “3x/x” captures the situation exactly. To be fair, Mann himself did not claim that the changes were “robust” to weighting factors; his claims were to things like all dendroclimatic indicators; he himself never mentioned that he used weighting factors. This particular issue was raised by Wahl and Ammann. Again to be precise, they didn’t make the claim of robustness about the weighting factor L (using the nomenclature of the post) where the claim is trivial but true, but about the weighting factor P where the claim is false. This is the Team after all.

One Trackback

  1. […] To MBH’s credit, they at least employed a form of the CCE calibration approach, rather than LNA’s direct regression of temperature on proxies. There are, nevertheless, serious problems with their use (or non-use) of the covariance matrix of the residuals. See, eg, “Squared Weights in MBH98″> and “An Example of MBH ‘Robustness’”.> […]

Follow

Get every new post delivered to your Inbox.

Join 3,419 other followers

%d bloggers like this: