Buell: ‘Castles in the Clouds’

CA reader hfl, who cited Buell’s documentation of the dependence of principal component patterns on shapes, has sent me a scanned pdf version now available here. It concludes by observing that analyses that fail to consider this phenomenon (and there is ample evidence that Steig et al falls into this category) “may well be scientific level with the observations of children who see castles in the clouds”. [Update: Here is a rendering by a CA reader (now using Lucy Skywalker hockeystick version)]

Here is hfl’s summary:

Forgive me if this has been discussed in past threads on PCA (although I couldn’t find it in a quick search of the site), but it’s worth noting that the issue of principal component pattern dependence on domain shape is (or was) well known within the atmospheric sciences community. It was first documented by C. Eugene Buell in a 1975 paper published in the Proceedings of the Fourth Conference on Probability and Statistics in Atmospheric Sciences. Buell’s work focused on square/rectangular domains, particularly because the latter approximated the shape of the conterminous U.S. This was followed by a second paper at the Sixth P&S Conference in 1979 which states: “When a region with well defined boundaries is concerned, the EOF’s computed over this region are expected to be very strongly influenced by the geometrical shape of the region and to a large extent independent of where the region is located. As a consequence, the interpretation of the topography of the EOF’s in terms of geographical area and associated meteorological phenomena should be looked on with suspicion unless the influence of the effect of the shape of the region has been completely accounted for. Otherwise, such interpretations may well be on a scientific level with the observations of children who see castles in the clouds.”

Buell’s work generated considerable discussion within the atmospheric sciences literature because PCA (or, as they referred to it, EOF [empirical orthogonal function]) analysis was in widespread use. Mike Richman, in a paper published in the Journal of Climatology (1986) entitled “Rotation of Principal Components” made the case that component rotation appears to eliminate the problem of domain shape dependence. Richman’s paper is worth reading in that it reviews and considers nearly all of the important work that had been done using PCA on meteorological and climatological data up to that time, including work by Gerry North and others familiar to CA readers. So these problems have been well documented and understood for a long time. Like so many other elements of meteorology and climatology, though, modern climate science appears to have forgotten some important statistical insights produced by it’s own practitioners . . . indeed, some of the practitioners themselves appear to have forgotten what they wrote.

51 Comments

  1. Posted Feb 26, 2009 at 3:31 PM | Permalink | Reply

    With all due respect, I find the “castles in the clouds” statement somewhat funnier than “done on the back of an envelope.”

    It’s interesting to see older papers like this with hand-drawn graphs compared to modern equivalents with computer-generated plots which are simultaneously more accurate (not to imply scientific accuracy or fitness) and aesthetically pleasing. Science sure has changed.

  2. MichaelJ
    Posted Feb 26, 2009 at 3:43 PM | Permalink | Reply

    The doesn’t contain anything as poetic as “castles in the clouds”, but here’s the full paper on the problem as recognized in population genetics. Instead it uses phrases like “mathematical artifacts” and “providing a’null’ expectation against which observed PCs may be compared and contrasted”.

    http://www.eeb.ucla.edu/Faculty/Novembre/NovembreStephens2008.pdf

    • Steve McIntyre
      Posted Feb 26, 2009 at 3:59 PM | Permalink | Reply

      Re: MichaelJ (#2),

      nice reference. I think someone might have mentioned this paper in a prior thread last year. It looks a bit familiar.

      • MichaelJ
        Posted Feb 27, 2009 at 8:34 AM | Permalink | Reply

        Re: Steve McIntyre (#4), Steve, I posted this last year, as well. I figured it was worth are re-post if for no other reason than the irony of appearing in a Nature publication.

  3. jeez
    Posted Feb 26, 2009 at 3:56 PM | Permalink | Reply

    Sorry, a personal note/funny coincidence. My brother coauthored papers with the those population genetic researches.

  4. jeez
    Posted Feb 26, 2009 at 4:07 PM | Permalink | Reply

    and I definitely need to proof my posts.

    • Dave Dardinger
      Posted Feb 26, 2009 at 4:18 PM | Permalink | Reply

      Re: jeez (#5),

      I wonder if someone could make a nice version of the CA masthead with a castle among the clouds? Perhaps with a signboard reading “Team RC” and a BBQ with a weather station nearby.

      • Jeff Alberts
        Posted Feb 26, 2009 at 11:32 PM | Permalink | Reply

        Re: Dave Dardinger (#6),

        Be careful what you wish for, Dave ;)

        http://jalberts.net/images/CloudCastle.jpg

        Didn’t have a BBQ model, but I made a crude MMTS, and placed in the best possible location for optimal temperature measurements.

        • RomanM
          Posted Feb 27, 2009 at 5:57 AM | Permalink

          Re: Jeff Alberts (#10),

          The RC flag is a nice touch. :D

        • Dave Dardinger
          Posted Feb 27, 2009 at 7:40 AM | Permalink

          Re: Jeff Alberts (#10),

          Thank you! I suppose you could have some smoke rising near the MMTS and leave the BBQ to our imagination. Unfortunately there don’t seem to be any windows nearby to have an air conditioner in. The arm on the flag does have a nice hockey stick shape, though, I notice.

  5. bernie
    Posted Feb 26, 2009 at 4:29 PM | Permalink | Reply

    If this shape dependency holds, how is it accounted for in the determination of the AAO which is part of Steig et al’s finding? see link and associated method. Surely this must be accounted for in the generation of the AAO?

  6. Willis Eschenbach
    Posted Feb 26, 2009 at 7:49 PM | Permalink | Reply

    MichaelJ, nice paper. I particularly enjoyed this part (emphasis mine):

    In summary, we have shown that (i) when analyzing spatial data,
    PCA produces highly structured results relating to sinusoidal functions
    of increasing frequency; and (ii) insofar as PCA results depend on the
    details of a particular dataset, they are affected by factors in addition to
    population structure, including distribution of sampling locations and
    amounts of data
    . Both these features limit the utility of PCA for
    drawing inferences about underlying processes, a fact previously noted
    in climatology (for example, ref. 17).

    Reference 17 is Richman, M.B. Rotation of principal components. J. Climatol. 6, 293–335 (1986).

    I can’t find a copy online, but the abstract says:

    Recent research has pointed to a number of inherent disadvantages of unrotated principal components and empirical orthogonal functions when these techniques are used to depict individual modes of variation of data matrices in exploratory analyses. The various pitfalls are outlined and illustrated with an alternative method introduced to minimize these problems via available linear transformations known as simple structure rotations. The rationale and theory behind simple structure rotation and Procrustes target rotation is examined in the context of meteorological/climatological applications. This includes a discussion of the six unique ways to decompose a rotated data set in order to maximize the physical interpretability of the rotated results.

    The various analytic simple structure rotations available are compared by a Monte Carlo simulation, which is a modification of a similar technique developed by Tucker (1983), revealing that the DAPPFR and Promax k = 2 rotations are the most accurate in recovering the input structure of the modes of variation over a wide range of conditions. Additionally, these results allow the investigator the opportunity to check the accuracy of the unrotated or rotated solution for specific types of data. This is important because, in the past, the decision of whether or not to apply a specific rotation has been a blind decision. In response to this, a methodology is presented herein by which the researcher can assess the degree of simple structure embedded within any meteorological data set and then apply known information about the data to the Monte Carlo results to optimize the likelihood of achieving physically meaningful results from a principal component analysis.

    So the problem has been recognized for over twenty years … but in the meantime, the Team has moved on. Kinda reminds me of the Steve Martin movie where he’s in a restaurant and says something like “Hey, you can’t fool me. This wine is twenty years old! Bring me some of the new stuff!”

    w.

    • AnonyMoose
      Posted Feb 26, 2009 at 10:07 PM | Permalink | Reply

      Re: Willis Eschenbach (#8),
      Hey, that’s twenty year old research. That’s obsolete, nobody needs those numbers anymore.

      Incidentally, this has interesting back-of-the-envelope hand drawn graphs: thisisindexed.com

    • bernie
      Posted Feb 27, 2009 at 7:10 AM | Permalink | Reply

      Re: Willis Eschenbach (#8), Willis:
      It is beyond me, but you may be able to tell whether the current method for measuring the AAO takes the issues associated with PCA into consideration.

  7. mugwump
    Posted Feb 27, 2009 at 9:44 AM | Permalink | Reply

    Is it just me? There really is a castle in those clouds…

    Gotta stop spending so much time with the kids. Dang this economy.

  8. bender
    Posted Feb 27, 2009 at 10:02 AM | Permalink | Reply

    No slight to hfl in the OP, but Richman (1986) has been mentioned before – in the context of Wilson’s dendro work:
    http://www.climateaudit.org/?p=1285
    It is also cited in the paper mentioned by MichaelJ in #12 and it contains the start of the answer to bernie’s #12. By finding a “suitable” rotation you emphasize “simple structure” and your eigenvectors move away from being a parody parade of harmonics. That’s the theory. Read it.

    • bernie
      Posted Feb 27, 2009 at 11:35 AM | Permalink | Reply

      Re: bender (#16), Nice tip. Personally, and rightly or wrongly, I have always used a rotation to make sense of a factor. With survey data it is very difficult to interpret the factors any other way because the definition of the factors has to somehow reflect the questions that were asked. (We are plagued however by our own forms of autocorrelation such as leniency or halo).

  9. Jeff Alberts
    Posted Feb 27, 2009 at 10:12 AM | Permalink | Reply

    Wow, I made the OP, lol.

    FYI, to see the pic in full size, in Firefox, right-click the image and select View Image, then you can zoom in and see more detail.

    • RomanM
      Posted Feb 27, 2009 at 5:54 PM | Permalink | Reply

      Re: Jeff Alberts (#17),

      FYI, to see the pic in full size, in Firefox, right-click the image and select View Image, then you can zoom in and see more detail.

      Without using the RegEM filter in Photoshop? ;)

      Re: bernie (#33),

      The world is a complex place and climate factors have been known to affect different geographical cofigurations (high altitude-low altitude, continental coast-interior) in different ways. Rotation of axes can, in some cases emphsize those relationships.

      Look at a simple (non-climate) example. Suppose you graph a bunch of peoples’ heights versus their weights. Assume that you scaled them in such a way that most of the subjects fall close to a 45 degree line in the graph going through the origin. Now rotate the coordinates 45 degrees retaining orthogonality of the axes. It is not inconceivable that the new axes represent somewhat different factors than just height and weight. One could label one axis as “size” which differentiates larger and smaller persons, and the other might be viewed as “body shape” (e.g. taller thinner persons or heavier and shorter persons) in the population from which you took the subjects.

      It is not inconceivable that similar considerations could apply here. However, without reasonable statistical considerations AND demonstrated physical (non-teleconnective) causes, it would be no more than arm-waving and pointing at the castles in the skies.

      As a side note. when I was teaching, I used to call it “horsies in the clouds”. I suppose that my students weren’t as mature as as the crowd inhabiting CA…

  10. kim
    Posted Feb 27, 2009 at 10:21 AM | Permalink | Reply

    So will Prof. Steig reveal his data and confirm or collapse your conjectures, or will he withdraw the paper and conceal his castle in clouds forever? Decisions. Decisions.
    ==============================================================

  11. Posted Feb 27, 2009 at 11:52 AM | Permalink | Reply

    Castle in the Air with Hockey Stick

  12. bender
    Posted Feb 27, 2009 at 12:34 PM | Permalink | Reply

    Jolliffe (1987):

    Domain shape dependence
    The predictable patterns which often occur for PCs in climatological studies are due mainly to the spatial correlation usually present in such data, with domain shape acting as a significant, but secondary, factor.

    Bold mine.

    • bender
      Posted Feb 27, 2009 at 12:44 PM | Permalink | Reply

      Re: bender (#22),
      Buell (1975):

      It is evident from these examples that the shape of the boundary of the data points to which the correlation coefficients (or covariances) of the matrix problem belong is a dominant factor in determining the topography of the proper functions (EOFs) of its solution.

      Bold mine.

      • bender
        Posted Feb 27, 2009 at 12:46 PM | Permalink | Reply

        Re: bender (#23),
        Richman (1987):

        The exact importance of the maximum variance and orthogonality restrictions to domain shape is open to debate.

        Bold mine.

  13. bender
    Posted Feb 27, 2009 at 12:50 PM | Permalink | Reply

    And this is for Dr. Steig:

    Richman (1987):

    Whichever final solution is applied (rotated, unrotated), the onus of proof of physically meaningful modes (if the PCs are to be interpreted) lies squarely with the analyst.

    This is one of Steve M’s main contentions.

  14. bender
    Posted Feb 27, 2009 at 1:16 PM | Permalink | Reply

    Dr. Jolliffe,
    May I invite Drs. Steig & Mann & Schmidt to spend a month in your lab this summer?
    Thank you for entertaining this request.
    bender

  15. Steve McIntyre
    Posted Feb 27, 2009 at 2:12 PM | Permalink | Reply

    Reflecting a little on Buell and Richman, I think that there are some important aspects of the relationship that have been reported here than are not present in the earlier literature.

    I haven’t parsed the articles in detail and do not exclude the possibility that I’ve missed something relevant, but in a first read, I don’t see that either article identified the form of the patterns as clearly as we’ve pinned down here. We’ve put a functional form on the eigenfunctions, relating them to Chladni patterns (and other homomorphic structures). I didn’t see that in the earlier literature.

    Given the present (more informed??) interpretation of these patterns as Chladni-type patterns, I’m not at all sure that rotation of principal components is an omnibus cure, so the specific recipes in Richman would need to be parsed. I would place more weight on the recognition of the problem, than the prescription.

    bender, as to the bolded item in Jolliffe’s 1987 comment, I doubt that Jolliffe would preclude the possibility of domain structure being dominant in some circumstances or that he would disassociate himself from the statistical requirement that the authors show that their eigenvectors are not mere castles in the clouds (as in the nice Richman quote that you made on this topic.)

  16. bernie
    Posted Feb 27, 2009 at 2:34 PM | Permalink | Reply

    Steve:
    If the pattern of the non-rotated PCs are essentially associated with the spatial autocorrelation, in what ways would a rotation actually reveal some underlying structure reflective of physical processes as opposed to spatial relationship. Shouldn’t one or more of the rotated factors actually reflect the spatial relationships? For example, if all that was there was the spatial relationship then they should appear as the rotated factors? But I am treading at the edge of my current ability in this arena – which may be obvious by my questions.

  17. bernie
    Posted Feb 27, 2009 at 9:12 PM | Permalink | Reply

    RomanM:
    That’s how I recall it. I guess the question I have is the extent to which the AAO is a statistical artifact. Given the horsepower available, it is hard for me to imagine that those responsible are not aware of these spatial correlation issues.

    • bender
      Posted Feb 27, 2009 at 10:30 PM | Permalink | Reply

      Re: bernie (#35),
      Of course they are aware of it. The question is why, given the known issues, do they make claims that they don’t bother to prove. And I think the answer is simply Nature’s ultrastrict word limit. Nature has no business fast-tracking speculative climatology papers. Save that for the specialist climatology literature.
      .
      The story here is not so much what Steig et al did inside the paper. It’s what the Nature reviewers didn’t do, and, even more, what the Nature media people did to spin the story. Steig et al. went along for the ride. And why not?

  18. Steve McIntyre
    Posted Feb 28, 2009 at 7:59 AM | Permalink | Reply

    Reflecting a little on Buell and Richman, I think that there are some important aspects of the relationship that have been reported here than are not present in the earlier literature.

    I haven’t parsed the articles in detail and do not exclude the possibility that I’ve missed something relevant, but in a first read, I don’t see that either article identified the form of the patterns as clearly as we’ve pinned down here. We’ve put a functional form on the eigenfunctions, relating them to Chladni patterns (and other homomorphic structures). I didn’t see that in the earlier literature.

    A CA reader writes offline:

    Steve,

    Have a look at the literature on fluid dynamics.

    Turbulence, Coherent Structures, Dynamical Systems and Symmetry

    Holmes was a great teacher. I seem to remember that the eigenfunctions of homogeneous turbulence can
    be shown to be sines and cosines. This is intuitive because homogeniety implies translational invariance:
    you move from point to point and everything looks the same. Sines and Cosines (or more formally complex exponentials)
    are translationally invariant eigenfunctions. Translating just introduces a constant phase factor Psi(x)=exp(I*k*x)
    Psi(x+dx)=Exp(i*k*dx)*Exp(I*k*x)

    In other words the eigenfunctions for KL/PCA are determined by the symmetry properties of the system under decomposition.

    Other names similar if not identical techniques go by:

    Method of Snapshots (image processing, see sirovich http://camelot.mssm.edu/publications/larry.html particularly#32 on his publications list)

    Karhunen-Loeve

    I think Holmes and Lumley also discuss cutting off the decomposition usually at 90-95% of the variance for turbulence.
    Sorry I can’t give you better direction now because my copy is at the office.

  19. Jeff Alberts
    Posted Feb 28, 2009 at 12:49 PM | Permalink | Reply

    Steve, sorry to post this here, but didn’t have an email address for you.

    I rendered up a new header graphic for the site, if you care to use it.

    http://jalberts.net/images/CA_Banner_JA.jpg

    • Steve McIntyre
      Posted Feb 28, 2009 at 1:30 PM | Permalink | Reply

      Re: Jeff Alberts (#38), cute, but that’s making a little more of a statement than I want to make. But I like the idea – the present header’s been fine but, after a while, it’s a little nondescript.

      • Jeff Alberts
        Posted Feb 28, 2009 at 1:53 PM | Permalink | Reply

        Re: Steve McIntyre (#39), Well, I wasn’t really reading much into it, just making something visually appealing and at least partially relevant. ;)

    • MrPete
      Posted Mar 1, 2009 at 7:24 AM | Permalink | Reply

      Re: Jeff Alberts (#38), (and for anyone interested in header graphics) — it’s best when the header can seamlessly repeat (one way or another). People have different screen widths; it’s nice when the header “works” for everyone.

      • Jeff Alberts
        Posted Mar 1, 2009 at 3:43 PM | Permalink | Reply

        Re: MrPete (#45),

        I agree, MrPete. However, I was using Steve’s current header graphic as a template for width and height (copied the image and used the raw size as opposed to the visible size in the browser). The width is greater than 1600 pixels, so while you may have a screen res wider than that, is it likely that you’d have your browser taking up the entire width? A lot of sites also have fixed width content areas with fixed width headers, so it’s not a given by any means. Anyway, it was a quick and dirty…

  20. Posted Feb 28, 2009 at 4:09 PM | Permalink | Reply

    I did a couple of videos of the satellite temperature trend because I wanted to see how PC3 changed the temp anomaly. It is visible in the anomaly video, you can see the oscillations back and forth along PC2 and up and down from PC1 and a sudden left-right motion in PC3. I then did a simple enhancement to make it more visible, what I didn’t expect to find was that the actual satellite data from Steig’s conclusion has a temperature reversal from the eastern rim temps to the center of the antarctic.

    Unless I’m missing something, you will probably be surprised.
    http://noconsensus.wordpress.com/2009/02/28/a-little-bit-of-magic/

    • RomanM
      Posted Feb 28, 2009 at 5:11 PM | Permalink | Reply

      Re: Jeff Id (#41),

      Nice job, Jeff! That’s a great way to show 3,305,400 numbers and see what they tell you.

      You could become a statistician if you work at it. ;)

      • Posted Feb 28, 2009 at 9:52 PM | Permalink | Reply

        Re: RomanM (#42),

        Quite a compliment, another method to bore my friends to death. :)

        When I see a semicircle of temps different from the interior it bothers me a bit. The ocean certainly would dampen trends, I’ve lived on both sides of lake Michigan for years at a time. On the west, there’s not usually much effect but on the east wow! It mau be possible with circular weather patterns that the edge of the antarctic would all have similar temps, I’m not sure. I’m guessing the muted temparature half the way around the antarctic could actually exist in the satellite data overlaying the surface stations. This could cause regem to weight the interior with a heavier or even inverted trend from what the surface stations show (e.g. if the trend in the interior is stronger in the sat data, trends at the coast will be amplified in the interior. Since these surfacestations by my above assumptions are therefore in a high gradient area, small adjustments of even things like the 50Km gridsize or .5 degrees of latitude could make a real difference in result.

        Either way, it’s interesting to see the PC’s at work. I think the reason Steig chose not to describe the physical meaning of the 3rd PC may be because it doesn’t exist pre-1982. Without the sat data it’s just speculation of course.

  21. Posted Mar 1, 2009 at 2:21 PM | Permalink | Reply

    As requested, Steve, versions
    Cloud Castle 01
    Cloud Castle 02
    Cloud Castle 03
    Cloud Castle 04
    Hope it’s what you want.

  22. Posted Mar 3, 2009 at 3:08 AM | Permalink | Reply

    Alan Stern, NASA’s “hard-charging” and “reform-minded” Associate Administrator for the Science Mission Directorate, resigned on March 25, 2008,[44] to be effective April 11, after he ordered funding cuts to the Mars rovers and Mars Odyssey that were overturned by NASA Administrator Michael D. Griffin. The cuts were intended to offset cost overruns for the Mars Science Laboratory. Stern, who served for nearly a year and has been credited with making “significant changes that have helped restore the importance of science in NASA’s mission.”, says he left to avoid cutting healthy programs and basic research in favor of politically sensitive projects. Griffin favors cutting “less popular parts” of the budget, including basic research, and Stern’s refusal to do so led to his resignation.

  23. Reil
    Posted Mar 3, 2009 at 3:09 AM | Permalink | Reply

    In response to the internal review, policies at NASA would be changed in a variety of ways: Flight surgeons would be present during the pre-mission suit-up activities, flight surgeons would receive additional training in psychiatric evaluation, and although there was an unofficial code of conduct in place, an official “Code of Conduct” would be written up for employees NASA

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 3,132 other followers

%d bloggers like this: