Phil Stumped at Calculating A Trend

Phil Jones spends much of his time looking down his nose at the heathen, but then confesses to Bob Ward that he is unable to calculate a trend on his own, as in this hilarious exchange at Bishop Hill:

I’m not adept enough (totally inept) with excel to do this now as no-one who knows how to is here.

Nor it seems in Matlab, R, ODL, Fortran or any other language. No wonder that he regarded someone who could calculate principal components (like Mann) as a sort of computational prodigy.

Last year, Phil was ranked one of England’s top 100 scientists. Just imagine the ranking that he could have achieved if he knew how to calculate a trend by himself.

42 Comments

  1. Thomas H
    Posted Nov 23, 2011 at 2:36 PM | Permalink

    In related news. CRUTEM3 code available?

    http://bishophill.squarespace.com/blog/2011/11/23/the-crutem-code.html

    • steven mosher
      Posted Nov 23, 2011 at 6:19 PM | Permalink

      yup

  2. Posted Nov 23, 2011 at 2:37 PM | Permalink

    There are several emails this time which are so incompetent as to leave my jaw in my lap. In particular the discussions of PCA regarding MM and Whall Amman (I always spell their names wrong). Minus a few lucid moments, they don’t know what the heck they are talking about.

    • John
      Posted Nov 24, 2011 at 8:26 AM | Permalink

      Phis needs to go on a Matlab course!!

      • Bengt Abelsson
        Posted Nov 24, 2011 at 9:18 AM | Permalink

        Well, first he should attend : Adding and Subtracting, level 1

  3. Nicholas Hallam
    Posted Nov 23, 2011 at 2:48 PM | Permalink

    Ultimately it is the obvious lack of competence, rather than the allegations of deliberate deceit, which convicts.

    • Steve Garcia
      Posted Nov 23, 2011 at 6:13 PM | Permalink

      It may be Jones’ technical lacks that take him down, since he doesn’t even seem to know that deleting an email off his PC does not remove it from the central server. Nor that his emails in this batch convict him of lying to the inquiries, even if they didn’t ask him the actual question while under oath. If the authority behind the inquiries has any gumption, it will convene long enough to ask him.

      Somehow I don’t think, “My dog ate my homework,” will suffice if he ever does get asked under oath. Like in the docket.

  4. Craig Loehle
    Posted Nov 23, 2011 at 2:59 PM | Permalink

    A common thread for Jones and Mann is that they do not like to compute (or can’t). Neither will stoop so low as to look at their data and see if there are funny things going on, missing values, outliers, six sigma, transposed numbers, etc. Jones had Hapless Harry to do all the coding for the global data. Really? This was the core of his business and he couldn’t bother to look into the code? Probably because he is not capable of doing so. Likewise, Mann is a terrible coder (even if he knows some advanced math) and does not know when he does not know what he is doing (like with his PCA and what that implies, that he is weighting certain series heavily with that method, or that picking series based on correlation results in mining for hockey sticks). Looking at the data is undignified for high priests.

    • Posted Nov 23, 2011 at 9:59 PM | Permalink

      I’m still amazed that you think he didn’t know what he was doing.

  5. Selgovae
    Posted Nov 23, 2011 at 3:01 PM | Permalink

    As someone who does regression calcs in Flash, I would probably respond in a similar way if the only tool around were Excel.

    From the same mail, I think the following line may reveal more about Jones’ mindset.

    “It isn’t an issue with climatologists. All understand.”

    Something spooky about that.

  6. observer
    Posted Nov 23, 2011 at 3:02 PM | Permalink

    I think this is proof positive of these “emails” being a forgery.

    It is inconceivable that a modern quantitative scientist would either (a) be that ignorant of Excel or (b) be incapable of quickly figuring out how to do that computation in Excel.

    Moreover, if both (a) and (b) were true, the scientist would be reluctant to admit such incompetence to others.

    I think we can safely ignore these emails.

    Observer

    • Josualdo
      Posted Nov 23, 2011 at 3:41 PM | Permalink

      Observer, I’m now retired, but I wished I could have worked in a place where my betters actually would have taught me something other than dealing with bureaucracy and institutional politics. (At which I admit I’m awful.)

    • Steve Garcia
      Posted Nov 23, 2011 at 6:16 PM | Permalink

      Observer, your confirmation bias is showing. You will give a pass, based on the flimsiest of reasons, if it exonerates one whose side you are on.

      • JonasM
        Posted Nov 23, 2011 at 11:02 PM | Permalink

        I just assumed Observer forgot to add the tag…..

        • JonasM
          Posted Nov 23, 2011 at 11:02 PM | Permalink

          That was meant to have a /sarc tag in it…..

    • Jim Good
      Posted Nov 25, 2011 at 7:55 AM | Permalink

      Hey, nice try. If this were the only evidence that Phil Jones lacks numeracy skills, then you’d have a point. As it is, this thing about Excel just sums up a whole lot of other stuff.

  7. Michael Jankowski
    Posted Nov 23, 2011 at 4:51 PM | Permalink

    Jones should’ve just emailed Mann so that he could have Rutherford do it. That guy is a step above Jones in Excel.

  8. Posted Nov 23, 2011 at 4:56 PM | Permalink

    Maybe we’re not giving Jones enough credit here: he may not be able to use Excel, but he can use Wikipedia.

    1743.txt:

    Kaufman to someone (unclear in the thread) regarding draft of what would later be published in Science as Arctic reconstruction

    To lay it bare, we’re focusing on 20-year means, and our calibration
    period is only 100 years, so n = 5. The standard error of the regression is ± 0.04°C,
    and the Durbin- Watson statistic, d = 3.9, indicating that autocorrelation is not
    significant.

    Bradley forwards to Jones. Jones to Bradley:

    Ray,
    Had a quick look and weak on statistics isn’t the half of it!
    …I’ve calculated numerous DW statistics in my time. I’ve never got one approaching
    3.9. According to wikipedia it is possible (range is 0 to 4). However, what you’re
    supposed to do with the values above 2 is to subtract them from 4. So this one should be 0.1.
    What it then means is that there is very, very, very, very high negative autocorrelation
    (because of the subtraction). It doesn’t look as though there is, so I think
    the calculation is screwed!

    The Kaufman paper went on to be published in Science after receiving 2 lightweight reviews (0900.txt) and then was shown on CA to have used a key sediment series upside down (3511.txt).

    • Posted Nov 23, 2011 at 5:23 PM | Permalink

      It looks as though they knew that the sediment series was upside down and that they misinterpreted what high X=ray density meant.

      But of course, what they did was make out that it was all McIntyre’s Fault for not realising how wonderful and naive they were.

  9. Jimmy Haigh
    Posted Nov 23, 2011 at 5:22 PM | Permalink

    Awww! Come on now. Give the guy a break. Anyone can calculate a trend but it takes a climatologist to calculate the ‘correct’ trend…

  10. Posted Nov 23, 2011 at 8:09 PM | Permalink

    jimmy

    and it takes an observer to give you a pass

  11. Scott Scarborough
    Posted Nov 24, 2011 at 12:13 AM | Permalink

    Something is wrong here. A climate scientist can’t be stumped at calculating a trend, at least in Excel, it’s too easy. I don’t calculate trends in Excel but it took me less than 5 minutes to figure out how. Excel has a help function – tells you exactly how to do it.

  12. David Smith
    Posted Nov 24, 2011 at 12:26 AM | Permalink

    Professor Jones should have asked Professor Wigley for math help (#5213)

    >>With four ways to handle non-CO2/SO2 emissions as suggested by Rob, this
    gives 105 times 4 equals 410 runs.>>

    • Tony Hansen
      Posted Nov 24, 2011 at 6:49 AM | Permalink

      Or maybe he could have just asked Eric for a spot in a matlab(?) class.
      I believe Eric was somewhat keen on offering these at one time.

  13. Martin A
    Posted Nov 24, 2011 at 4:04 AM | Permalink

    Perhaps he’s a wizz at Lotus 1-2-3 and never bothered to master Excel?

  14. Max Beran
    Posted Nov 24, 2011 at 6:26 AM | Permalink

    It has to be realised that in Britain Climatology is taught and thought of as a branch of Geography and many take up Geography precisely because maths is not their strong suite. Geography departments have often rebranded themselves as Environmental Science departments to get round the slur. (I personally am not denying that climatology is a legitimate area of non-numerate study). I am however surprised at this admission from Phil Jones as he had worked on expressing anomalies as departures from a standard Gamma distribution and recognised the difference between one and two-tail tests when evaluating significance (raising the bar on rejecting the null hypothesis). Both suggested to me a better than usual grasp (and what is rarer, internalisation) of statistics. Perhaps he was repeating what others had told him.
    A few years ago I had reason to dig into the boxed archives of Gordon Manley (the much esteemed originator of the CET dataset) and discovered there a private exchange between himself and Craddock, a Meteorologist at the British Met Office appealing for a simple explanation of “standard deviation”, an absolutely astonishing revelation given the centrality of what he was doing by stitching together short term records into a longer amalgam of data intended to be analysed as a single series. The gist of it was that he still very much at sea with any maths beyond what might be required of a club treasurer.

    • Posted Nov 24, 2011 at 6:40 AM | Permalink

      Great testimony, thanks Max. The mention of Met expertise of old – even in advanced concepts like standard deviation – brings to mind this sterling finale from Andrew Orlowski to his summary of Climategate 2 yesterday:

      So the mewling infant that we call Climate Science – a 40-year-young offshoot of meteorology – has been thrust into a political role long before it’s capable of supporting the claims made on its behalf. From the archives we can see the scientists know that too, and we can read their own reluctance to make those claims, too.

      It is possible to feel empathy for the writers of some of these emails and I think we should. Even so, Phil Jones in the top 100 may be a bridge too far until trend calculation has been mastered.

    • Steve McIntyre
      Posted Nov 24, 2011 at 9:06 AM | Permalink

      Max, you have a few cameos in the new emails: 116 1403 3585 4997

      • Max Beran
        Posted Nov 26, 2011 at 6:48 AM | Permalink

        A little more than “cameo” I hope. The one point I raise with Keith Briffa – which is the correct calibration sign for temperature on tree ring; the one signified by the recent data or the one implied by earlier data, is well ahead of the curve. The other refering to the strong negative feedback on carbon accumulation in the atmosphere if traditional dendrochronology is “true” still merits more than the zero consideration it is granted by “believers”. If summer temperatures rise through 8 standard deviations then so do tree ring widths and the associated biomass increment must take a chunk of CO2 out of the atmopshere.

        My impression is that Keith Briffa comes out of the latest release of emails relatively well as an individual much put-upon by others of the “team” with greater leverage and not least by the Sir Humphrey Appleby of the Ministry (see email #2495). My own obsequious tone of those old emails stands testimony to this pressure from on high not to rock the boat.

        • Steve McIntyre
          Posted Nov 26, 2011 at 8:22 AM | Permalink

          Touche. Your feedback point is an interesting one and does merit analysis.

          The attitudes of Briffa (and Ed Cook who also has misgivings about the MBH camp) appear to me to change when we come onto the scene. They quickly forget whatever disputes they might have within the field in order to combine against the common enemy. Briffa and Osborn abandon their nascent analysis of Mann’s “dirty laundry”. Briffa was very quick to join the pack mentality in attacks against us without realizing understanding the issues.

        • Posted Nov 26, 2011 at 9:04 AM | Permalink

          Steve I agree about the attitudes changing when you and Ross arrived on the scene but my hunch is that this is what the CRU-level scientists were told to do, very strongly, from further above, via the link men like Jones. It doesn’t excuse anything and of course I can’t prove it but it’s the way I’ve read it from the moment I first read you here.

        • Posted Nov 26, 2011 at 11:14 AM | Permalink

          Re: Richard Drake (Nov 26 09:04), More context shown by following the story of Alan Kendall of UEA from 1999 to 2009 as revealed by FOIA2011

        • CoPete
          Posted Nov 26, 2011 at 11:30 AM | Permalink

          Ummm… briffa doesn’t understand the issues? perhaps you’d like to comment on his evisceration of your submitted comment to Nature on dating issues Briffa et al (1995)? (email 0591)

        • Posted Nov 26, 2011 at 11:06 AM | Permalink

          Thanks Max for that lead. I’ve distilled essentials of EMAIL 2495 here. May 2009

          Robin Mortimer is Director of Climate Change Adaptation, Air Quality, Landscape & Rural Affairs (CALR), DEFRA. On 18 May 2009 he emails Kathryn Humphrey of DEFRA about the “Weather Generator”:

          Bob Watson, Peter Unwin and I [all DEFRA] spoke to Jackie McGlade this evening… She is very supportive of the approach being taken to downscaling to 25km grid square level…This is good news. We all agreed … that there was of course a cascade of confidence as we move down from continental to 25km scale, but, so long as the uncertainties were clearly presented, this was robust.

          The remainder of the conversation focussed on the Weather Generator. JM raised two issues on this (a) that the level of “statistical noise” involved in moving down from 25km to 5km scale meant that the results became so uncertain [as to be] virtually worthless and (b) the very fact that so many sensible scientific experts shared this opinion risked discrediting the rest of the UKCP09 package – essentially everything was getting `tainted’ by the WG. She argued we should drop it…

          Next day, Kathryn Humphrey forwards this email to others including Phil Jones:

          Could you please send me some comments urgently on the issues [above]. I have a meeting with Hilary Benn this afternoon and this is likely to come up (at 4.30). As you know we have already given him some advice on the weather generator but perhaps I need to give them more… I think the problem here is that the people in these meetings don’t have the technical background to be able to head off these criticisms at the time…

          Phil Jones replies within the hour:

          What would be a good start to the first question would be to switch it around to who doesn’t need the WG! As I’ve said on numerous occasions, if the WG isn’t there, all the people that need it will go off and do their own thing…

          Just seen Geoff’s reply – totally agree, the scale issue is a red herring. What matters is daily output! …Users will need to know what they are doing when they get their output. Those working for the EA, UKWIR and within the above projects will. It is likely that Joe Public and media types won’t. Have any of these ‘sensible scientific experts‘ read the WG Report or read the Briefing Document? …Why don’t we (Chris, Geoff, me) get a chance to talk to these ‘sensible scientific experts’! Are we not deemed ‘sensible’, ‘scientific’, or ‘expert’ enough?…

          Half an hour later, Kathryn Humphrey replies:

          Thanks Phil, Chris and Geoff- I am whacking all of this into a document to take through with Robin later. I know this is extremely frustrating for you and completely understand where you are coming from.

          This is a political reaction, not one based on any scientific analysis of the weather generator… I can’t overstate the HUGE amount of political interest in the project as a message that the Government can give on climate change to help them tell their story. They want the story to be a very strong one and don’t want to be made to look foolish. Therefore, every time they hear about any criticisms from anyone, they jump

        • Max Beran
          Posted Nov 27, 2011 at 11:39 AM | Permalink

          Thanks Lucy for your close and necessary probing of the background to 2495. It reinforces the need to understand context (just like what the apologists say) and not do, as I had done, jump to the conclusion that the statement referred to climate change/ global warming in general rather than the narrow issue of how much spatial and temporal detail to offer to the impacts community (customers of the weather generator). I speculate nevertheless that those worried about Jackie McGlade’s constituency of sensible scientific experts were not making the distinction but were worried about any contrary voices on any matter no matter how remote, hence the so-called “political” nature of the intervention. My bottom line was also certainly the case – not to rock the boat. I was a global change research manager during the 1990s and felt the pressure first hand. It’s probably a lot worse 10 years on.

          Doubly disappointing was to find that the “Humphrey” of the email was not the nearly eponymous Mandarin out of “Yes Minister” (nor the Downing Street cat before Cherie Blair took against it).

          I was in occasional touch with Jackie McGlade in her immediate post-Julich days at Warwick University and she certainly cut a powerful figure. Her research background had been in ecodynamics and had worked on self-organizing systems especially in the marine environment so well familiar with the accumulation of chaos in natural systems and, from her vantage point in EEA, possibly in touch with a few scientists who were tolerbaly well grounded. Having said that, given the artificial nature of downscaling output of 2080 climate maps, I can’t see that the passage from 25 km to 5 km spatial scale presents any major leap in incredibility that is not already present at the original continental scale. Those engaged in climate impacts activity (I can’t bring myself to call it research) would just accept the gridded data at face value and not consider the product was any better or worse that any other rules they were told to apply to provide the input they wanted to drive their guesstimations.

  15. kim2ooo
    Posted Nov 24, 2011 at 1:24 PM | Permalink

    Geeessssseeeee…everyone is mentioned in the emails – but me.

    I feel so……..

    …………………………………..

    Happy Thanksgiving :)

  16. John Norris
    Posted Nov 24, 2011 at 2:52 PM | Permalink

    There are more Phil Jones excel confessions in #3738. He is helping Michael Mann rebut MM results in a USA TODAY opinion piece with a draft reply. Immediately before he restates his excel weaknesses, he spills this telling statement:

    3. Related to the above there is the fact that their results just don’t look right. I
    always say that data analysts need to have a feel for the data. Here, the result just looks plain wrong. I try to drum this into my students and post-docs – saying go back and find the mistake, the results aren’t right !

  17. Niklas
    Posted Nov 24, 2011 at 3:53 PM | Permalink

    2893.txt. Ian Harris says:

    “Skills – in general CRU staff have wideranging computing and
    statistics skills that give confidence in writing proposals that we
    can do the work. Also – we share skills well.”

  18. Bob B
    Posted Nov 25, 2011 at 4:47 PM | Permalink

    More help for Phil

    3279.txt-
    3279.txt- Dear Phil,
    3279.txt: I looked around, and find little help about how to calculate the 95% uncertainty
    3279.txt- range of trend of the climate series. Dave’s suggestion is asking for your help.
    3279.txt- Would you give some instructions?

  19. Jon
    Posted Nov 25, 2011 at 5:06 PM | Permalink

    Maybe Jones could have used his flying time to climate junkets by practicing with Excel. See Help-Sparklines: Sparklines: Use tiny charts to show data trends.

  20. AntonyIndia
    Posted Nov 26, 2011 at 12:21 AM | Permalink

    Through Bishop Hill, #4091:

    >>> Phil Jones 10/29/09 8:49 PM >>>

    Don,
    Here’s a couple of more recent papers…..

    A couple of other thoughts. Circulation does influence temperature – phenomenon like ENSO, NAO/NAM and the SAM, but precipitation influences will not be that strong. It’s more likely to be temperature influencing precipitation.

    You can add back the absolute – if you do the way I said. Calculate your 86 regions as time series in anomalies then add back the 12 monthly means from our absolute file (these monthly means are the averages of the boxes within your regions). The ONLY difference between the anomaly and the absolute numbers is a CONSTANT. They will have absolutely no differences in variance. I know this is always a problem for statisticians, but ones like Peter Bloomfield, Richard Smith and Rick Katz (who have all worked extensively with climate data) understand why.

    If you’re anomaly files differ by more than a CONSTANT from the absolute ones you’ve made a mistake. Sorry to go on about this. I’m just trying to make sure you’ve got the right series.

    date: Fri, 30 Oct 2009 05:26:48 +1100
    from: “Don McNeil”
    subject: Re: statistical methods
    to:

    Dear Phil:

    The treatment of anomalies versus absolute temperatures is not quite as simple as you say. It might be if there were only 12 monthly constants in all, but in fact there are 12 for each region, and there are a lot of regions, and as we know the regions have different patterns of temperature increase. So creating anomalies is essentially the same as fitting a model with a large number of parameters (12 times the number of regions) to the data for the 30-year period 1961-1990 and then using this model to adjust ALL the data. As a result, the variability of the data outside the period used to fit the model is generally not the same as the variability within this period. The extent of this difference will depend on the distribution of the data, but for the hadCRUT3 data the variability increases with year after 1961. The reason for this is that the temperatures are generally increasing with different rates for different regions, so they are increasingly “getting away” from the model based on the 1961-1990 data.

    I’m puzzled by your comment that precipation is more likely an effect than a cause of temperature increase. Isn’t precipation due to atmospheric water vapour, one of the greenhouse gases that cause global warming?

  21. michael hart
    Posted Nov 26, 2011 at 1:35 AM | Permalink

    Phil Jones does at least appear to be consistent in this matter. You may recall that about two years ago Willis Eschenbach quoted him as quipping [like Fermat] that “the margin is too small to contain it” [the page margin that is, not error margins for data he was refusing to share with those who questioned his postulates].

    If he reads email #5324 [Tom Crowley] he will see an example of an email that is unencumbered by the data it supplies . I conjecture that perhaps somebody might be willing to help him get started.

2 Trackbacks

  1. […] Over at Climate Audit, Steve McIntyre writes: […]

  2. […] Over at Climate Audit, Steve McIntyre writes: […]

Follow

Get every new post delivered to your Inbox.

Join 3,379 other followers

%d bloggers like this: