A 2002 Request to CRU

In May 2008, I collated correspondence requesting CRU station from various parties commencing with Warwick Hughes’ correspondence in July 2004. See here. On a couple of occasions, I’d referred to some correspondence with Phil Jones in pre-Hockey Stick days (fall 2002). At that time, I was surprised by the promptness of the response and the extra effort that Jones had put into the response. (I think that I noted Jones’ courtesy as a correspondent from time to time in the first years of the blog.)

The correspondence is interesting to re-read in light of subsequent developments and subsequent positions. (It also shows some development in my own technical skills. In fall 2002, I hadn’t started using R and, like many of you, looked at things in Excel.)

On 8 Sep 2002, I sent the following to CRU Information; I think that this letter to the CRU contact marks my very first climate inquiry:

In Journal of Climate 7 (1994), Prof. Jones references 1088 new stations added to the 1873 stations referred to in Jones 1986. Can you refer me to a listing of these stations and an FTP reference to the underlying data? Thanks, Steve McIntyre

This was referred to Jones, who provided the following helpful reply on Sep. 12, 2002. Note that Jones said that he would be “putting the station temperature and all the gridded databases onto our web site” once the paper is published (which occurred in early 2003.) In fact this seems to have happened, as station data for 5070 stations (presumably some duplicates were eliminated between Sept 2002 and Feb 2003) was placed on their website in Feb 2003, where it remained until July 31, 2009. (However, no link was placed to the station data, nor was the name of the file ever published. To find the file without email advice, you’d have to look behind their subsequent data refusals and parse candidate files one by one, something that I did recently after taking a renewed interest in this file in June 2009.)

Sent: Thursday, September 12, 2002 12:36 PM
Subject: Fwd: Jones 1994 Data Set

Dear Steve,
You are looking into station lists from papers in the early 1990s and 1980s. These are now out of date. There will be a new paper coming out in J. Climate (probably early next year). I’m attaching the station list (5159 stations) from that paper. In this file the first number is the WMO number ( or an approximation to it or just a number – large number of US stations at the end). Official WMO numbers are those here divided by 10. The first station (Jan Mayen has a WMO number of 01001, but in our list it is 10010).
then

Latitude (degrees*10 so 589 is 58.9 N, -ve will be S)
Longutude ( similar to latitude with E -ve)
Height (m , with missing of -999)
Name
Country (this field isn’t always there and doesn’t always take into account changes of the last 20 years. We don’t use this field, so don’t bother keeping up to date with it.

Also names are common English names for countries not their official ones that the UN uses).
First year of data
Last year of data (Most of the 2001s also include 2002 but this file hasn’t been altered)
Then some other numbers.
The first file (above description) is what we call station headers. They mean we have temperature data for the years between the first and last year for each station. However there may be lots of missing data or the data may be deemed inhomogeneous (see the papers you have), so a station may not be used in the our analysis for a whole raft of reasons. As we work with station anomalies we also have a file (also 5159 lines) of stations normals (average temps in deg C*10 for 1961-90). If this second file contains -999 (missing values) then the station temperatures will not get used so the station isn’t used.
Once the paper comes out in the Journal of Climate, I will be putting the station temperature and all the gridded databases onto our web site. The gridded files on our web site at the moment are from our current analysis. The new analysis doesn’t change the overall character of the gridded fields, it is just easier for me to send the new lists of stations used from the new analysis.
I hope this helps.
Phil Jones

I have a file entitled allnorms6190.dat of length 5159 dated Sept 13, 2002, which appears to be the file enclosed in this email though it doesn’t precisely match the description. It is a list of length 5159 (as described in the Jones letter), but it does not have station names, lats and longs; instead it has the station normals – useful information not presently available anywhere that I’m aware of. The first few lines of allnorms6190.dat are shown below.

10010 1921 2001 1961 1990 -57 -61 -61 -39 -7 20 42 50 28 1 -33 -52
10050 1912 1979 951 970 -117 -123 -122 -94 -33 17 48 43 8 -38 -73 -101
10080 1911 1999 1961 1990 -153 -163 -158 -124 -44 18 58 48 4 -55 -103 -133 ]

In 2007, when Willis Eschenbach sought station data, the current version of the information was refused under various pretexts, requiring him to make repeated requests, ultimately resulting in a list of 4138 stations being placed on the CRU website after 3 or 4 FOI requests.

In response to Jones’ helpful email, on Sept 17, 2002, I sent a short note to Jones thanking him for the list and inquired about a concordance of his numbers to GHCN numbers – something that I started doing recently based on more recent versions. I referred to the availability of the 1991 version of station data (then available at CDIAC) and inquired about the 1994 version. Subsequently, it turned out that a variation of the 1994 version (cruwlda2.zip) had been online at CRU since 1996. Home computers at the time were not nearly as handy as they are now. And I hadn’t discovered the magic of R. As a result files like the one at CDIAC were then awkward for me to handle. (My skills have definitely changed on this front over the past years.)

Thanks for this. It seems awkward not to use exact WMO station numbers – do you by any chance have a concordance of your numbers to WMO numbers where they do not correspond? I (think I) noticed that sometimes your numbers are also in use for a nearby but different GHCN station, which seems a bit awkward. I also noticed a few stations in which the lat-long’s do not seem to tie into GHCN data and can forward these possible errata if you like. Wouldn’t it make more sense to convert over to WMO station numbers carrying a concordance to your past numbers?

I’m still interested in the 1994 data set as it has become so standard. Is there a FTP from which the underlying station data and mean temperatures (either as anomalies or absolutes) can be downloaded? I’ve located an FTP for your 1991 version, but have had little success in locating the 1994 version.

When you do publish the 2002 version, I would urge you to make FTP available annualized data for individual station boxes as well as for grid-boxes, so that readers interested in regional studies can carry out verifications. (If this is not currently available for the older data, it would also be nice for it as well.)

It would also be nice if annualized data were also available as I am sure that many of your users are mostly interested in this. The 12-fold reduction in dataset size is fairly important for fitting into Excel spreadsheets, which work nicely on annual data.

Regards, Stephen McIntyre

On Sept 18, 2002, Jones sent me two files: cruwld.zip, the then published version of the station data (from Jones 1994) and a file of normals for the 1994 version normup6190.dat. In the email a few days earlier, Jones said that he would place station data online at the time of publication of the new version (Jones and Moberg 2003), but in this letter, he resiled somewhat adding the word “possibly”, alluding to a desire to avoid problems such as those supposedly experienced between some European countries and GHCN, a point that recurs in later correspondence.

Dear Steve,
Attached are the two similar files [normup6190, cruwld.dat] to those I sent before which should be for the 1994 version. This is still the current version until the paper appears for the new one. As before the stations with normal values do not get used.

I’ll bear your comments in mind when possibly releasing the station data for the new version (comments wrt annual temperatures as well as the monthly). One problem with this is then deciding how many months are needed to constitute an annual average. With monthly data I can use even one value for a station in a year (for the month concerned), but for annual data I would have to decide on something like 8-11 months being needed for an annual average. With fewer than 12 I then have to decide what to insert for missing data. Problem also applies to the grid box dataset but is slightly less of an issue.

I say possibly releasing above, as I don’t want to run into the issues that GHCN have come across with some European countries objecting to data being freely available. I would like to see more countries make their data freely available (and although these monthly averages should be according to GCOS rules for GAA-operational Met. Service.
Cheers
Phil Jones

At the time, I was less attuned to some climate science practices, but there are some interesting points here. Jones sent me a file of exactly the same type as the one now requested. What changes took place between 2002 and 2009 that are relevant to a refusal decision? My qualifications are much greater now than they were in 2002. Warwick Hughes, at the time of the 2004 refusal, had published five articles in peer-reviewed literature? What relevant change had taken place between 2002 and 2004?

There’s nothing in here about confidentiality agreements and if there were relevant agreements governing cruwld.zip, then they obviously didn’t prevent Jones from sending me the data or posting the data on the CRU website. (In this light, what is the justification for the deletion of cruwlda2.zip from the CRU website on July 31, 2009?)

Jones alludes to problems between Europe and the GHCN and his desire to have more countries make their data freely available. Surely the best way of accomplishing this is to place some sunshine on the matter. Let’s find out who the problem countries are, if any, and publicize the problems. It’s hard for me to imagine any European country that could sustain such obstruction in the face of international publicity. Anyway, it’s well worth finding out. If Jones really “would like to see more countries make their data freely available” as he says here, then surely we’re on the same side.

38 Comments

  1. Matthew
    Posted Aug 6, 2009 at 1:34 PM | Permalink

    I think we need a “beer summit” to resolve this.

    • Scott Brim
      Posted Aug 6, 2009 at 2:02 PM | Permalink

      Re: Matthew (#1),

      I think we need a “beer summit” to resolve this.

      Oktoberfest will be arriving in a few months. If the beer molecules line up — and perhaps the dissolved CO2 molecules too — maybe the FOI requests will be resolved by then and all these questions will be moot.

  2. Calvin Ball
    Posted Aug 6, 2009 at 1:45 PM | Permalink

    First, they need to negotiate whether the beer will be warm or cold.

  3. Posted Aug 6, 2009 at 1:46 PM | Permalink

    My qualifications are much greater now than they were in 2002.

    You guessed it.

    • Keith
      Posted Aug 6, 2009 at 2:46 PM | Permalink

      Re: Juraj V. (#3), Re: Ryan O (#6),

      Maybe that’s it. The Team is happy to help out a duffer who is unknown, possibly thinking they will make a convert. Once someone questions the data and hypothesis, cordiality goes out the window. Didn’t Anthony Watts say on one of the other threads that when he used his computer and email address to write something to Gavin that he got no response, but when he used his wife’s, Gavin replied promptly? So, what is needed is someone that the Team cannot identify as knowing anything about ACC to ask for assistance. Of course, that person would have to have no traceable connection to any skeptical blog.

  4. James Allison
    Posted Aug 6, 2009 at 2:06 PM | Permalink

    Jones’s credibility hadn’t been seriously questioned back in 2002.

    • Posted Aug 6, 2009 at 5:54 PM | Permalink

      Re: James Allison (#5),

      I don’t think that that was the reason. Whatever happened in Phil Jones’ attitude happened suddenly when he fired back at Warwick Hughes:

      We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.

      Up until that point (and there was nothing in the preceding correspondence with Warwick that even hinted at a problem), Jones had been very cordial.

      After that email, getting anything out of Phil Jones has become increasingly difficult to the point of absurdity, as this ongoing saga with the FOIA requests has demonstrated.

      Also I don’t recall Jones being criticized for his credibility – merely his attitude to perfectly normal and reasonable academic requests for data and methodology.

      • KevinUK
        Posted Aug 7, 2009 at 6:26 AM | Permalink

        Re: John A (#12),

        “Whatever happened in Phil Jones’ attitude happened suddenly when he fired back at Warwick Hughes:”

        That’ll be about the time that Dorothy (aka Lucia) and co. (the Lion aka Willis E, the Tin Man aka Bender, and the Scarecrow aka Anthony Watts). arrived at The Emerald City (Team HQ) after following the yellow brick road (Mann’s obfuscated data and methodology) and our curious resident Toto decided to find out what was behind the curtain. The Wizard of EAU didn’t take too kindly to having his real identity revealed by a scruffy little mut.

        Our resident Toto

        KevinUK

        • KevinUK
          Posted Aug 7, 2009 at 6:33 AM | Permalink

          Re: KevinUK (#27),

          Also Steve by any chance do you have a middle name and if so is it ‘Terry’ (short for terrier)?

          See here

          KevinUK

        • KevinUK
          Posted Aug 7, 2009 at 6:53 AM | Permalink

          Re: KevinUK (#27),

          And of course this thread would not be complete without visiting the following link

          Pay No Attention…

          You see Dorothy was right, Dr Phil isn’t a bad man he’s just a ‘very bad Wizard’.

          KevinUK

  5. Ryan O
    Posted Aug 6, 2009 at 2:14 PM | Permalink

    This reminds me of my own correspondence with Team members started . . . which also appears to be heading in the same direction.

  6. PaulSh
    Posted Aug 6, 2009 at 2:43 PM | Permalink

    A simple equation: MO=MOD, therefore OSA>FOI. QED (questions effortlessly denied).

    But yes, Oktoberfestbier could be implicated in CO2 emissions.

  7. Calvin Ball
    Posted Aug 6, 2009 at 3:04 PM | Permalink

    8, I don’t know the details, but there are ways of communicating anonymously through proxies. For what you describe, getting a disposable yahoo account would probably be sufficient.

  8. Posted Aug 6, 2009 at 3:12 PM | Permalink

    There aren’t dates on everything but the way it reads to me there may actually be some perceived restrictions from Euro-[smi[] on the data. Before I thought it was a complete fabrication. It still wasn’t enough to prevent sharing the data as recently as 03 apparently.

    If there are real restrictions then surely he could release the rest of the data noting the problem countries. If we start WWIII over getting them to release their top secret thermometer records we would at least know who to bomb!

  9. Joachim
    Posted Aug 6, 2009 at 4:41 PM | Permalink

    Reading about Dr. Jones I get that stupid pop song in my head:
    Dr. Jones, dr. Jones, calling dr. Jones …. wake up now

    The song is so annoying I could almost hurt my self.

    Steve: Yup. The Team playlist is adding up.

    • Larry Hulden
      Posted Aug 6, 2009 at 10:51 PM | Permalink

      Re: Joachim (#11),

      Clicking on the YouTube arrow gives this information:
      “This video is not available in your country due to copyright restrictions”.

      Could it be P… Jones ?

  10. MikeU
    Posted Aug 6, 2009 at 6:08 PM | Permalink

    “What changes took place between 2002 and 2009 that are relevant to a refusal decision?”

    Nothing, to an objective outside observer. However, scientists are human beings like everyone else. Back in 2002, you weren’t busy giving climate scientists a collective black eye, exposing their mistakes, their biases, and their sometimes puerile behavior. I’d guess his friend-or-foe detector identified you quite clearly as “foe” in 2003, so any cooperation came to a grinding halt. You’re one of “them”, and it’s clear he’s going to fight his “foes” at every turn. His career, his profession, his credibility, and his grant money are all hitched to the global warming locomotive, so when he talked about having “25 years invested”, I don’t think he was just talking about the data.

  11. James Allison
    Posted Aug 6, 2009 at 7:01 PM | Permalink

    We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.

    Do we have any psychoanalysts reading this blog? Such person(s) should have a field day analyzing this comment. Particularly as it was written by a tax paid scientist.

  12. pete m
    Posted Aug 6, 2009 at 8:12 PM | Permalink

    wow, he used to be so nice. Shame.

  13. Posted Aug 6, 2009 at 8:53 PM | Permalink

    snip – use of prohibited word

  14. Ed Rowland
    Posted Aug 6, 2009 at 10:19 PM | Permalink

    I’m working a piece of software that allows browsing and analysis (currently) of the GHCN dataset. I was curious, wanted to see for myself. If it reaches sufficient stability, I may post source and binaries on a opensource hosting site.

    I’m working with the current GHCN v2 datasets. Are these currently the best publicly available datasets?

    Could somebody give me a brief rundown on what the actual benefits of the CRU data is over GHCN data? (I do understand the importance of CRUTEM, but are there real substantive differences between CRU and GHCN land datasets?)

    Love the site, but I find it difficult to come up to speed on a lot of the issues in play, since terminal postings often assume a great of expertise from newcomers.

    • Steve McIntyre
      Posted Aug 6, 2009 at 10:32 PM | Permalink

      Re: Ed Rowland (#17),

      Could somebody give me a brief rundown on what the actual benefits of the CRU data is over GHCN data?

      To do that, you’d have to be able to obtain CRU station data. Why don’t you request it from Phil Jones?

    • steven mosher
      Posted Aug 6, 2009 at 11:10 PM | Permalink

      Re: Ed Rowland (#17),

      Ed that is the question. CRU draws heavily from ghcn, then there are some extra bits. stevemc is trying to answer the question you asked.

      Neat project. keep us informed

  15. Ed Rowland
    Posted Aug 6, 2009 at 11:07 PM | Permalink

    fwiw, I can echo Jones’ concerns about working with annualized data. There seems to be a catastrophic disappearance of good stations in the 90s in critical parts of the dataset in the high arctic, in North America; and Eurasian high arctic stations have serious problems with consistent reporting. This is all critical because arctic stations account for a dramatic portion of overall warming trends. Somewhere in excess of 1/3 of total warming effect seems to be attributable to ferocious warming in northern Russia, Kazastan, Mongolai and Northern china alone. (Still trying find ways to deal with the massive loss of observation stations in the North American high-arctic circa 1990, so I can’t say much about the north american high arctic yet ).

    Mishandling data disconuities can result in dramatic swings in estimated global average temperature (easily 1/3 of a degree or more); but rejecting all stations that don’t have continous data for a year dramatically reduces available datasets. Based on what I’ve seen so far, I don’t think there’s a safe bias-free procedure for generating annualized data that doesn’t involve so much cooking of data that it becomes impossible to speculate as to whether data handling introduces more noise than the overall temperature increase.

    Take a look at some of the russian high-arctic station data to get a sense of why discontinuities pose such a signficant challenge. Most Russian station datasets miss occasional monthly observations; and many have gaps of many months or years. or consist of a few years of isolated observations. Cross-station averaging doesn’t work because stations within the same grid may have average temperatures that differ by five or ten degrees. A drop-out of a station at the northern or southern extreme of a grid cell can massively bias bias temperatures within the cell. Most Russian high-arctic grid cells have only one or two station datasets at any given point of time, as it is. The same is true for Canadian and Alaskan stations, starting in the 90s. 80% of the high-quality arctic Canadian stations seem to go offline in the 90s.

    The best procedures for calculating temperature trends seem to involve month-by-month comparisons against a month-by-month reference set from the same station. Procedures that annualize data too early seem to produce results that depend critical on when the reference period is; procedures that perform caluclations month-by-month seem to produce results that are consistent regardless of which reference period is used.

    Based on what I’ve seen so far, I don’t think annualized station data would be of any use at all. Gridded annual data — even more so. And gridded data seems to be only as good as the procedure that gridded it.

    Steve: I also agree with the point about monthly versus annual. At the time, I had no idea that so much data was missing.

    • steven mosher
      Posted Aug 6, 2009 at 11:26 PM | Permalink

      Re: Ed Rowland (#20),

      Cross-station averaging doesn’t work because stations within the same grid may have average temperatures that differ by five or ten degrees. A drop-out of a station at the northern or southern extreme of a grid cell can massively bias bias temperatures within the cell. Most Russian high-arctic grid cells have only one or two station datasets at any given point of time, as it is. The same is true for Canadian and Alaskan stations, starting in the 90s. 80% of the high-quality arctic Canadian stations seem to go offline in the 90s.

      sounds like a job for steig reconstruction inc

    • Steve McIntyre
      Posted Aug 7, 2009 at 5:25 AM | Permalink

      Re: Ed Rowland (#20),

      Still trying find ways to deal with the massive loss of observation stations in the North American high-arctic circa 1990, so I can’t say much about the north american high arctic yet )

      If you browse through the site (use the Categories), you’ll see discussion of this sort of question. IMO the “loss of stations” is a myth. GHCN has failed to collect data, much of which is readily available on the internet. Dawson, Canada, for example, has a continuous record since 1991, only GHCN has failed to get. This occurs for hundreds of stations.

      • Geoff Sherrington
        Posted Aug 7, 2009 at 8:28 AM | Permalink

        Re: Steve McIntyre (#25),

        Also in my browsings I find some authorities discontinue datasets in the middle of nowhere for reasons that are hard to guess. The recent tendency Down Under is to start the official record later than the first recordings, like starting in 1910 when records go back to 1860. The overarching rationale is that Stevenson screens were generaly in place by 1910 and errors of up to 1 deg C are avoided by dropping the early data. Trouble is, when you look hard at the early data, it’s very hard to see any steps in the figures indicating a change from other surrounds to Stevensons. Despite the BOM deciding to drop such early data, some other authorities continue to in-fill and use it.

        Any chance of a short summary from anyone on http://www.pbjmgt.co.uk/
        which has a number of pre-1992 CRU stations with monthly temp means. Where was it used? I’m getting confusion because of the close agreement of versions whose timing does not match the stated compilation dates.

        [RomanM: What’s with the link?]

        • Geoff Sherrington
          Posted Aug 9, 2009 at 1:58 AM | Permalink

          Re: Geoff Sherrington (#31),

          For Roman M, You asked about the URL I gave. Answer, I don’t know. There was a glitch, probably with the IP assignment. I have never seen
          http://www.pbjmgt.co.uk/ before.

          The primary URL is http://cdiac.ornl.gov/ftp/ndp020/

          This gives a number of stations with Phil Jones temperature data, e.g.

          jonesnh.dat 24-Jun-1993 17:04 8.2MB
          jonessh.dat 24-Jun-1993 17:04 1.2MB

          Plus about a 30 other files, some of which are code. Resembles the working sheets in preparing a paper.

          In the NH file, just a few random USA stations are (WMO number also given)
          BIG DELTA
          GULKANA
          ANCHORAGE
          MATANUSKA/EXP
          VALDEZ
          EAGLE
          NORTHWAY AIRPORT
          CORDOVA
          ST.PAUL

          These seem to be the sources of one of the Jones papers and I posted them on CA in case people wanted to capture at least some Jones data
          before it all disappeared. I was trying to find which paper used this info.

          I have plotted several long term stations in Australia from 1870s to now, annual averages from BOM early, BOM later CD, BOM current online, Giss homogenised, KNMI/GHCN, and now, for some stations where given above, the CRU version to year 1991. This is the first time I have found actual station data from CRU and it is monthly. The spaghetti graphs are all over the place, with spreads of several degrees C in a given year being common. The aim is to track down the most credible master version then place weights or pass/fail on later adjustments of the master.

          There is so much data floating around that it’s hard to keep track of prevenance and publication. Some of it it so bad it would be misleading if used by an innocent trying to explain why earth temp varies.

          My apologies. I did not try to waste your time.

  16. theduke
    Posted Aug 6, 2009 at 11:08 PM | Permalink

    Steve wrote:

    What changes took place between 2002 and 2009 that are relevant to a refusal decision?

    That’s easy: M&M, 2003. That was a seminal paper. It was the first time of which I am aware that the proponents of AGW had their methods and their objectivity challenged in a very public way. Previous to that, most people (including scientists) didn’t have the time or inclination to challenge the science that was coming from the group we now refer to as “the Team” and other fellow travelers associated with the IPCC who were patiently building a body of work that supported the hypothesis of CO2-influenced warming. Everyone assumed that this theory of global warming was legitimate because climate scientists were confirming its existence in study after study. M&M, 2003 changed all that.

    Jones recognized immediately that you were a threat to the AGW narrative and that anything he might divulge could be used against him.

    Mann’s humiliation in front of a Congressional Committee ratcheted up the fear factor in the AGW/IPCC community of scientists even more.

    Result: Steve McIntyre becomes Public Enemy #1 in the community of scientists and policy-makers who believe in the hypothesis of AGW.

  17. Posted Aug 7, 2009 at 3:55 AM | Permalink

    Ed Rowland, I think I recall that one has to be a bit careful about saying that data is missing, and that there may be some instances of station data not being in GISS or GHCN that are available elsewhere eg weather underground or the Russian weather service meteo.infospace.ru?
    Tell you what – post details of some stations and dates where data is missing, and see if any CA readers can find it.
    (PS. it’s not just newcomers who have difficulty getting up to speed – even regular visitors find it hard to keep up!)

  18. Ray Harper UK
    Posted Aug 7, 2009 at 6:05 AM | Permalink

    Re Ed Rowland (20)
    I recall reading an article on a website (I’ve forgotten which) regarding the payment of cold weather subsides to communities in arctic Russia/Siberia during the communist years.
    It would be very natural for the person responsible for sending in the temperature records to the Kremlin, to either slightly inflate the cold records or even to forget to send any when a short warming spell occurred.
    Might this be part of the answer as to why this part of the world is having higher temperature anomalies since the demise of communism,
    and from what I recall in the article the end of cold weather subsides.

  19. Don Keiller
    Posted Aug 7, 2009 at 6:34 AM | Permalink

    For info. I have just put in a FOI request to the MET Office (UK), quoting both the FOI (2000) Act and the MET Office’s stated FOI Policy;
    “The Met Office is committed to openness and transparency in the conduct of its operations and to the sharing of information.

    The Met Office fully supports the aims and objectives of Freedom of Information and is committed to making information available wherever feasible.

    The Met Office, as part of the Ministry of Defence, will:

    Proactively publish and release information: through the MoD Publication Scheme for generally available information and, wherever possible, in response to specific questions.
    Help individuals to exercise their rights of access to their personal data: as set out in the Data Protection Act 1998.
    Deal with requests for information in a timely manner: the Met Office will treat all requests fairly and without prejudice taking into account the public interest in making accurate and timely information available wherever possible.
    Give clear explanations of decisions if information cannot be published or released
    Support the right to appeal: if dissatisfied with the response to a request, applicants have the right to appeal under the Met Office’s FOI Complaints Process.
    Scope This statement applies to all information held in Met Office, i.e. all the information created, received and maintained by staff of the Met Office in the course of their work.
    See (http://www.metoffice.gov.uk/corporate/legal/foi.html)”

    What I have asked for is all temperature data and related data and computer code relevant to the MET Office’s prediction of future global climate.
    I have had a response acknowledging receipt.

    I’ll keep you posted

  20. JFD
    Posted Aug 7, 2009 at 8:48 AM | Permalink

    Dr. Phil Jones may or may not have errors or omissions in his data handling or analytical methodologies, but he is concerned if he does you and/or others on Climate Audit will find them. Given your well noted success at auditing climate science or lack thereof in recent years, I can well understand Jones and others concerns about you and your band of intellectual bulldogs biting an appendage and not turning loose.

    Jones would be better off if he would ask Climate Audit to take a look at controversial sections of his draft technical papers before he sends them for publication, but his fear factor is too great to let he understand that.

    If Jones had ever worked in industry, he would take a different approach to seeking the truth by using the best talents available to him. Definitely, Jones is a High S personality. As such he cannot help being a bit concerned about fair treatment from a High D such as you. I wouldn’t worry about the Jones of the world if I were you. It is time to mush forward before the snows get any deeper or the ice gets any thinner.

  21. theduke
    Posted Aug 7, 2009 at 10:25 AM | Permalink

    In fact this seems to have happened, as station data for 5070 stations (presumably some duplicates were eliminated between Sept 2002 and Feb 2003) was placed on their website in Feb 2003, where it remained until July 31, 2009. (However, no link was placed to the station data, nor was the name of the file ever published. To find the file without email advice, you’d have to look behind their subsequent data refusals and parse candidate files one by one, something that I did recently after taking a renewed interest in this file in June 2009.)

    Iow, the mole burrowed in from the other side of the fence. This is detective work worthy of Sherlock Holmes.

    Also, you ask:

    What relevant change had taken place between 2002 and 2004?

    I think the answer lies in Jones’ reply to Warwick Hughes in July of 04:

    However, it was hinted at to me a year or two ago that I should also not make the station data available.

    I wonder who the mystery hinter was.

  22. Ed Rowland
    Posted Aug 7, 2009 at 12:50 PM | Permalink

    Well. A funny thing happened.

    I went looking for the missing Canadian data. I didn’t find the missing data; but guess what I did find: a license agreement on Canadian temperature data that would prevent it from being released as part of either the CRU or GHCN dataset!

    Here’s the dataset: http://www.cccma.ec.gc.ca/hccd/

    Here’s the license: http://www.cccma.ec.gc.ca/hccd/license.shtml

    The license specificly prohibits distribution of the data except for “educational or research purposes for which you have received explicit written permission from Environment Canada”.

    Ironically, GHCN seems to contain this data already, and seems to publish it without restrictions in a manner that clearly violates the terms of the Environment Canada license. There are about 200 Canadian series that continue past 1990 in GHCN (the same as the dataset on the Environment Canada website. I’ve applied for a license to download the data, and will let you know whether the stations are contained in this product are those that are contained in GHCN V2. I strongly suspect that they are. (Letting you know whether the data is the same does not appear to be prohibited by the terms of the license).

    • Keith
      Posted Aug 7, 2009 at 1:31 PM | Permalink

      Re: Ed Rowland (#34),

      Ed, from examining the site, the data is for the “historically adjusted temperature data”, so it is not the raw feed (http://www.cccma.bc.ec.gc.ca/hccd/data/temperature/tempdata.shtml). But based upon their copyright page, the use to which you wish to use the data would be allowed as long as you follow their guidelines.

      Non-commercial Reproduction

      Information on this site has been posted with the intent that it be readily available for non-commercial use. Unless otherwise noted by the application of the copyright symbol (Environment Canada, Date), materials on this site may be reproduced, in part or in whole and by any means, without charge or further permission from Environment Canada. We ask only that:

      * Users exercise due diligence in ensuring the accuracy of the materials reproduced;
      * Environment Canada be identified as the source department; and,
      * The reproduction is not represented as an official version of the materials reproduced, nor as having been made in affiliation with or with the endorsement of Environment Canada.

      Based upon their stations list (http://www.cccma.bc.ec.gc.ca/hccd/data/temperature/temperature_stations.txt), most of the data runs through the early 2000’s. Only problem I see is that you only get to access the data one station at a time.

    • Earle Williams
      Posted Aug 7, 2009 at 1:44 PM | Permalink

      Re: Ed Rowland (#34),

      GHCN distributing what appears to be the same data may not be in violation of their license agreement with Environment Canada. It certainly is odd that one can acquire the data without fee or restriction from GHCN, but if you want it from EC you are subject to the license agreement. One assumes of course that the data that EC is providing is in fact subject to Canadian copyright law. In the US data is not subject to copyright as it is not a creative expression. Anyhoo, I don’t mean to take this thread off topic, as Steve has made it clear that he doesn’t want his blog littered with those arguments. I just find it curious. Back to my original point, one would presume that GHCN has a seperate agreement with EC such that the data can be redistributed without violation.

  23. a reader
    Posted Aug 7, 2009 at 2:03 PM | Permalink

    For Jan Mayen in my “World Weather Records” book, the record starts in 1921 as in the notation in your post. In the station notes, it shows the station moving 7 times between 1921-1950 and gives the dates, lat. long. coordinates, and general descriptions of the different sites and why it was moved. The means were computed using the formula m=n-k(n-Min). The authority is Det Norske Meteorologiske Institutt, Blindern, Norway.

    The editorial notes say that the identification numbers in the WWR book bear no relationship to the WMO international index numbers, but this must be the data that Dr. Jones used for early papers–how much different data could there have been between 1921-1950 for Jan Mayen?

    Other interesting things I’ve noticed in the book; there have already been numerous adjustments done to the data, several different formulas have been used to arrive at the means(maybe this is common even now, I don’t know), many stations contain multiple moves where the records have been harmonically smoothed, corrections have been applied for different types of shelters(in the tropics apparently they used thatched roof shelters sometimes on verandas), and each volume seems to contain extensive errata tables where errors have been noticed for previous volumes. It would be interesting to know if all of this was taken into account when they were digitized. Anyways this may be stuff you already knew–hope I’m not wasting your bandwidth.

5 Trackbacks

  1. By A Change In Climate, Part 2 « ORBIS on Dec 18, 2009 at 1:34 PM

    […] was surprised by the promptness of the response and the extra effort… put into the response (Climate Audit, A 2002 Request to CRU). Jones became more hostile when he realized what they were up to (i.e., checking his […]

  2. […] https://climateaudit.org/2009/08/06/a-2002-request-to-cru/ […]

  3. By The Final Straw « Watts Up With That? on Mar 2, 2010 at 4:17 AM

    […] McIntyre had no publications in climate science. He wrote to Jones requesting temperature data. The history of their exchange in detailed in this Climate Audit Post. Jones sent data to McIntyre along with the following mail: Dear Steve, Attached are the two […]

  4. […] advocates, such as Jones have been circling the wagons against further public scrutiny.McIntyre recounts the former openness of Jones that disproves his testimony from yesterday:“At that time, I was […]

  5. […] been reading over at CA, searching for info on what Steve McIntyre had and when he had it, and came across this post with the text of his email to Jones dated September 8, 2002: In Journal of Climate 7 (1994), Prof. Jones references 1088 new stations added to the 1873 stations […]