CRU Responds

CRU has posted up an undated webpage on data availability here , responding to the various recent FOI requests for station data and confidentiality agreements. Here they “list the agreements that we still hold”.

I’m preparing a post on this extraordinary document and am posting this thread as a placeholder for now.

Instalment 1 (Aug 11 8 pm) :
Obviously this is a pretty pathetic combination of excuses and whining. Both CRU and the Met Office should be cringing with embarrassment. Obviously there will be more shoes to drop. But let me reiterate one of my own baseline positions (and one which I do not wish to argue about with readers.) Regardless of how pitiful CRU’s management of data and contracts turns out to be, it is not my position that this is an excuse for delaying climate policy until the original data is found and documented. Neither do I think that any exigencies of the big picture excuse negligence in the small picture.

Lost Data
Surely the most surprising revelation is their confession that they’ve lost all their original data – all they have is their “value added version”. They say:

Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

I was around in the 1980s and the 1970s. People used filing cabinets back then. You’d have alphabetically arranged files by customer. If you got paper from Andorra or Zambia, you’d put the data in the Andorra or Zambia file. If a decision were made on the handling of an account, you’d put a memo in the file. How is it possible that they don’t have ANY documentation on the construction of their data? This is getting worse and worse.

CRU Excuses
Imagine [you fill in the name] saying something like this:

Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s.

Nobody would take it seriously. Nobody would believe that they were that incompetent. I wonder what would happen if Lonnie Thompson moved offices. Would he lose all his unarchived ice core data?

Or this excuse as to why they can’t get data that any one of us can locate on the internet:

Much climate data are now additionally available through the internet from NMSs, but these are often difficult to use as data series often refer to national numbering systems, which must be related back to WMO Station Identifiers.

Poor babies. Imagine having to do a concordance of CRU numbers to national numbers to enable downloading from NMSs. That would be so boring. After all, they’re climate scientists. Things to do, people to see. When’s the next IPCC authors’ workshop?

But their trials and tribulations get even worse. They report that:

a number of NMSs make homogenized data … available in delayed mode over the internet. Some that provide both raw and homogenized versions, generally do not link the two sets of data together.

Y’mean, that someone somewhere would actually have to inquire as to how to do the links. Hey, Phil, I’ve got an idea. If they won’t tell you, send them an FOI. Or better yet, we’ll save you some trouble. Make a list of all the NMSs that are troubling you and we’ll send FOIs for you. Just have your people contact our people.

And as to why they haven’t documented the source of their data. It’s not their fault – that’s impossible. The reason why they didn’t document anything is that they “never had sufficient resources”.

We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value.

160 Comments

  1. Posted Aug 11, 2009 at 1:48 AM | Permalink | Reply

    I have had a reply to my FoI request to the Met Office for a copy of the agreement with Phil Jones which prevents the Met Office disclosing the temperature data that Jones has passed on to them.

    You requested: A copy of the agreement/letter with Dr Jones in which he makes these restrictions on redistribution of the data.
    In order for the Met Office to do a thorough review for you, it will be necessary to extend the time to respond to you, as records cover a 30 year period. We will respond to you by 10 September 2009 or sooner if possible.

    Two months to locate a single agreement? I’m speechless.

    • Scott Brim
      Posted Aug 11, 2009 at 8:55 AM | Permalink | Reply

      Re: Bishop Hill (#93),

      Two months to locate a single agreement? I’m speechless.

      It might be in a huge government warehouse somewhere, inside of a wooden box just next to the one which holds the Ark of the Covenant. Possibly they are afraid to open the lid early and thus risk having the demons fly out both ahead of schedule and without prior written authorization.

    • steven mosher
      Posted Aug 11, 2009 at 12:41 PM | Permalink | Reply

      Re: Bishop Hill (#93),

      Wow, even if they use offsite data storage facilties this is long time.

    • Steve McIntyre
      Posted Aug 11, 2009 at 2:24 PM | Permalink | Reply

      Re: Bishop Hill (#1),

      Bishop, based on their request for another 30 days to answer your request, they had no direct knowledge of whether there was an actual confidentiality agreement or not, when they used this as an excuse.

      We’ve run into problems with the Met Office just making thing up before – John Mitchell’s supposed destruction of his emails; John Mitchell supposedly acting in a “personal capacity” as IPCC Review Editor.

      There doesn’t seem to be any downside for them if they mis-speak.

      • Ryan O
        Posted Aug 11, 2009 at 2:29 PM | Permalink | Reply

        Re: Steve McIntyre (#24), I bet David Palmer is cursing MET right now. He can’t be having fun with this.

        • Steve McIntyre
          Posted Aug 11, 2009 at 2:40 PM | Permalink

          Re: Ryan O (#25),

          I doubt it. Let’s suppose that they can’t find a confidentiality agreement – probably because there never was one. They will blame us for asking – rather than looking into the mirror and challenging whoever said that there was a confidentiality agreement.

        • Ryan O
          Posted Aug 11, 2009 at 2:43 PM | Permalink

          Re: Steve McIntyre (#28), Ah . . . probably true. Perhaps I should have said if I were David Palmer, I’d be cursing MET.

        • Steve McIntyre
          Posted Aug 11, 2009 at 2:58 PM | Permalink

          Re: Ryan O (#29),

          It looks like Palmer held CRU’s feet to the flame on the “confidentiality agreements” and wouldn’t protect them any further.

  2. Mark Smith
    Posted Aug 11, 2009 at 10:11 AM | Permalink | Reply

    Here is the FOI request I sent, and the reply:

    The request I sent:

    Dear Mr Palmer,

    I hereby make a EIR/FOI request in respect to any confidentiality agreements)restricting transmission of CRUTEM data to non-academics involing the following countries:
    FRENCH SOUTHERN AND ANTARCTIC LANDS
    GABON
    GAMBIA
    GHANA
    GUADELOUPE

    1. the date of any applicable confidentiality agreements;
    2. the parties to such confidentiality agreement, including the full name of any organization;
    3. a copy of the section of the confidentiality agreement that “prevents further transmission to non-academics”.
    4. a copy of the entire confidentiality agreement,
    I am requesting this information for the purposes of academic research.
    Thank you for your attention.
    Yours truly,
    Mark Smith

    The text of the reply recieved, today:

    Dear Mr. Smith
    FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST
    (Our Ref: FOI_09-111)
    Your request for information received on 27 July 2009 has now been considered. .
    Pursuant to your rights under section 1(1)(a) of Freedom of Information Act 2000 to be informed whether information is held, this letter is to formally advise you that we do not hold the requested information.
    You have the right of appeal against this decision. If you wish to appeal please set out in writing your grounds of appeal and send to me at:
    University of East Anglia
    Norwich
    NR4 7TJ
    Telephone: 0160 359 3523
    E-mail: foi@uea.ac.uk
    You also have a subsequent right of appeal to the Information Commissioner at:
    Information Commissioner’s Office
    Wycliffe House
    Water Lane
    Wilmslow, Cheshire
    SK9 5AF 
    Telephone:   01625 545 700
    http://www.ico.gov.uk
    Yours sincerely

    David Palmer
    Information Policy & Compliance Manager
    University of East Anglia

  3. Charlie
    Posted Aug 11, 2009 at 10:17 AM | Permalink | Reply

    CRU has responded to one of my 3 requests.

    It now has a webpage that lists all of the confidentiality agreements they hold. Reading through that page, it appears that the only confidentiality agreement they currently hold is for BAHRAIN.

    CRU Data Availability basically says “No, you’re not going to get raw data, and in some cases we have destroyed it”.

    A few key extracts:

    Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period. In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties.

    The inability of some agencies to release climate data held is not uncommon in climate science.

    These data are not ours to provide without the full permission of the relevant NMSs, organizations and scientists. We point enquirers to the GHCN web site.

    Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

    • Armand MacMurray
      Posted Aug 11, 2009 at 11:14 AM | Permalink | Reply

      Re: Charlie (#170),
      Since the Bahrain agreement is with the airport authority/manager, my guess would be that the Bahrain data is solely from the airport there, and thus perhaps freely available from other sources.

  4. Charlie
    Posted Aug 11, 2009 at 10:22 AM | Permalink | Reply

    The text of CRU’s response to my request was similar to Comment #169, above, but had the following text at the bottom:

    All written agreements that we possess in relation to any data received from any country or geographic area are now all available via the Climate Research Unit website at: http://www.cru.uea.ac.uk/cru/data/availability/. This page also has information regarding the compilations, processing and handling of any data received.  The manner in which the station data was collected, the changes in national boundaries and entities, and the nature of the processing of the data by the CRU have all evolved & changed over the time of collection of the data.  This means that there is not always a direct correlation between any agreement and the geographic location of the data or stations that the agreement covers

  5. Bob Koss
    Posted Aug 11, 2009 at 10:59 AM | Permalink | Reply

    Hmmm.

    He starts out saying “all written agreements”. Should I infer there remain undisclosed verbal agreements they would feel compelled to honor?

    • Charlie
      Posted Aug 11, 2009 at 11:10 AM | Permalink | Reply

      Re: Bob Koss (#174),

      Yes, on the webpage referenced in the FOIA response, CRU says

      Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period.

  6. tallbloke
    Posted Aug 11, 2009 at 11:08 AM | Permalink | Reply

    I got my reply today too.

    Your request for information received on 25 July 2009 has now been considered. .
    Pursuant to your rights under section 1(1)(a) of Freedom of Information Act 2000 to be informed whether information is held, this letter is to formally advise you that we do not hold the requested information for the requested countries save that relating to Gibraltar.
    In regards Gibraltar, it is not possible to provide the information directly. In accordance with section.17 of the Freedom of Information Act 2000 this letter acts as a Refusal Notice, and I am not obliged to supply this information. The reasons for exemption are as stated below:
    Exemption
    s.21(1)(a), Information reasonably accessible to applicant by other means
    Reason
    s.21(1)(a), Information reasonably accessible to applicant by other means

    All of the requested information can be obtained from the Climate Research Unit website

    So, who holds the details of the relevant agreements with Egypt, Greece, Cuba, and Greenland.

    Concerned polar bears want to know.

  7. henry
    Posted Aug 11, 2009 at 11:31 AM | Permalink | Reply

    Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

    The way I read this is that they no longer have the raw data. It was the raw data that had the “confidentiality agreement” against it, not the “value added” data. Can this “value added data” have the same confidentiality agreement against it? Does the agreement state that the raw is not to be released, or can it include the post-processing of the data?

    If processed data can be lumped in with the agreement, how could ANY scientist release anything about this data without violating the agreement?

    Also, if they consider the value-added data they hold to be “quality controlled”, how can they check the quality against data they no longer have?

  8. Willis Eschenbach
    Posted Aug 11, 2009 at 12:07 PM | Permalink | Reply

    I got my answer today as well. This is absolutely priceless. Rather than keep the raw data, from which the “quality controlled and homogenized” data could be produced at a moment’s notice, they have thrown away the raw data.
    .
    The fact that these good folks have the balls to style themselves as “scientists” defies belief.
    .
    A few other oddities:
    .
    1) Some of the data which can’t be released is from overseas British territories such as Gibraltar, Stanley Island, and the like …
    .
    2) Other than the British agreement, only three other countries are on the list, Norway, Bahrain and Spain.
    .
    3) The Spanish agreement says nothing about passing the data on to third parties. It simply says:
    .

    El abajo firmante se compromete a utilizar la informacion requerida para los
    fines especificado en este solicitud y citar siempre el origen de la misma en
    cualquier publicacibn o trabajo en que se use.

    .
    This means “The person signing below swears to use the requested information for purposes specified in this request, and to cite the origin of the data in whatever publication or work in which it is used”. Nothing about third parties at all. The purpose specified is “Public”, as opposed to “Private”.
    .
    However, they go on to say:
    .

    We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value.

    .
    Now, I don’t think that “we might have made a confidentiality agreement sometime with somebody from somewhere, but we don’t know who or when” is adequate to shield data from an FOI request … but then, I’m not an FOI lawyer. I’d welcome comments on this. If the FOI allows this as a response, it is useless. Any lawyers want to chime in on this?
    .

    In any case, the fun goes on. I’ve just sent this response:

    Dear Mr. Palmer:

    Thank you for your reply.

    I previously formally requested you to release the CRUTEM station data, under FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04).

    You said at the time you could not do it because of confidentiality agreements. I formally requested copies of the relevant confidentiality agreements (see below).

    In response to my latest request, you point to the CRU web page which states inter alia:

    Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties. Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s. Some date back at least 20 years. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period. In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.

    We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value.

    Call me crazy, but I don’t think that “we think we might have made a confidentiality agreement sometime with somebody from somewhere, but we don’t know who or where or when” is an adequate excuse to shield data from an FOI request.

    In addition, the statement:

    In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties.

    is not true. You only show one single request for data, and that is not even for a foreign country but for British Territories. It is a long way from one request for British data, which was not made to an NMS, to “our requests for data from NMSs have always stated”. You have not given one single example of a request to an NMS upon which to base your statement.

    Science depends on replicability. You are promoting your dataset as a suitable basis for making billion-dollar decisions on what we should do on regarding the “global warming” supposedly shown by your dataset. But under your secrecy policy, your results cannot be replicated.

    As such, you have two ethical scientific options, and one unethical option:

    1. Release the data, or

    2. Retract the dataset as being unreplicable anecdotal evidence only.

    or …

    3. Keep stonewalling.

    I hereby formally appeal your decision not to supply the CRUTEM station data as requested in my FOI_07-04. “My dog ate the confidentiality agreements” doesn’t cut it in the scientific world, where billions of dollars hang on your data. If you can’t show your figures, you should be ashamed to publish them under the guise of scientific data.

    w.

    PS – The agreement with Spain does not support your argument, it says nothing about confidentiality or passing the information on to third parties. In fact, the Spanish agreement specifically says that you want it for “Public” use, as opposed to “Private” use, so you are breaking the agreement by not releasing the data.

    .
    Gotta love these guys, “science” at its finest …
    .
    w.

  9. Posted Aug 11, 2009 at 12:48 PM | Permalink | Reply

    I also had a request into CRU for the agreeements with national met offices which prevent disclosure of land station data supplied to CRU. My request preceded the group effort here by a couple of weeks. This request has now been refused as follows:

    All written agreements that we possess in relation to any data received from any country or geographic area are now all available via the Climate Research Unit website at: http://www.cru.uea.ac.uk/cru/data/availability/. This page also has information regarding the compilations, processing and handling of any data received. The manner in which the station data was collected, the changes in national boundaries and entities, and the nature of the processing of the data by the CRU have all evolved & changed over the time of collection of the data. The agreements relate to the stations, which have stayed the same over the years, despite changes to national boundaries.

    The webpage cited is new to me and I’m not sure if the information there is new or not. I’ve been offline today so I haven’t had a chance to look at anything there but there is a page called agreements which I assume is relevant.

  10. Posted Aug 11, 2009 at 1:09 PM | Permalink | Reply

    OK, I’ve taken a look at the agreements document. It looks to me as if non-disclosure terms have been put in place by

    a)Denmark [Steve - nope, this is Norway,]
    b)Bahrain

    There are also documents filed for:

    c) the UK Met Office – there is a document from Mike Hulme requesting British overseas territory data, in which he undertakes not to pass info on to third parties. There is also another document which mprobably a red herring. It looks to be a printout of a website page rather than a formal agreement. It does say that UK Met Office data is commercially valuable and should only be used under the terms agreed.
    d) one non-headed document, which appears to be Spain. My spanish is poor but I can see nothing that looks like a non-disclosure clause although there is something which seems to say data should only be used for the agreed purposes which may amount to the same thing. Strangely though I can see nothing where the purposes are defined.

  11. Posted Aug 11, 2009 at 1:18 PM | Permalink | Reply

    And now I see you’ve been discussing it on a different thread for two hours!

  12. Posted Aug 11, 2009 at 1:53 PM | Permalink | Reply

    Re: the agreement with Bahrain.

    I would think that a true confidentiality agreement would need the signature of the receiver, not the sender. In other words, the person who is to keep the confidence…

  13. Ryan O
    Posted Aug 11, 2009 at 1:56 PM | Permalink | Reply

    Based on the wording of the UKMO memo, it seems clear to me that the disclosure agreements do not prevent the individual NERC Data Centres (of which CRU is one) from releasing the information for scientific/academic purposes. Paragraph 1 states that the NERCs have the authority to release the information to specific individuals for a pre-defined purpose. It is also clear that the computer code used to convert the raw data into a temperature index is not covered by whatever agreements exist with other countries/institutions. As they provided this memo in defense of the non-release of information, I have submitted an additional FOI request for the data and, in keeping with the requirements listed in the UKMO memo, I have stated the scope of the project.
    .
    The request is for:
    .
    1. All original station temperature data used to produce the CRU gridded land temperature data set.
    .
    2. The code used to convert the original station temperature data series into the gridded land temperature set, to include all homogenization adjustments, adjustments to remove non-climatic influences, and any other adjustment to the original station time series. Text format of the code in the native language is acceptable.
    .
    The scope of the project is:
    .

    1. Investigation of the effects of changing land use, urban heat island effects, and other non-climatic influences;
    .
    2. Sensitivity analysis of parameters used to grid and adjust data;
    .
    3. Sensitivity analysis of methods to calculate monthly means (e.g., min/max, max only, min only);
    .
    4. Analysis of the source of differences between the CRU land temperature index and other indices (e.g., GISTEMP and NOAA); and,
    .
    5. Analysis of the effect of using satellite-derived covariance information to interpolate between land stations.
    .
    #5 is especially because Phil Jones issued the “incorporate satellite data” challenge, and if the data can be released for that purpose, then it should be able to be released to other parties attempting to accomplish the same thing.
    .
    My read of the information presented at the data availability link is that this particular FOI request fulfills the requirements to release the data (though I would not be permitted to distribute it further). To deny this request, they would have to find something more substantial than what they’ve provided to this point.

    • bluegrue
      Posted Aug 11, 2009 at 3:59 PM | Permalink | Reply

      Re: Ryan O (#19),
      You may still not qualify. From the UKMO memo:

      It is to be expected that ‘bona fide academic research’ using the data will eventually result in scientific publications in the open literature. ‘The Data Centres will request details of such publications in due course, and if they do not arise UKMO may seek further evidence that this was at least the intention at the outset of the research.

      I don’t think blog posts will meet their requirements, so be aware they expect you to sincerely plan to write and publish a research level article using the data in order to qualify for data access.

  14. yodajeez
    Posted Aug 11, 2009 at 2:05 PM | Permalink | Reply

    In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties.

    The irony is strong with this one.

    Steve: Yup. CRU climate scientists have never worried too much about logic.

    • Steve McIntyre
      Posted Aug 11, 2009 at 2:37 PM | Permalink | Reply

      Re: yodajeez (#21),

      In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties.

      Let’s see: they provide 4 documents from 1993-94 pertaining to CRU and a copy of the present webpage setting out the form of agreements between the Met Office and NERC.

      The statement above refers to CRU “requests for data from NMSs”. There are only two request documents, one of which is to the Met Office and not to a NMS. The only document demonstrating the form of their requests is the application to Spain, which asks only that Spain be cited.

      The other two documents are replies from NMSs of Norway and Babrain. But these do not show the language of the original request.

  15. R.Barker
    Posted Aug 11, 2009 at 2:06 PM | Permalink | Reply

    Just to keep the list up-to-date with regard to responses from David Palmer, today I recieved the following respones to my FOIA request:

    (original request):

    I hereby make a EIR/FOI request in respect to any confidentiality agreements restricting transmission of CRUTEM data to non-academics involing the following countries: Guinea; Guinea-Bissau; Guyana; Haiti; Hong Kong

    1. the date of any applicable confidentiality agreements;
    2. the parties to such confidentiality agreement, including the full name of any organization;
    3. a copy of the section of the confidentiality agreement that “prevents further transmission to non-academics”.
    4. a copy of the entire confidentiality agreement,

    I am requesting this information for the purposes of academic research.

    (Response):

    All written agreements that we possess in relation to any data received from any country or geographic area are now all available via the Climate Research Unit website at: http://www.cru.uea.ac.uk/cru/data/availability/. This page also has information regarding the compilations, processing and handling of any data received. The manner in which the station data was collected, the changes in national boundaries and entities, and the nature of the processing of the data by the CRU have all evolved & changed over the time of collection of the data. The agreements relate to the stations, which have stayed the same over the years, despite changes to national boundaries.

    As has been mentioned, the webpage referred to does not provide any confidentiality agreements for the countries I requested. To tell Steve that they can’t provide the data because of pre-existing confidentiality agreements, but then to be incapable of providing them is surely indefensible?

    I await Steves post with eager anticipation – looks like we have them by the short and curlies!

  16. R.Barker
    Posted Aug 11, 2009 at 2:17 PM | Permalink | Reply

    I have just sent the following email to David Palmer:

    Dear Mr Palmer,

    Thank you for your response to my request for any confidentiality agreements restricting transmission of CRUTEM data to non-academics involing the following countries: Guinea; Guinea-Bissau; Guyana; Haiti; Hong Kong. Given that you have stated that you do not hold any confidentiality agreements concerning these countries, I make the following request:

    I hereby make a EIR/FOI request for a copy of any digital version of the CRUTEM station data involving the following countries: Guinea; Guinea-Bissau; Guyana; Haiti; Hong Kong.

    I am requesting this information for the purposes of academic research.

    Thank you for your attention.

    Yours truly,

    R. A. Barker

  17. jae
    Posted Aug 11, 2009 at 2:40 PM | Permalink | Reply

    Wow! All these excuses! Why didn’t Dr. Phil explain these “problems” when he was originally asked for the data?

  18. mpaul
    Posted Aug 11, 2009 at 2:44 PM | Permalink | Reply

    I requested ARUBA, ANTIGUA AND BARBUDA, AFGHANISTAN, ALGERIA, ASCENSION ISLAND.

    Your request for information received on 24 July 2009 has now been considered. .

    Pursuant to your rights under section 1(1)(a) of Freedom of Information Act 2000 to be informed whether information is held, this letter is to formally advise you that we do not hold the requested information.

    Its a bit odd that they would phrase it as an affirmative fact: “we do not hold the information” as opposed to saying something like, “we’ve searched our records and can’t find documents that meet your request”. One can only concluded that they know for a fact that they do not have NDAs with these jurisdictions.

    • Terry
      Posted Aug 11, 2009 at 3:34 PM | Permalink | Reply

      Re: mpaul (#30),

      I received the same response today – the five I requested were Greenland, Iceland, Ireland, New Zealand, and Thailand.

  19. a.s.
    Posted Aug 11, 2009 at 2:57 PM | Permalink | Reply

    The letter from the Norwegian Meteorological Institute is surely out of date. The institute is in the lead in Europe when it comes to make data available for free: “From September 2007 weather data has been made available for everyone for free. See more information about our free data policy (in Norwegian only).”

    Here you can find the Klimadata Web Services and information in english.

    • Steve McIntyre
      Posted Aug 11, 2009 at 3:00 PM | Permalink | Reply

      Re: a.s. (#31),

      That’s particularly useful in that Norway is almost the only example of a “confidentiality agreement” that CRU could come up.

  20. Scott Brim
    Posted Aug 11, 2009 at 3:00 PM | Permalink | Reply

    Oh, what a web we weave, when we seek to retrieve.

  21. Steve McIntyre
    Posted Aug 11, 2009 at 3:09 PM | Permalink | Reply

    BTW I haven’t got an answer to my request yet.

  22. Dan White
    Posted Aug 11, 2009 at 3:30 PM | Permalink | Reply

    I got the usual reply for the following countries:

    ACORES, ALBANIA, ANDAMAN AND LA, ANTARCTICA, ANTILLES.

  23. David Holland
    Posted Aug 11, 2009 at 3:35 PM | Permalink | Reply

    For anyone wishing to press CRU please remember station data are obviously “environmental information” exempt from disclosure under the FOIA as is any information about administrative measures, like confidentiality agreements, in connection with it. You should immediately request an Environmental Information Regulations 2004 r.11 reconsideration if you are not satisfied with what CRU say. This has a 40 working day time limit whereas FOIA has none.

    EIR has a presumption of disclosure and no absolute exception for confidentiality agreements. Even if disclosure could be shown to be actionable, which is vanishingly unlikely, a public interest test must be made having regard to the public interest and the ICO is unlikely to maintain nondisclosure. I think a Court would give CRU short shrift if they press this issue. Unfortunately as Jones and Mann have shown it is possible to spin this process out for years.

  24. Posted Aug 11, 2009 at 3:35 PM | Permalink | Reply

    I also received a reply last night, similar to many described above, saying “…we do not hold the requested information.” For the record, my request read as follows:

    I hereby make a EIR/FOI request in respect to any confidentiality agreements restricting transmission of CRUTEM data to non-academics involving the following countries:

    DOMINICAN REPUBLIC,
    ECUADOR,
    EL SALVADOR,
    ERITREA, and
    ETHIOPIA.

    For these countries, I request all information about:

    1. the date of any applicable confidentiality agreements;
    2. the parties to all applicable confidentiality agreement, including the full names of any organizations;
    3. a copy of the sections of all applicable confidentiality agreements that prevent further transmission of the data to non-academics; and
    4. a complete copy of all applicable confidentiality agreements.

    I am requesting this information for the purposes of academic research.

    And for the record, here is the reply:

    Your request for information received on 25 July 2009 has now been considered.

    Pursuant to your rights under section 1(1)(a) of Freedom of Information Act 2000 to be informed whether information is held, this letter is to formally advise you that we do not hold the requested information.

    You have the right of appeal against this decision. If you wish to appeal please set out in writing your grounds of appeal and send to me at [etc.]

  25. rafa
    Posted Aug 11, 2009 at 3:52 PM | Permalink | Reply

    Data from Spain (Tmax, Tmin, Tmean, Precipitation, etc) for the station network is freely available for the period 1971-2000. I could not find available in the web page the 1961-1990 data requested by CRU to the spanish MET.Office. I do not think the data can be subject to any confidentiality agreement other than not to be used for commercial purposes.

    See for example data from various stations in Madrid area here (1971-2000)

    best

  26. Kenneth Fritsch
    Posted Aug 11, 2009 at 4:28 PM | Permalink | Reply

    We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value. Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

    To repeat: We, therefore, do not hold the original data but only the value-added (i.e. quality controlled and homogenized) data.

    In other words, while the dog ate some of the contractual agreements with the sources for the data, more importantly and embarrassedly the dog also devoured the original data. Given that information is it not totally futile to ever expect to see the original data at least from CRU? Is it not also futile to ever expect CRU to go back to the original data and expect them to attempt to improve on any of their adjustment techniques?

    One could laboriously attempt a CRU reconstruction of original data from GISS data or going back to the data originators as Steve M has attempted. Meanwhile, as a scientist, one could accept CRU on good faith or say, to hell with your sloppiness, and move onto to other data sources where, at least, the originals are available.

    • Ryan O
      Posted Aug 11, 2009 at 4:34 PM | Permalink | Reply

      Re: Kenneth Fritsch (#44), I don’t buy it. They have the original data. It might not all be in one, compact package like the value-added data, but they physically possess it. It will take much more than this statement to convince me otherwise.

      • Kenneth Fritsch
        Posted Aug 11, 2009 at 8:34 PM | Permalink | Reply

        Re: Ryan O (#45),

        I don’t buy it.
        They have the original data. It might not all be in one, compact package like the value-added data, but they physically possess it. It will take much more than this statement to convince me otherwise.

        Admitting that you have lost your original data for a temperature data base with the prestige of CRU has to be damn near fatal for future use by climate scientists – assumming those scientists care about the apparent sloppiness revealed and what their use of sloppy data would imply about their own work.

        My question would be why make such a damaging admission if it were not true. Maybe after years of being without the original data they finally had a Bernie Madoff moment.

    • John
      Posted Aug 12, 2009 at 2:03 AM | Permalink | Reply

      Re: Kenneth Fritsch (#44),

      I would like to know whether the “value” can be subtracted from the “value added” data. If the adjustments are systematic, they might be reversible, meaning the original RAW might to a degree be approximated. That would be if the original methods, constants and rationales were properly published or available.

      • Kenneth Fritsch
        Posted Aug 12, 2009 at 8:28 AM | Permalink | Reply

        Re: John (#96),

        I would like to know whether the “value” can be subtracted from the “value added” data. If the adjustments are systematic, they might be reversible, meaning the original RAW might to a degree be approximated. That would be if the original methods, constants and rationales were properly published or available.

        One would have to assume that the corrected/adjusted data is “correct” and would thus leave no traces of its former unique problems. Of course, if a record was kept of the exact correction made on the original data and the record of the original data were then lost than your suggestion would work – and no doubt applied by CRU and thereby avoiding their fatal admission.

  27. curious
    Posted Aug 11, 2009 at 5:28 PM | Permalink | Reply

    Not sure Ryan. Compare:

    “We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s.”

    re: confidentiality agreements and:

    “Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues.”

    re: data.

    Whilst I agree the actual no. of bytes of info. would have been relatively small; how were they receiving the data in those days? File transfer? Physical media? Written copies for data entry? Transcription by phone? What machine would they have used to perform the “value add” operations? If this were an individual’s PC it is quite possible the raw data never got onto a mainframe for a run and maybe never found its way into an archive and back up system. If they only had physical originals it would not be a big surprise if these went the same way as the confidentiality agreements? More of the dog’s missing dinner perhaps?

    • Ryan O
      Posted Aug 11, 2009 at 5:39 PM | Permalink | Reply

      Re: curious (#46), Perhaps, but the qualifier they add “not able to keep the multiple sources for some sites” would imply that for others, they did keep them. In any case, this would not prevent them from providing whatever original data they actually possess. Additionally, the advancement of the explanation that data storage availability in the 1980s is the reason means this condition can only apply to a very small subset of the data. And if they actually also lost the handwritten data sheets (along with all of the supposed confidence agreements, then they are truly more incompetent than I had imagined.

    • Steve McIntyre
      Posted Aug 11, 2009 at 6:18 PM | Permalink | Reply

      Re: curious (#46),

      Why don’t you re-read my chronology of CRU versions.

      Anything from the 1980s was delivered up to DOE and is available at CDIAC. So even if they made agreements in the 1980s that they can’t find, they broke the agreements and published the data, and can’t now selectively rely on their broken agreements.

      • tetris
        Posted Aug 11, 2009 at 6:41 PM | Permalink | Reply

        Re: Steve McIntyre (#51),
        Is this why NOAA has stated that it doesn’t consider the HadCru data to be of value/signifance? :-)

        When do you think it would be appropriate to “out” this sorry lot in a “peer reviewed” publication?

        snip – please do not over-editorialize.

  28. curious
    Posted Aug 11, 2009 at 5:59 PM | Permalink | Reply

    Yes, but if “not able to keep…” translates to “we only lost some…” (imagine 1 box of 5 1/4 in. floppies going astray in a move) then they might prefer to just dance around it rather than come clean. Obviously this would go against all previous indicators…. :)

    • Ryan O
      Posted Aug 11, 2009 at 6:22 PM | Permalink | Reply

      Re: curious (#48), My take on this is that they do not have all of the original data (but they have the vast, vast majority of it) and are using semantics to avoid handing out anything.
      .
      I find this to be the most plausible explanation, as:
      .
      1. They clearly released data to Peter Webster of Georgia Tech,
      .
      2. Jones recently published an article finding evidence of UHI at Chinese sites (which requires the original data),
      .
      3. Steve’s June FOI request was denied with no mention of missing data,
      .
      4. Jones clearly had the vast majority or all of the underlying data archived as of September, 2002, and was intending to put it on line,
      .
      5. The MET Office has clearly indicated in several past correspondences that CRU possesses the underlying data, including a written statement from John Kennedy that CRU and Phil Jones were, indeed, in possession of the underlying data,
      .
      6. The MET Office was apparently so convinced that Jones had the raw data that they even provided a FAQ describing how to request this data from Jones.
      .
      In short, the statement that CRU (not the MET Office) does not have the original data is a recent development that has appeared out of left field. Therefore, I question the veracity of their claim.

  29. Dave Dardinger
    Posted Aug 11, 2009 at 6:01 PM | Permalink | Reply

    Why doesn’t CRU simply send a notice to any questionable government saying something like:

    “Given the general interest in data related to the AGW theory, we intend to release all such data we presently hold. If you have anything in your records regarding a confidentiality agreement between us and you, please forward a copy to us so that we can determine it’s applicability under present circumstances. If we have not received such a document by 1 Oct 2009 we will release the data to public view.”

    Once that was done and results obtained, it would be clear the scope of privacy agreements, if any. An entity which didn’t wish to maintain such an agreement could simply not reply while those who came late to the party might or might not have a chance to have their data from public view.

    • Steve McIntyre
      Posted Aug 11, 2009 at 6:14 PM | Permalink | Reply

      Re: Dave Dardinger (#49),

      Doubtless, at one point, they might have said that it was “too much work” to write to the NMSs in question and seek permission. But they’ve shown that they can handle 50 or so FOI requests. So it shouldn’t be that hard for them to send 50 emails to 50 NMSs. Or 200 NMSs.

      • steven mosher
        Posted Aug 11, 2009 at 11:59 PM | Permalink | Reply

        Re: Steve McIntyre (#50),

        In fact I would suggest that are required to. In Annex G of their procedures there are guidelines for contracting with third parties and they have requirements to advise third parties of the potential conflict between confidentiality and FOIA. If they continue to receive data from a third party and have no records of the contractual basis of that transfer then it would appear they have an obligation to clarify matters.

  30. Charlie
    Posted Aug 11, 2009 at 6:31 PM | Permalink | Reply

    I find the simple direct approach works often enough that it is worth trying.

    I just left a message at the online contact form at the Bahrain CAA website.
    http://www.caa.gov.bh/contactus.php

    For A.Majeed Isa
    ASSISTANT UNDERSECRETARY FOR METEOROLOGY :

    I am trying to replicate and verify the gridded global temperature series HADCRU3 issued by the Climate Research Unit of East Anglia University and the Hadley Centre of the UK Met Office.

    The CRU declines to release the raw station data needed to replicate and verify this important and highly influential scientific dataset used in the many IPCC climate change reports and in other scientific work.

    One justification for not releasing the data is a confidentiality agreement between East Anglia University and the Bahrain CAA, dated 12th Feb 1994. See page 8 of 8 at http://www.cru.uea.ac.uk/cru/data/availability/agreements.pdf

    I am trying to contact someone that can rescind this confidentiality agreement, thereby allowing this important scientific product to have an effective peer review.

    Would it be possible for the 1994 agreement be modified such that CRU can allow others to access the raw station data as part of an exercise to replicate, audit, verify and peer review the process used to generate the HADCRU3 gridded dataset?

    I greatly appreciate it if you could arrange to remove this impediment to the open and transparent scientific validation of a very important global climate metric.

    Respectifully,

    • Willis Eschenbach
      Posted Aug 12, 2009 at 12:44 AM | Permalink | Reply

      Re: Charlie (#53), following on your excellent template, I have sent the following to the Norway DNMI:

      Dear Sirs:

      I am trying to replicate and verify the gridded global temperature series HADCRU3 issued by the Climate Research Unit of East Anglia University and the Hadley Centre of the UK Met Office.

      The CRU declines to release the raw station data needed to replicate and verify this important and highly influential scientific dataset used in the many IPCC climate change reports and in other scientific work.

      One justification for not releasing the data is a confidentiality request to East Anglia University from the DNMI, dated 10th Nov 1993. See pages 2-3 at http://www.cru.uea.ac.uk/cru/data/availability/agreements.pdf

      I am trying to contact someone that can rescind this confidentiality agreement, thereby allowing this important scientific product to have an effective peer review.

      Would it be possible for the 1993 agreement be modified such that CRU can allow others to access the raw station data as part of an exercise to replicate, audit, verify and peer review the process used to generate the HADCRU3 gridded dataset?

      I greatly appreciate it if you could arrange to remove this impediment to the open and transparent scientific validation of a very important global climate metric.

      Respectifully,

      Willis Eschenbach
      Independent Climate Researcher

      Since Bahrain and Norway were the only NMS’s with even a pseudo-agreement, that covers all their bases.
      .
      Onwards, ever onwards,
      .
      w.

      • steven mosher
        Posted Aug 12, 2009 at 1:50 AM | Permalink | Reply

        Re: Willis Eschenbach (#92),

        probably want to advise them that CRU have likely violated the agreement ( assuming one existed )by posting the data.

      • Thor
        Posted Aug 18, 2009 at 5:41 AM | Permalink | Reply

        Re: Willis Eschenbach (#92),

        I have sent the following to the Norway DNMI…

        The public journal at DNMI seems to contain no reference to your request yet.

        • Willis Eschenbach
          Posted Aug 18, 2009 at 12:25 PM | Permalink

          Re: Thor (#158), Thanks for following this. You say:

          The public journal at DNMI seems to contain no reference to your request yet.

          Dunno about that part, I dont read Norwegian. The reply I received, dated the 11th of August, was:

          Takk for henvendelsen! E-posten vil bli besvart så raskt som mulig,
          og senest innen 5 virkedager.

          Vennlig hilsen
          Meteorologisk institutt

          Thank you for your inquiry. Your e-mail will be answered as soon as
          possible, and within five work days at the latest.

          Kind regards,
          Norwegian Meteorological Institute

          Now, it’s been more than five work days, so I’m expecting a reply very soon. If not, I’ll write back to them.

          w.

  31. Charlie
    Posted Aug 11, 2009 at 6:39 PM | Permalink | Reply

    A key argument made by the CRU is that agreements not to further pass on the data to 3rd parties is part of the normal, accepted way of doing business in the climate science / national met office world, and that even though the CRU has very few copies of such agreements, that virtually all data has been acquired under such understanding, either implicit or explicit.

    This of course, ignores the long standing scientific tradition of making published work transparent and reproducible.

    I at least two ways around this problem:

    1) individually getting each (potential) source of historical station data to waive any confidentiality or restrictions on the use of the data for scientific research, such as Steve McIntyre’s proposed replication of the HADCRU3 gridded data.

    2) getting enough public and scientific community and political pressure applied to this issue that the assumptions of what is normal are changed to accomodate the use of the raw station data for scientific purposes such as SM has proposed. Seeking and acquiring such approval for scientific use by a few dozen countries would, in my opinion, go a long ways towards convincing the Met and CRU that they should produce what data they have not yet discarded.

    Thoughts? Comments?

    Steve: I haven’t proposed personally replicating HadCRU calculations. I don’t have the time or interest to carry out the examination of sites and metadata that would be required in the sort of study that I think needs to be done. Nor for that matter do I think that climate policy should be delayed while this sort of thing is done. But equally I don’t think that any exigencies of the big picture excuse negligence in the small picture.

    • Steve McIntyre
      Posted Aug 11, 2009 at 7:01 PM | Permalink | Reply

      Re: Charlie (#54),

      Absolutely. I urge people to write to NMSs directly. It’s well worth reading WMO REsolution 40 mentioned by Jones.

      • Ryan O
        Posted Aug 11, 2009 at 7:03 PM | Permalink | Reply

        Re: Steve McIntyre (#56),

        Or better yet, we’ll save you some trouble. Make a list of all the NMSs that are troubling you and we’ll send FOIs for you. Just have your people contact our people.

        .
        I read that on your update and seriously laughed out loud. Awesomeness.
        .
        And we’d do it, too.
        .
        Hell, we’ll probably do it regardless . . . Charlie’s already boarded that train.

  32. Steve McIntyre
    Posted Aug 11, 2009 at 7:02 PM | Permalink | Reply

    I’ve added some first comments in the thread. It’s hard to know where to begin – this is a very embarrassing document.

  33. Mike
    Posted Aug 11, 2009 at 7:08 PM | Permalink | Reply

    I have been reading this website for over a year now. It seems to me that the only court that the obfuscators must respect would be that of the law. How do we prosecute this? How do we shine a light into these dark corners? Are there any readers who share the desire for truth and objectivity who have been called to the bar in the UK and have the passion and resources to help?

  34. Posted Aug 11, 2009 at 7:15 PM | Permalink | Reply

    Congrats Steve. These documents sound like a confession under duress. Wish someone would teach them how to use an html stylesheet – this plain vanilla html is so 1990s.

  35. Andrew
    Posted Aug 11, 2009 at 7:15 PM | Permalink | Reply

    “Nor for that matter do I think that climate policy should be delayed while this sort of thing is done.”

    What if it’s nonsensical climate policy? I’m not sure that this statement I quoted even makes any sense. You brought it (climate policy) up, BTW.

    Andrew

    Steve: I also asked readers not to discuss this position – we’ve done so in the past. Any “bad” policy is “bad” not because the dog ate CRU’s homework but for other reasons. If the only thing that stands against a policy is CRU mismanagement, then the policy is probably OK. And I don’t wish to debate this with you or discuss it on this thread.

  36. Steve McIntyre
    Posted Aug 11, 2009 at 7:30 PM | Permalink | Reply

    We hope in the future that we may be able to provide these data, jointly with the UK Met Office Hadley Centre, subject to obtaining consent for making them available from the rights holders.

    They’ve had many years to work on obtaining these “rights”. I wonder what concrete steps they’ve taken over the years towards removing any perceived encumbrances? :) Have they ever notified the WMO or IPCC of problems with encumbered data?

    I wonder it there are any documents evidencing that they’ve expressed this hope to any third party in the past. :)

  37. Armin
    Posted Aug 11, 2009 at 7:33 PM | Permalink | Reply

    Nobody would believe that they were that incompetent

    Eh, yes, I would. During my master’s degree in physics – already over 10 years ago – I worked for an IT company that worked for many mid-sized companies. I was amazed how poorly data archiving was. And on just about everything. From pay-checks, taxes, financials, their inventory, customers, etc. And if backups were made, they were rarely even checked on if the backups actually contained what was the intention.

    We should realise, that untill the Hockystick debacle nobody cared or thought about data availability or checking. We are talking about 25+ years of nobody checking anyones data in climate science. People trusted eachother (and possible – except for honest errors – could as the topic wasn’t politics then). So it makes more than sense to me, the data was lost, or only pro-processed data is available.

    Never assume wrong intent, when plain stupidity or even more common sloppyness can be blamed.

    Much more worrying is actually that only pre-processed data is available. That means assumptions once made, are now final and beyond dispute. Most likely even the exact proprocessing is not available either …

    Steve: Please don’t misinterpret my comment. In this case, I am definitely not imputing anything other than incompetence. The incompetence here apparently reaches beyond a lack of care with the data versions, but to the paper trail. As to your statement that prior to the Hockey Stick debacle, nobody “cared or thought” about data availability or checking. Maybe that was true in climate science, but it was definitely not true in the stock market. Such things are routine. I was dumbfounded at the attitudes in climate science and remain amazed.

    • Ryan O
      Posted Aug 11, 2009 at 7:37 PM | Permalink | Reply

      Re: Armin (#63), The only problem with this position is that, until just now, they had the data. Excuses about limited storage capacity in the 1980s were not present in 2002, when according to Jones himself a complete set of the underlying data was being prepared for posting. So unless they have a time machine and whisked themselves back to the 80’s to pull an Enron on themselves, incompetence is not the answer.

      • Steve McIntyre
        Posted Aug 11, 2009 at 7:47 PM | Permalink | Reply

        Re: Ryan O (#64),

        Ryan, I think that what Jones contemplated archiving (and did archive in Feb 2003) was the “value added” station data – not the “original” data.

        Of course, the “confidentiality agreements” didn’t seem to present a problem. Funny how they only became a problem for Climate Audit readers and Warwick Hughes.

    • BarryW
      Posted Aug 11, 2009 at 8:30 PM | Permalink | Reply

      Re: Armin (#63),

      Sheesh, this is the government we’re talking about. The stock market (theoretically) has watchdogs making them be disciplined. Who watches the watchers? For the government, no one but themselves usually. I ‘ve been associated my whole career with government agencies, including R&D facilities. I would have been surprised if they were able to give you what you asked for, not that they say they can’t.

  38. Andrew
    Posted Aug 11, 2009 at 7:43 PM | Permalink | Reply

    Andrew,

    No it wasn’t you, it was the “bad” Andrew who mismanages his comments. lol ;)

    Andrew

  39. John A
    Posted Aug 11, 2009 at 8:03 PM | Permalink | Reply

    What do you call a data product where at least part of the source material cannot be identified back to the source?

    It’s not a trick question. I’d really like to know.

  40. crosspatch
    Posted Aug 11, 2009 at 8:04 PM | Permalink | Reply

    It is very odd that an organization in the climate monitoring business should lose its climate data during a move. It is understandable that something could be misplaced in an office move but the thing that is the core of what an organization does is generally not the thing that is misplaced. And something else seems “fishy”, too. Haven’t there been others who have claimed to have used their raw data since then? It is quite possible I am mistaken but I was under the impression that other academics had access to that data since then and only the ones who wanted to “find something wrong with it” had been denied.

    They apparently didn’t lose their payroll records, or accounts receivable, or their Rolodexes. That an organization whose primary function is to provide analysis of such data should lose it puts them in a double-bind. They are either being deceptive or are unbelievably incompetent. There is no good way out of this for them.

  41. JimInIndy
    Posted Aug 11, 2009 at 8:24 PM | Permalink | Reply

    This is absurd, incompetent, and unscientific. I’ve been involved in data management since the early ‘60s. For source data storage and retention, there was original hard copy, microfilm, initial punched cards, mag. tape and printouts of interim files. Failure to store, index, and assure recoverability of such data (and related legal restrictions) is an admission of incompetence. The assurance of data integrity is incumbent on every scientific endeavor. Without retrievable (and reperformable) base data, calculated conclusions are worthless.
    CRU, with their several above responses, is declaring their findings/output to be scientifically unreliable.

  42. clazy
    Posted Aug 11, 2009 at 8:41 PM | Permalink | Reply

    If work cannot be replicated, how can it be considered science? I cannot get my mind around that one. I’m baffled that anyone takes work like CRU’s seriously. What joker came up with the expression “value added” to describe CRU’s data?

  43. Geoff Sherrington
    Posted Aug 11, 2009 at 8:45 PM | Permalink | Reply

    The fundamental purpose of this exercise must surely be to recover the earliest, least-adjusted records; and then to study the subsequent adjustments for logic, error, assumption etc.

    Two years ago I was told that CRU no longer held data I sought. This raises an intrigue as to whether in a legalistic way, CRU has assigned guardianship to another body which needs to be identified then FOI’d. See Kenneth Fritsch at #44.

    There are hints that a mass of data in early form exists. I tend to agree with Ryan O at #45 & #52 and Steve at #51.

    I wrote to our NMS a week or more ago, asking if we had an agreement of the type mentioned, but have had no response so far. Also, I have posted the copyright statements that now accompany public release of Australian data. I have not succeeded in finding copyright statement versions operating in the early 1990s.

    There are several graphs posted in recent weeks where Australian temp data are shown from various sources. So far, no systematics have emerged from the subsets I have examined, though it seems that the “most” original 1993 BOM set often plots higher than some others such as KNMI. It is not uncommon for a year to have a spread of 1 deg C from various current public sources. Presumably, Tmax and Tmin variation can be greater again.

    The 1993 BOM set is already homogenised to a degree that I have not examined. It is a lengthy job, station metadata sheets one-by-one for 1,000 or more stations.

    Having noted this, it is possible that the gaining of the CRU original data might be a hollow victory for future research, but a severe lesson for the conduct of climate science. The reason is that any “original” set that might be extracted from GB would have to be examined, piece by piece, to see what adjustments had already been made, if any, by the donating parties.

    In an overall glum view, if serious science is to be done on the instrumental temperature records, it really needs to be restarted from the ground up, with original data from NMS or country sources including metadata.

    We have learned a great deal (thank you Steve) about how the job should be done properly. In the meantime, I have refused to use CRU-Hadley data for any purpose other than to show inconsistency and uncertainty. I do not bother to read papers that use it as a base.

    Unfortunately, given the Australian provenance of a good deal of SH information and its prior poorly-understood homogenisation, many other countries will be in a similar position. I cannot recommend any global surface instrumental data set as being adequate for correlation/causation studies in climate science.

  44. Bill Jamison
    Posted Aug 11, 2009 at 8:45 PM | Permalink | Reply

    I’m not surprised that they didn’t keep the original data back in the 1970’s and 80’s. In their view, I imagine the raw data had no value. The “good” data was that which had been processed. You have to remember just how expensive data storage was back then. All you have to do is look back to the Y2K problem to understand just how far programmers would go to minimize the data stored. Was it shortsighted to not keep the raw data? Absolutely! But seriously who could have predicted the current importance of temperature records from obscure location around the globe from 40 years ago???

    Anyway, I don’t consider the raw data to be “lost” but rather a business decision was made that it was unimportant to retain it.

    • Kenneth Fritsch
      Posted Aug 11, 2009 at 8:51 PM | Permalink | Reply

      Re: Bill Jamison (#76),

      I’m not surprised that they didn’t keep the original data back in the 1970’s and 80’s. In their view, I imagine the raw data had no value. The “good” data was that which had been processed.

      The original data is the heart of the data base and without it how would or could CRU justify the value added part – or ever go back for a Version 2 as USHCN did. GISS rules.

    • TAG
      Posted Aug 12, 2009 at 3:32 AM | Permalink | Reply

      Re: Bill Jamison (#76),

      I’m not surprised that they didn’t keep the original data back in the 1970’s and 80’s. In their view, I imagine the raw data had no value. The “good” data was that which had been processed. You have to remember just how expensive data storage was back then.

      In the 70s and 80s, on line memory was expensive. Off line memory was very cheap. The data could have been placed on t 9-trackape and kept in storage, If someone wanted access to the data, they would send a message to the computer center and ask for the tape to be mounted. In my experience, this took only a few minutes.

      There would be nothing unusual about this. Any university would have the facilities to do this. Pictures of computer centers of that era commonly have racks of 9-track tapes shown in them. One tape would have sufficient storage for all temperature data both raw and processed.

    • rephelan
      Posted Aug 12, 2009 at 10:57 AM | Permalink | Reply

      Re: Bill Jamison (#76),

      “You have to remember just how expensive data storage was back then. All you have to do is look back to the Y2K problem to understand just how far programmers would go to minimize the data stored.”

      The problem wasn’t data storage, Bill, but rather the very limited amount of core memory available (RAM) for volatile processing. With a memory space today measured in gigabytes I would have no problem programming a huge, multi-demensional array and retaining and processing it in memory. As recently as the 80’s we were too often restricted to 32 – 64 Kilobytes of RAM: far too little to hold a data array and the instructions to process it. Intermediate structures had to be written back to “temporary” files and then read back in as needed. Even in those days the biggest time constraint to processing was I/O, and jobs running in seconds today took hours back then. Yeah, we economized on dates back then, but it was due to processing time and memory constraints rather than the cost of storage. There is no good excuse for losing scientific data.

  45. AnonyMoose
    Posted Aug 11, 2009 at 8:56 PM | Permalink | Reply

    Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, …”

    * That implies both that they know that they lost data for some sites (which ones?) but not others (which ones?).
    * Have they tried to contact the relevant agencies (when?) so they can complete their database for modern analysis?
    * Why haven’t they been trying to reanalyze their old data, when GISS and NOAA seem to often reanalyze and adjust their data?
    * What policy decisions, studies or research are the basis for their decision to not reanalyze data?
    * What are their quality control procedures to ensure that their data is valid or useful?
    ** This Met Office page mentions ISO 9001 standards, but the page only refers to data they gather from UK locations.
    * Have they already studied their data and decided that it is not valid or not useful, or not worth maintaining?
    * How much to they spend to maintain this data?
    * What are they doing with the data maintenance funds?

  46. jim edwards
    Posted Aug 11, 2009 at 8:57 PM | Permalink | Reply

    BAHRAIN:

    There is NO confidentiality agreement with Bahrain, if the posted document is all they have. As James Goneaux [#18], above, noted, Jones and Hulme didn’t sign anything.

    More than that, however, it’s clear Jones and Hulme asked for data and it was freely given by the Bahraini agent. The agent didn’t withhold production of the data until he received a written or oral promise of confidentiality.

    Only after giving the data did he ASK,

    “Please do not provide third parties with this data…”

    ‘Please’ does not create a condition of use.

    SPAIN:

    There is also NO confidentiality agreement with Spain.
    It’s convenient that the portion of the agreement which was attached is now missing

    Area geografica:
    Spain – see attached list of 38 stations

    As Willis notes, above [#13], at the location for date and signature [FECHA Y FIRMA], the boilerplate requires data provided to be used for purposes specified and to be cited in all publications. Has this been done ?

    Please note that there is NO SIGNATURE at the signature line [FECHA Y FIRMA]. No signature generally = no agreement.

    Willis kinda gets it wrong about “public purpose,” however. Hulme describes himself as a member of the public [Empresa: Publica].

    The directions for filling out the form tell the applicant to “Briefly describe the use for which you are requesting this data.”

    Describir brevemente para que fines solicita la informacion. (section 2, third page)

    In the portion of the form set aside for Hulme to describe his planned uses, for which he might, hypothetically, be limited to, he states:

    2 – Informacion Meteorologica requerida:
    Solicita para:
    “I require monthly climatological normals based on 1961-1990 for nine variables and for 38 stations”

    So, we have in black and white that Hulme merely stated he wanted the data; he never stated that he intended to use it for any particular purpose. The Spanish gave him the data without any restrictions.

    Even IF he had signed the request, there would be nothing to bind him to. He’s free to use it however he wishes.

    NORWAY:
    There may be SOME confidentiality requirements tied to Norwegian data, but given the age of the data – I doubt it. Bjorne Aune provided both “a report and a disk” to Hulme in 1993. Confidentiality requirements were only attached to the then-unpublished data found on the disk.

    The disk and the following precipitation data [to be sent later by Aune - J.E.] are sent to you without charge. The condition is that you do not use them commercially or give them to a third party.

    DNMI wanted to protect gathered, but unpublished data back in 1993. It is arguably reasonable for a Norwegian public institution to control the use of their data for a short period of time, until publication. The purpose to which DNMI attached confidentiality requirements has long passed. It’s been sixteen years. They’ve had their period of exclusive use to feather their own curricula vitae. They don’t care about confidentiality, anymore.

    Stronger still, there’s no signature by Hulme – agreeing to the proposed condition. He likely isn’t bound to the terms of the agreement.

    ORAL AGREEMENTS:
    There are NO confidentiality agreements based upon oral promises.

    One could always raise the question of authority to enter into these agreements, but before getting there, anybody who attempted to enforce one of these agreements would have to deal with the Statute of Frauds , which generally requires certain agreements to be in writing [and signed by the bound party] in order to be enforceable.

    Among the agreements covered by the Statute of Frauds are agreements which, by their nature, cannot be completed within a year. An agreement to be a sole supplier or retailer of goods for sixteen years cannot be completed within a year, so it is covered by the Statute of Frauds . The agreement must be in writing or it’s unenforceable.

    This will likely be true for confidentiality agreements in the UK. Short ones [11 months] may be oral, long ones [> 1 year] must be written and signed by the party bound [Jones / Hulme].

    Any oral agreements were legally void or have long-since expired.

    • Steve McIntyre
      Posted Aug 11, 2009 at 10:35 PM | Permalink | Reply

      Re: jim edwards (#79),

      Jim, note that there’s a bit of a disconnect between the paraphrase of the regulation and the regulation itself.

      Palmer’s refusal letter used the terminology:

      Reg. 12(5)(f) – Adverse effect on the person providing information – Information is covered by a confidentiality agreement…

      Regulation 12(5)(f) applies because the information requested was received by the University on terms that prevent further transmission to non-academics

      12(5)(F) doesn’t actually use the term “confidentiality agreement” although it is related. Here’s what 12(5)(f) says:

      (5) For the purposes of paragraph (1)(a), a public authority may refuse to disclose information to the extent that its disclosure would adversely affect –

      (f) the interests of the person who provided the information where that person—
      (i) was not under, and could not have been put under, any legal obligation to supply it to that or any other public authority,
      (ii) did not supply it in circumstances such that or any other public authority is entitled apart from these Regulations to disclose it, and (iii) has not consented to its disclosure;

      Interpretation of the exemptions is at http://www.defra.gov.uk/corporate/opengov/eir/guidance/full-guidance/pdf/guidance-7.pdf.

      Is the phrase “did not supply it in circumstances such that or any other public authority is entitled apart from these Regulations to disclose it” equivalent to being governed by a confidentiality agreement?

      I wonder if there are any cases on this clause.

      • jim edwards
        Posted Aug 12, 2009 at 12:56 PM | Permalink | Reply

        Re: Steve McIntyre (#87),

        Did you notice the case that was linked in footnote #13, at the bottom of page 18 of the the guidelines for interpretation you linked to ?

        [* I can’t get the link to work, but the BNRR case is at http://www.birminghamfoe.org.uk/bnrr/judge1.htm%5D

        The case involves “environmentally information” provided to the government within the terms of a confidential contract to build and operate a toll road, pursuant to a publicly-granted monopoly.

        A couple of environmentalists decided they wanted to know the terms under which this preferential deal was made and requested data, then litigated when it was denied.

        The court went through a number of exceptions, focusing most of its efforts on confidentiality of clearly commercially valuable data and practices. [i.e. - not really relevant to this discussion]

        At the very end of the case, the court looked at the analogue of 12(5)(f), but in a very confusing way.

        The old numbering for EIR 12(5)(f) was, apparently, 4(3)(c).

        A few things came out of the case:

        In the BNRR case, BOTH the government and MEL [the party building the toll road] were opposed to disclosure of “environmental information.” [i.e. 12(5)(f)(iii) - no question consent was denied by the 'person' who provided the information]

        In BNRR, the government received and used information to come to a determination, but it NEVER subsequently disclosed that information to a member of the public. This may be important as you allude to in your #88.

        If any consideration is paid for data, it fails the analogue of 12(5)(f)(i), and so no exception applies.

        Since MEL was negotiating details within the framework of a presumed “best endeavors” clause [after they already won the bid for the government contract], the information it gave the government was not voluntarily “supplied” within the meaning of the regulation, and was, therefore, not excepted from disclosure.

        Other information was provided by third parties [shareholders of MEL, et AL.] who were not parties to any negotiating requirements established by agreement. Such information was not entirely detailed but included the “Sponsor’s Support Agreement and a letter from the Concessionaire’s Financial Adviser.” This information wasn’t provided under a legal obligation [meets 12(5)(f)(i)]. The court noted that there was no evidence that consent was given, and further, “in view of the description of the documents in CMD1 it is inherently unlikely that they have consented.”

        An important factual distinction was that the parties who provided information, and were colluding with the government to prevent disclosure, actually owned the information they were attempting to protect. Can the same be said for EVERY person who supplied information to Jones / Hulme ?

        Consent:
        So, assuming that 12(5)(f) applies to the information you’re looking for, a court might look at the inherent likeliness that a private party would withhold consent. Consent seems less likely withheld for 30 year-old weather data than for private accounting statements.

        Consideration:
        I think another argument can be made that information was provided for consideration, or likely provided for consideration, or pursuant to a pre-existing agreement, or that it cannot be shown that information wasn’t provided for consideration or pursuant to an agreement. Any of these circumstances would cause a legal obligation to arise, and 12(5)(f)(i) would not be met. No exception would arise.

        Consideration could include money, as the Norwegians suggest.

        Consideration could also include a promise to provide a future benefit, like being named as a coauthor on a paper published in a prestigious journal.

        A promise to use one’s data and credit its provenance might also be consideration, where it’s reasonable to expect that being named as an important data source might bring prestige and continued third-party funding.

        Promises wouldn’t necessarily have to be reciprocal. It’s possible for one party to fully perform [i.e. - deliver the information] in expectation that Jones or Hulme would fulfill promises in the future.

        Ownership:
        If I steal/convert information, I don’t magically acquire rights to direct how it may be used. Unlike in the BNRR case, I think there may be genuine issues of ownership present. The regulation doesn’t explicitly state so, but a “person” under 12(5)(f) must be an owner of the information, in order to withhold personal consent.

        Assuming that there are agreements to keep publicly-funded information confidential, can Jones show that the parties who gave him the data had the legal right to withhold consent to distribute the information ?

        Public Authorities and International Law:
        An interesting question to me is whether foreign “public authorities” remain “public authorities” or become “persons” [for the purposes of 12(5)(f)] when they give information to UK “public authorities” under EIR.

        At first blush, the interpretational guidelines basically define only UK institutions [minus the Scottish ones... they have their own EIR] as “public institutions.”

        But the EIR seems to be a departure from normal UK law, like FOIA. The interpretational guidelines note that the purpose of EIR is to implement the Aarhus treaty. [See, e.g., interpretational guideline 4.12.] I think that there is a reasonable argument that a Finnish meteorological agency doesn’t count as a “person” for the purpose of 12(5)(f). An agent of a foreign “public authority” would, likewise, not be a “person” for this exception, as they aren’t providing information that belongs to themselves, personally.

        Non-contractual confidentiality agreements:
        Phil Jones may, in fact, have a legitimate moral duty to prevent disclosure of borrowed data, based upon his prior oral commitments to the agents of foreign public authorities. Dr. Jones’ gentleman’s agreements are NOT binding on the UK, however. His personal failings, if any, don’t keep the UK from being “entitled” to disclose data in their possession.

        To quote from the BNRR case:

        Even if an estoppel operates as between the Secretary of State and MEL that cannot bind third parties exercising their fight to seek environmental information under the Regulations.

        But merely because there has been a common mistaken assumption it does not follow that there must be an estoppel: it must be unjust or unfair for one of the parties to resile from the convention, and the estoppel will apply “only for the period of time and to the extent required by the equity which the estoppel has raised”: see Bingham LJ (as he then was) at p. 352 of The Vistafjord.

        The public inquiry was held in 1994 – 1995. It is now 1998 and the Secretary of State has decided that the BNRR shall be built by MEL. Even after this lapse of time it might still be unconscionable for the Secretary of State to decide, of his own motion and for some purpose of his own, to release details of the Agreement. But it does not follow that it would be unconscionable for him to disclose those details pursuant to a request made to him under the Regulations .

        So, after the data has been used and three years have passed, Jones’ gentleman’s agreements may bind him, but it doesn’t follow that the data necessarily can’t be supplied pursuant to a request for publicly-held information. Jones would only be bound if somebody stood up and said. “Hey, that’s my information !” It sounds like he doesn’t even remember his promises, if any.

    • QBeamus
      Posted Aug 12, 2009 at 1:46 PM | Permalink | Reply

      Re: jim edwards (#79), Your argument regarding the statute of frauds isn’t quite water-tight. In most jurisdictions, the statute of frauds is now regarded with disfavor, which, under the currently popular judicial philosophy (which I call by the inflamatory title, “the imperial judiciary”) means it is construed narrowly–bizzarrely so. In most jurisdictions, the statute of frauds is read to apply only to agreements that cannot possibly be executed in less than a year. This means that an oral agreement can remain ienforceable, theoretically indefinately, as long as it is possible for it to end sooner. In this specific case, for example, if the agreement were to maintain the data in confidence until the source published something with it, that agreement would not fall under the statue of frauds in most jurisdictions, since the publication could possibly happen in less than a year.

      Even in jurisdictions that apply a more traditional, broader rule, the issue would be controlled by the intent of the parties. That is, if the parties entered that hypothetical agreement believing that the publication would happen in 9 months or so, then it would be enforceable, even after 16 years, but if they believed publication would take 15 months, the agreement would be unenforceable. (However, even in the former case, after 16 years the contract might be–likely would be, IMO–reformed by a court due to the situation being so far outside what the parties contemplated.)

      None of this should be read to contradict your larger conclusion. To the contrary, I agree; I see no evidence of any confidentiality agreement applicable to the present issue, namely public disclosure of the HADCRU raw data.

  47. jim edwards
    Posted Aug 11, 2009 at 9:03 PM | Permalink | Reply

    The admission that confidentiality agreements have been lost means that the confidentiality excuse can’t be used to deny documents under FOI.

    All a person has to do is appeal one’s denial and state that you question the existence of any agreements legally limiting the proliferation of the data. The denying agency would have to present the agreements at appeal. It’s up to them to prove that you don’t deserve the data.

    No agreements = no proof.

    No proof = reversed denial.

  48. Dave Dardinger
    Posted Aug 11, 2009 at 9:09 PM | Permalink | Reply

    Ok, so what was the data which was unearthed by the mole? Steve reported that he wasn’t told the material was not public at one time. And since he wasn’t told to destroy it, it should now be available for examination. So is it part or all of the HADCRUT original data or not? If not, what was the big deal in the first place? If it is original data, does that contradict what CRU now says?


    Steve:
    The “mole”, who appears to have been Phil Jones himself, placed station data online in Feb 2003 (versions from 1996 and 1990 were online as well.) This was the data which, in 2009, was supposed to be subject to “confidentiality agreements”. Up to July 27, 2009, CRU had simply disregarded the supposed agreements.

  49. Gerald Machnee
    Posted Aug 11, 2009 at 9:26 PM | Permalink | Reply

    Is there such a person as an Auditor General there that can be asked to examine their methods?

  50. Alan Wilkinson
    Posted Aug 11, 2009 at 10:07 PM | Permalink | Reply

    Standard letter received in respect of CAROLINE ISLANDS, SOLOMON ISLANDS, WALLIS ISLANDS, COOK ISLANDS, NIUE ISLAND

  51. Calvin Ball
    Posted Aug 11, 2009 at 10:13 PM | Permalink | Reply

    Imagine [you fill in the name] saying something like this:

    Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s.Nobody would take it seriously.

    Nobody would believe that they were that incompetent. I wonder what would happen if Lonnie Thompson moved offices. Would he lose all his unarchived ice core data?

    Not quite the same thing. Losing agreements should a lot more difficult than losing data, because agreements should be kept at the attorney’s office. This is doubly incompetent.

  52. MarkB
    Posted Aug 11, 2009 at 10:13 PM | Permalink | Reply

    Granted all the harrumphing about the loss of data, or the missing confidentiality agreements, but it does make me think of the recent news that the original film of the first moon walk has been lost by NASA. As reported recently in the news, the degrated television video is what we all remember from seeing it over the years. The actual high quality film was put away years ago, and probably thrown out. If it doesn’t occur to anyone at NASA to save the original film of the first humans walking on the freakin’ moon, I can well imagine a British climate agency losing track of sets of temperature data from around the world.

  53. Alan Wilkinson
    Posted Aug 11, 2009 at 10:16 PM | Permalink | Reply

    A UK citizen must be able to bowl over the limited and now documented evidential basis for refusing to supply the data based on UK law if Jim Edwards is correct.

    Even if it necessitated seeking a release from one or more suppliers or simply excluding those countries.

  54. Steve McIntyre
    Posted Aug 11, 2009 at 11:19 PM | Permalink | Reply

    The interpretation bulletin says:

    7.5.7.1 The exception only applies where:

     there are no other circumstances that entitle the public authority to disclose it, and

    If there are “no other circumstances that entitle the public authority to disclose it”, then CRU could not have disclosed the data to Peter Webster. So obviously they were circumstances that entitled the public authority to disclose the data.

    They also disclosed the data to the Met Office.

    They also sent a version to me in 2002.

    They also placed versions on their website (taken down only in July 2009.)

  55. Harry Eagar
    Posted Aug 11, 2009 at 11:57 PM | Permalink | Reply

    While I admire everybody’s persistence in going through channels, in my business (newspaper reporter), if I get the stall and runaround like this, I go to the top.

    Not sure who the top would be here. Minister of Defence?

    The minister is an MP, right? So he has a top, too. Several, starting with his constituency committee, I believe it is called.

    You’re proved that going through channels, although amusing, is not productive.

  56. Julian Flood
    Posted Aug 12, 2009 at 12:24 AM | Permalink | Reply

    I’m surprised there has, as yet, been no refusal on cost grounds.

    JF

  57. Posted Aug 12, 2009 at 1:22 AM | Permalink | Reply

    Sounds to me like a serious case of “We don’t want to show you the real data because big chunks of it blow away our findings so we ignored them.”

    Steve: As I’ve said over and over, I do not share that opinion.

  58. Geckko
    Posted Aug 12, 2009 at 1:46 AM | Permalink | Reply

    This is now getting close to an end game. The prosaic truth is far from a Robert Ludlum novel, simply: “we run such a sloppy house that we don’t even know what we have or have not lost”.

    The reputation saving obsfucation crumbles as it must. Oh what tangled webs we weave…

  59. Alan Wilkinson
    Posted Aug 12, 2009 at 2:47 AM | Permalink | Reply

    Isn’t the term “non-academics” really just a proxy for “commercial”?

    • AnonyMoose
      Posted Aug 12, 2009 at 11:41 AM | Permalink | Reply

      Re: Alan Wilkinson (#97),
      No. Forbidding non-academic access restricts it to whatever “academic” means. Forbidding commercial access would still allow access by anyone else — cryptologists looking for secret messages which might have been inserted in data streams, artists needing museum selection information, trainspotters looking for rolling stock in good condition, nonprofit global regatta founders.

      Re: Eric (#130),
      No, the rumor was that NASA had found the lost lunar walk footage. They haven’t, they’re merely restoring TV footage. NASA found records that the original video (the only one stored on a data tape rather than video tape) was recycled for other data. There were two test tapes, which due to being an equipment test were outside the normal record handling, whose status is still unknown.

  60. Tony B
    Posted Aug 12, 2009 at 2:54 AM | Permalink | Reply

    Regarding routes that could be taken, perhaps this stunning situation should be raised with:

    1. The Information Commissioner – it is the responsibility of the IC to ensure that the FOI rules are adhered to
    2. The Commons Public Accounts Committee – they have the power to audit the performance of public bodies (but are of course on vacation for weeks to come)
    3. The Shadow Environment Secretary (I would not bother with the Government’s ES, go to the Conservative opposition representative)
    4. The Daily Mail, The Daily Telegraph, who will get great pleasure in exposing this sham of a government department
    5. Channel 4 TV who might well be interested, given that they did commission and broadcast “The Great Global Warming Swindle”

    I do sincerely hope that the end game is approaching here. If the UK’s most “prestigious/authoritative” climate organisation cannot/will not find the requested data and agreements then they are either terminally incompetent, or not to be trusted.

    Surely even the BBC could get that?

  61. Phillip Bratby
    Posted Aug 12, 2009 at 3:11 AM | Permalink | Reply

    Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.

    In my experience in the nuclear industry, all data before electronic storage was kept on duplicate microfilm or microfiche, the copies being stored in separate locations (e.g. one copy down a salt mine, the other in an on-site store). No rocket science needed – easy and relatively cheap.

    • Geoff Sherrington
      Posted Aug 12, 2009 at 3:44 AM | Permalink | Reply

      Re: Phillip Bratby (#99),

      Agreed,

      In 1974 the Australian Government became less than amused by the antics of its intelligence and security sectors and set up a Royal Commission. My mate George was Secretary and did most of the writing for the Judge.

      I was reading some of the report today and was struck by the parallels. Here are the words from 1975-6. The records are preserved in their entirety to this day. Of course it could be done properly.

      1. In the course of my inquiry, the Commission has accumulated some twenty shelf-metres of records. They comprise submissions by individuals and organizations; tapes and transcripts of hearings, and records of less formal ‘conversations'; exhibits tendered at hearings; submissions, documentary evidence literature, and briefings, provided by intelligence and security agencies and government departments, and by overseas agencies; correspondence and subject files; report drafts and working papers; and lists, registers and indexes.

      2. The question arises as to what will become of this material after my inquiry concludes. It is rendered both more difficult and more vital, by the facts that the great majority of items are highly classified, and that I have made directions … that some material not be published.

      3. The restrictions on use of records through the system of security classification and clearance is legally unsound. And there are various doubts regarding directions against publication under the ……

      4. Even so, the sensitivity of much of the records copels me to make some recommendations for the future. What is done is, of course, a matter for government.

      5. The objectives to be achieved in the disposal of the Commission’s records are two:
      (a) The continuing security of the records against improper use.
      (b) The continuing availability of the records for proper use.

  62. Pierre Gosselin
    Posted Aug 12, 2009 at 3:44 AM | Permalink | Reply

    They have the raw data, but don’t want you to have it. You’re getting the bigtime runaround. This raw data must be of very high value if it’s worth ruining the very reputation of the institute entrusted to keep it.
    I suppose the possibility exists that the CRU is really completely in shambles, as they say. If so, then it should never be used by or cited in any credible scientific works.

  63. Chris Wright
    Posted Aug 12, 2009 at 4:40 AM | Permalink | Reply

    @ Tony B,
    It just happens that Nick Herbert, the Shadow Environment Secretary, is my MP. Funnily enough, while reading the Daily Telegraph this morning, the idea of writing an email to him crossed my mind.

    snip – overeditorializing

    I think I’ll compose an email in the near future but wait until Parliament comes back from its well-deserved holiday. Quite likely Nick will be away on holiday during the coming weeks – probably in a country with a significantly warmer climate.
    Chris

  64. Posted Aug 12, 2009 at 4:53 AM | Permalink | Reply

    They have lost the original data? Those guys are not serious! The CRU temperature record might as well be a series of random number. It should not be used by anyone anymore.

  65. Tony B
    Posted Aug 12, 2009 at 5:43 AM | Permalink | Reply

    @Lars K – 105

    Wasn’t someone else using random numbers too (allegedly…)? Or might as well have been, according to some very useful local detective work.

  66. Tony Hansen
    Posted Aug 12, 2009 at 5:45 AM | Permalink | Reply

    If I am the gatekeeper for some historical data, and a most unfortunate set of circumstances leads to the most unfortunate loss of the original data…..then could I reasonably make and/or justify any adjustments to the historical (lost) part of the record after the losss of said data?

  67. Tony Hansen
    Posted Aug 12, 2009 at 5:47 AM | Permalink | Reply

    re: #107
    ‘after the loss of said data’
    sseriously ssad sspelling.

  68. Tony B
    Posted Aug 12, 2009 at 5:55 AM | Permalink | Reply

    @ Chris Wright – 104

    Isn’t the pre-requisite for a successful political career the ability to say the right things at the right time – oh, and that’ll be the same for a climate science career too, I suppose…

  69. Cardin Drake
    Posted Aug 12, 2009 at 6:18 AM | Permalink | Reply

    With all the problems in the U.S. data sets, you had to know the world-wide data had many issues as well.
    snip

  70. Craig Loehle
    Posted Aug 12, 2009 at 6:23 AM | Permalink | Reply

    Lack of computer space? Please. Even 20,000 stations daily data for 30 years would still fit on a flash drive or an old 9-track tape. I ran simulations in 1980 that generated that much output.

  71. Alexander Harvey
    Posted Aug 12, 2009 at 6:29 AM | Permalink | Reply

    A quick question:
    Who holds the original data logs etc. for the majority of the planet, i.e. the oceans? Hopefully it is safe somewhere.

    I am not sure, but I believe that Hadley SST data is used in other global data sets. Should there be a traceability issue with this data then it will be inherited by such other datasets.

    Also does the IPCC have any standards regarding the traceability of data products used in the construction of its reports?

    Steve BTW, I am having real problems posting comments via IE8 (Typing not appearing on the screen for up to a minute. This missive comes to you courtesy of Opera 9.26.

    Alexander

    • Steve McIntyre
      Posted Aug 12, 2009 at 6:43 AM | Permalink | Reply

      Re: Alexander Harvey (#111),

      Hadley SST is run by a different operation. ICOADS is the raw material for gridded SST and there is voluminous material online and there seems to be concerted effort to preserve the metadata and information attached to the measurements, not just the number.

      The SST people seem to have inherited maritime traditions of care with logbooks etc. Many logbooks from vessels have been preserved for decades.

      CRU’s reckless lack of stewardship with land data entrusted to them is totally at odds with the UK maritime tradition of stewardship of maritime data.

  72. Craig Loehle
    Posted Aug 12, 2009 at 6:31 AM | Permalink | Reply

    If they simply tag new data for a station onto old data, which is what it sounds like they are saying, and that the old data is not the raw but their processed data, then any corrections made by a country to past data for homogeneity or scribal errors etc, will never make it into the CRU database.

    • Ryan O
      Posted Aug 12, 2009 at 7:13 AM | Permalink | Reply

      Re: Craig Loehle (#112), This is why I believe they have at least a portion of the original data. While they may not maintain a duplicate of the GHCN database the idea that they have no original data for stations outside GHCN does not, in my opinion, pass the common sense test.

  73. whitty
    Posted Aug 12, 2009 at 7:13 AM | Permalink | Reply

    first time poster, long-time reader.

    As a government employee in the states, I would expect to be “terminated” were I to react in this fashion. It is this type of behavior which makes citizens distrustful of all government employees. “Who’s” hiding “what” and “why” – disgusting.

  74. Mac
    Posted Aug 12, 2009 at 7:18 AM | Permalink | Reply

    The loss of the original raw data means the original and subsequent science cannot now be verified.

    Human error or a human convenience?

    No ready answer, but we can take it that CRU now suffers from a lack of credibility.

    Just think of all those papers, journals, books, scientific news-pieces and editorials which referenced CRU value-added data that now have a large ? beside them.

  75. Alexander Harvey
    Posted Aug 12, 2009 at 7:40 AM | Permalink | Reply

    Thanks Steve,

    That realy is good news regarding SST data archiving, they have saved two thirds of the planet.

    I have to say that just uncovering the loss of the original land data alone, has probably made all your efforts worthwhile.

    Ignoring the politics and the amusement that can be had over this, how can this be fixed?

    Whatever your point of view and beliefs on climatic change, reliance on the integrity and traceabilily of the temperature record is likely to be a paramount concern.

    It seems that they have gone to a lot of effort to produce a very important data product and have let the foundations and perhaps several of the lower floors wash away. Leaving a “Castle in The Sky”. It is a tradegy.

    Hopefully someone somewhere has got enough of the original data to at least start to unpick this mess. Hopefully the CRU have made the original data available at sometime to someone and they have kept portions of it.

    Question to anyone: Are other data sets, e.g. GISS reliant on CRU value added land temperatures for any regions/countries, or do they hold all the original data to which they add value?

    Otherwise I fear that the MET/CRU may be the only data repository for ex-Britih Empire countries prior to independence. Or rather they should have been but neglected to be so.

    No joke, I feel sick about this. Not a good day for the UK.

    Alexander

  76. Charlie
    Posted Aug 12, 2009 at 7:47 AM | Permalink | Reply

    Does this exercise really matter? Can we expect the specific data gathering, collation, error checking, and adjustment process to affect the temperature record in a significant way, or are we just chasing ghosts.

    To educate myself on the variations between the various temperature products, I googled “giss vs hadcru3 temperatures”.

    Interestingly, the 1st result was a Climate Audit post, the 2nd result was Lucia’s Blackboard, and the 3rd was Watts Up With That.

    The Climate Audit Feb 2007 post, HadCRU3 versus GISS showed the difference between the HADCRU3 grid and the USHCN-2000 gridded product across the lower 48 USA. An difference in annual temps of about -0.35C in the 1930’s to +0.28C recently.

    So it appears that the DIFFERENCE in adjustments between USHCN-2000 and HADCRU3 is on the same order as the observed change in temperature.

    We already know from other posts that the total adjustments made by NOAA to the US surface station record are about the same size as the long term changes in temperature, but the above result showing that the differences in adjustment made by two sets of scientists also approach the magnitude of observed warming was somewhat surprising.

    The bottom line is that it tells me that this exercise in attempting to replicate HADCRU3 is worth the effort.

    • stan
      Posted Aug 12, 2009 at 9:55 AM | Permalink | Reply

      Re: Charlie (#118),

      It matters for even bigger reasons (for the AGW climate scientists even more than the public). The public has been told that they must defer to the expertise of climate scientists who supposedly have reached a consensus. Any information which casts light on the quality of the science practiced by the “experts” gives the public guidance on whether to acquiesce to the experts’ judgment.

      Over the last few years the curtain has been drawn back quite a bit. The sloppiness of so much of what passes for “science” in the climate area has been truly astonishing. The public really deserves to see everything behind the curtain.

      Note, for their own good and the public’s, I think it is imperative that those scientists whose scientific practices are top quality be as transparent as possible. The revelations of sloppy science by so many of their contemporaries threatens to reach a tipping point (if it hasn’t already) which will cause the public to lose confidence in all of them. That would be very unfortunate for the public and disastrous for the quality scientists.

    • KevinUK
      Posted Aug 18, 2009 at 11:33 AM | Permalink | Reply

      Re: Charlie (#118),

      “So it appears that the DIFFERENCE in adjustments between USHCN-2000 and HADCRU3 is on the same order as the observed change in temperature.

      We already know from other posts that the total adjustments made by NOAA to the US surface station record are about the same size as the long term changes in temperature, but the above result showing that the differences in adjustment made by two sets of scientists also approach the magnitude of observed warming was somewhat surprising.

      The bottom line is that it tells me that this exercise in attempting to replicate HADCRU3 is worth the effort.

      I agree. I think at the risk of being snipped for editorialising, that what Charlie has posted needs to be repeated. Because of various ‘issues’ (e.g. TOBS, UHI, satellite orbital decay, bucket adjustements etc etc) with all FOUR anomaly indices which result in the need to apply various adjustments to the raw data, the adjustments that are made to the ‘raw’ data whether it be ground stations or satellite or radiosonde balloon data etc are of the same magnitude as the claimed warming trend (claimed by some to be unprecedented in the last 1000 years!) in mean global surface temperature over the 20th century! Given the levels of uncertainty/variability in these measurements this means that we CANNOT be certain to any significant level of confidence that the claimed warming is nothing more than an artefact of the adjustments made to the raw data. This IMO is the main conclusion that can be reached from the auditing work that Steve has carried out so far on his blog. This IMO is the main reason why Steve’s request for raw data and details on how the data is adjusted are resisted/opposed by the ‘gatekeepers’.

      Since this THE key piece of evidence on which the IPCC claims that man (due to his continued use of fossil fuels) is having an appreciable effect on the earth’s climate, then it is clear that their claim is manifestly unsupportable. In reality because of the ‘issues’ with how we measure the earth’s temperature in the past and present continue the exist then the bottom line is we wil most likely never know to any significant level of confidence whether or not man has had or ever will have an appreciable effect on earth’s climate! Now given even if we could delude ourselves that we are are having an apprciable effect on our climate, the claimed warming trend is so small as it is (its most definitely not unprecedented and is NOT signifacantly greater than recent climatic warming anomalies like the RWP and MWP which were clearly beneficial to man) then why on earth are we introducing legislation to control the level of our emissions of a trace gas (vital to the continued existence of all life on this plant) into our atmosphere?

      KevinUK

  77. William Hughes
    Posted Aug 12, 2009 at 8:39 AM | Permalink | Reply

    Whithout attempting to justify the QA/QC (or lack thereof) in this data there is a point which can help explain the obfuscatory nature of *some* of the correspondence.

    Consultants routinely insist on a “no reliance by third parties” clause when releasing information or reports to their customers. The intent of the clause is to prevent, particularly in certain large North American countries, torts by persons who relied on the information without understanding the context. Say, in a feasibility study, for example.

    Let’s say I prepare a report for a junior exploration firm discussing what their mine might look like and what it might cost. I neglect to add the clause. They post excerpts fromt the report in their quarterly statements. Shares go up. Nothing is there. Shares go down. Consulting firm gets sued out of existance.

    The clause also ensures that data that is subject to change and revision is refreshed from the source each time. QA/QC can take effect. Potential customers are required to contact the consultant, develop a relationship, and maybe provide the consultant with more work.

    Now – lawers being who they are – this type of language gets more and more complex and people in the company who don’t understand WHY the clause is there pedantically insist on it’s inclusion on anything and everything without regards to its relevance. It can make it look like the consultant won’t stand behind their report. All sorts of terrible things (like losing a customer – worst possible) can happen.

    In this case it appears that the clause is being retroactively added to documents and data. Older data sets (pre-80’s) probably didn’t have any such thing as this issue did not exist in its current form then. The folks who reply to FOI requests would seem to be the exact type of staffers who would *assume* that such a “no release to third party” clause would apply to everything.

    Perhaps a useful FOI request is to inquire for a copy of the CRU current policy regarding the “no release to third party” clause, the standard text of the clause, and when it was first applied consistently as a policy.

  78. Don Keiller
    Posted Aug 12, 2009 at 8:39 AM | Permalink | Reply

    This would be comical if it were not for the fact that databases like CRU are used to formulate Government policy.

    With your pernission, Steve, I would like to use the information in this blog to
    a) inform some of the less climate-fevered press about these shenanigans
    b) make a formal complaint to Ed Miliband M.P. Head of the new (U.K) Climate and Energy Department”

  79. Douglas Hoyt
    Posted Aug 12, 2009 at 9:49 AM | Permalink | Reply

    I was told that the cruwlda2.zip data already has a correction for the UHI (presumably 0.05 C/century). By implication, so does newcrustnsall.dat.Z and later versions of the data. Thus, if anyone asks for raw data, they are asking for a data set that is NOT cruwlda2.zip or newcrustnsall.dat.Z. The earlier data may not have been retained, as they say, after the UHI correction. Therefore, they can truthfully deny all FOI requests for the raw data. It is rather tricky way for them to respond.

    The FOI request perhaps needs to be more specific, asking for the updated version of newcrustnsall.dat.Z and information on the known differences between this data and the raw data.

  80. dougW
    Posted Aug 12, 2009 at 10:22 AM | Permalink | Reply

    After what point in all this do the FOI requests end and the Lawyers are brought in?

  81. Robert Wood
    Posted Aug 12, 2009 at 10:25 AM | Permalink | Reply

    UK Met Office

    Arrangements have been set in place whereby bona fide academic researchers working on agreed NERC-endorsed scientific programmes may obtain access on favourable terms to UKMO data

    Does this mean only those supporting global warming can get the data?

  82. mpaul
    Posted Aug 12, 2009 at 10:26 AM | Permalink | Reply

    As a community, I think we tend to be too judgemental. People here are jumping to the conclusion that it was incompetance. I prefer to remain less judgemental and rather assume that many Climate Scientists are simply unfamilliar with many of the methods used in other branches of science. Climate science is a bit insular after all. In that spirit, I’ve begun work of a new article that I intend to submit to the Journal Science: ‘Novel Method of Climate Data Retension and Extraction Through the Use of “Filing Cabinets”‘.

  83. Clark
    Posted Aug 12, 2009 at 10:44 AM | Permalink | Reply

    Just think about what the lost raw data means to CRU. They can never modify their adjustments. Ever. If they determine at some point they want to use a different inhomogeneity calculation or adjust UHI differently, they simply can’t.

    Just wow.

  84. Andrew
    Posted Aug 12, 2009 at 10:51 AM | Permalink | Reply

    Clark,

    Of course they can modify their adjustments. Who’s going to stop them from making whatever adjustment they want?

    Andrew

  85. Eric
    Posted Aug 12, 2009 at 10:54 AM | Permalink | Reply

    OT (but interesting) response to #85 regarding NASA losing the original video of the original manned lunar landing. It was lost for 40yrs but they found it.

    http://www.neowin.net/news/main/09/07/16/nasa-to-release-high-resolution-film-of-first-moon-walk

  86. Steve McIntyre
    Posted Aug 12, 2009 at 11:28 AM | Permalink | Reply

    Pielke Jr comments here

  87. Ross McKitrick
    Posted Aug 12, 2009 at 11:39 AM | Permalink | Reply

    The response I received confirms no agreements are on file for France, Germany, Italy, Austria and the Ukraine.
    .
    I have often used the analogy of national Consumer Price Indexes to illustrate the ridiculous situation of the “Global Temperature” data. Each country has large professional staffs at their Stat agencies working on the monthly CPI using international protocols, using transparent methods, with independent academics looking over their shoulders weighing the various aggregation methodologies (e.g. Paasche, Laspeyres, Fisher, Tornqvist etc index number formulae), and with historical archiving rules that allow backward revisions periodically if needed. It’s by no means perfect, but it’s a far cry from the f**king gong show we’re seeing here. The reason CPI data, GDP data, etc. are handled professionally is that a range of policies (such as money supply control, pension indexing, intergovernmental transfers) etc depend on the numbers; also some labour contracts include CPI-based escalator clauses. In other words the numbers matter.
    .
    By contrast the Global Temperature numbers are coming from a bunch of disorganized academics chipping away at it periodically in their spare time. GISS numbers are handled (on Gavin’s admission) by a single half-time staffer, and the CRU says they’re stumped trying to find their original files back into the 70s and 80s, as well as the agreements under which they obtained the data and which to this day they invoke to prevent independent scrutiny.
    .
    I take Steve’s point that these issues are not sufficient to change one’s views on any particular policy which could be shown on other grounds to be desirable or otherwise. And this is not the thread (or the site) to debate policy. But bear in mind that the G8 Summit recently issued an undertaking to prevent the Global Temperature from going up by 2C compared to pre-industrial times, and the Copenhagen Agreement will probably contain similar language. That ‘2C’ number refers, in all likelihood, to a few digits that Phil Jones publishes in his spare time using data from files he has lost, etc. etc. If that kind of quality control is good enough for the purposes of G8 agreements and international protocols, then let’s not waste so much money running national statistical agencies to produce numbers that are used for such trivialities as central bank policy, national pensions and labour contracts. We’ll just ask some professors to cobble together a monthly number in their spare time, we’ll let them do historical revisions any time they want without notice or external scrutiny, and we’ll promise them that nobody will ever check their work.
    .
    And if the CRU staff want to plead that we are being too harsh on them because they “did the best they could” with such limited means and minuscule staffs, etc, fine. I’ll be the first to join in such a defence: the minute they put a big red disclaimer on their data products saying “Warning: These numbers are offered on an as-is basis and should not be used for any important public purposes.” But they can’t have it both ways, claiming their data products are good enough for policy purposes, while also claiming they cannot be expected to meet the quality expectations that apply to the other data series we use for policy purposes.

  88. PaddikJ
    Posted Aug 12, 2009 at 11:57 AM | Permalink | Reply

    OK, no arguments. I’ve only been an occasional lurker lately, so if this has been covered in more detail elsewhere, feel free ignore, but, whatever do you mean by

    it is not my position that this is an excuse for delaying climate policy

    As in to develop policy? or to implement policy (whatever it is)? or, Something Must Be Done!?

    (note that I did not ask, “Policy based on what?” No arguments here, not even by Socratic interrogation!)

    .

    I can almost pity the poor folk at CRU – they must feel like they’re under siege by a swarm of auditors (but then, they could have deferred on the super-duper computer until they had their house in order).

    .

    Even given the unbelievable sloppiness at CRU, I’m a little bothered by Steve’s harsh, heavy-handed sarcasm. The power of CA has always been its cooly analytical approach, leavened by the occasional zinger. I hope this thread is just a holiday.

  89. Posted Aug 12, 2009 at 12:05 PM | Permalink | Reply

    Steve’s battle is now mentioned in Nature Magazine

    http://www.nature.com/news/2009/090812/full/460787a.html

    • Steve McIntyre
      Posted Aug 12, 2009 at 12:18 PM | Permalink | Reply

      Re: Leif Svalgaard (#136),

      Thanks for the heads up. I was expecting something tomorrow and keeping it as a little surprise.

  90. Posted Aug 12, 2009 at 12:19 PM | Permalink | Reply

    I guess we can now add another debating point when discussing with the Other Side. It used to be “we have good non-proxy data from at least 1850″. This will now be, what, 1990 or so?

    re: #133 above

    Isn’t 2C the margin of error on tree ring proxies?

  91. Robinedwards
    Posted Aug 12, 2009 at 1:28 PM | Permalink | Reply

    In view of the wholesale “adjustments” made by those of the Hansen persuasion to data of various origins, though perhaps especially of data originating in the USA, I have to wonder what sort of effect the CRU manipulations, if any, or indeed misplacements, might have on our knowledge of large scale climate effects. It seems to be accepted quite widely that climatology is at best an approximate science in which opinion and methods of working can have a large influence on the outcome of a learned paper. It is easy to think of an example! However, I hazard that the loss/modification of original information on the scale that might have happened at CRU is unlikely to have such profound and global political effects as the methods adopted in MBH98 and ’99 have had.

    Perhaps the CRU data is still adequately wide-ranging and reliable enough for the general purpose of the goals of open-minded investigators to remain largely unaffected. If so we should be thankful for that. I recognise however, that this cannot be fully checked unless the original information surfaces somewhere.

    Nevertheless I fear that over-tenacious pursuit of some missing information may be counter-productive in the long run.

  92. John Silver
    Posted Aug 12, 2009 at 1:37 PM | Permalink | Reply

    Ha ha , they have been pulling your legs. This isn’t real, it’s an episode of “The Men from the Ministry”.
    You can listen to episodes on BBC 7:

    http://www.bbc.co.uk/programmes/b009smq5

  93. Posted Aug 12, 2009 at 1:37 PM | Permalink | Reply

    From my reading of this it seems like CRU have lost some, but not all, raw data and that some, but not all, of the raw data they have may be covered by a confidentiality agreement. It therefore seems sensible to request that they publish such raw data as exists and is not subject to a confidentiality agreement and to document such raw data that is known to be lost and to list the information about such data that exists but is under a confidentiality agreement.

    I believe I see an FOI request coming up

    • Posted Aug 12, 2009 at 1:54 PM | Permalink | Reply

      Re: FrancisT (#142),
      And here it is

      Dear Mr Palmer,

      I hereby make a EIR/FOI request in respect of the surviving raw data
      for the CRUTEM project.

      Based on the webpage http://www.cru.uea.ac.uk/cru/data/availability/
      and in particular the following statement

      > Since the 1980s, we have merged the data we have received into existing series or begun
      > new ones, so it is impossible to say if all stations within a particular country or if all of an
      > individual record should be freely available. Data storage availability in the 1980s meant
      > that we were not able to keep the multiple sources for some sites, only the station series
      > after adjustment for homogeneity issues. We, therefore, do not hold the original raw data
      > but only the value-added (i.e. quality controlled and homogenized) data.

      it seems like CRU have discarded some, but not all, raw data and that
      in addition some, but not all, of the raw data they have may be
      covered by a confidentiality agreement.

      I therefore request
      1) such raw data as exists and is not subject to a confidentiality agreement
      2) a list of stations (and dates if partial data exists) where such
      raw data is known to be lost
      and
      3) a list of stations and data providers (ie. national meterological
      organizations) for such data that exists but is under a
      confidentiality agreement.

      I am requesting this information for the purposes of academic research.

      Thank you for your attention.

      Yours truly,

  94. Dave Andrews
    Posted Aug 12, 2009 at 1:43 PM | Permalink | Reply

    Just wanted to say congratulations all who sent in their FOI requests. Do you think the University Admin that has to deal with this will be pleased with the way Phil Jones has behaved over the years? My guess is no and that they will demand greater rigour in the future. This will eventually be beneficial for climate science.

  95. Bill Curry
    Posted Aug 12, 2009 at 1:47 PM | Permalink | Reply

    Reporting In: I Received the standard notice (Ref. FOI_09-104) from David Palmer for QATAR, REUNION, ROMANIA, SAO-TOME-AND, TAIWAN.

  96. mpaul
    Posted Aug 12, 2009 at 2:51 PM | Permalink | Reply

    Had a private sector company said, “oh gee, we seem to have lost our ERP data, but we’re quite sure our earnings are properly stated”, there would be hell to pay. The government would impose huge fines and there would be calls for the execs to do jail time. But for some reason we hold the public sector to a lower standard.

    snip – policy

  97. Robert Wood
    Posted Aug 12, 2009 at 5:21 PM | Permalink | Reply

    Maybe HADley has LOSTly the original data, but they haven’t, surely, lost the algorithms for adjustment and normalisation, HADTHEYly?

  98. Charlie
    Posted Aug 13, 2009 at 12:52 PM | Permalink | Reply

    In addition to requesting info about confidentiality agreements, I had requested info about person or organizations to whom CRU had supplied data in the last 5 years. The response came back today. My questions in italics. CRU response in blockquote.
    .
    .
    1. Please supply a list of persons and/or organizations to whom the CRUTEM data set has been provided to in the last 5 years. If personal privacy concerns prevent you from identifying which specific persons have been provided such data, then it would be sufficient to simply supply the organizational affilliation of those persons. For example, it has come to my attention that most likely this data set has been provided to Peter Webster of the Georgia Institute of Technology in the USA. If you are unable to confirm that such data has been sent to Peter Webster, then it would meet my needs if you were to simply list Georgia Institute of Technology as the recipient.

    The entire CRUTEM data set has not been sent to anyone, or any organisation within the last 5 years. A subset of that data was provided to Georgia Tech University in January 2009.

    .
    .

    2. Please supply a copy of any agreements between your organization and the recipients or the institutional recipients of such data that impose any conditions of confidentiality, or which prohibit further transmission of such CRUTEM data sets, or which prohibit the public posting of such data sets on the internet.

    All written agreements that we possess in relation to any data received from any country or geographic area are now all available via the Climate Research Unit website at: http://www.cru.uea.ac.uk/cru/data/availability/. This page also has information regarding the compilations, processing and handling of any data received.  The manner in which the station data was collected, the changes in national boundaries and entities, and the nature of the processing of the data by the CRU have all evolved & changed over the time of collection of the data.  The agreements relate to the stations, which have stayed the same over the years, despite changes to national boundaries.

    .
    .

    3. Please supply a copy of any internal regulations or personnel regulations or rules regarding the supplying of such “raw” temperature data products to persons outside your organization; and any internal regulations, rules, personnel guidelines regarding the confidentiality of CRUTEM data records

    Whilst the University does possess internal guidelines regarding contracting generally, the making of research agreements regarding the conduct of funded research, and financial regulations governing any agreements with financial implications, there are no specific internal regulations or rules regarding the transfer of data to persons outside our organisation, nor any in relation specifically to the confidentiality of CRUTEM data records.

    He apparently misunderstood my question #2, which was for any agreements between CRU and the recipients (such as Peter Webster) of data from CRU. He answered with info about agreements between CRU and the raw station data providers. I will request clarification.

    I poorly phrased my question #1, as it could easily be construed to inquire only about persons/organizations to whom the entire dataset has been supplied. In any future FOI or EIR requests like this I will of course include, .. or portion thereof …..

  99. Matt Brook
    Posted Aug 13, 2009 at 8:50 PM | Permalink | Reply

    I’ve been considering holidaying in Kent, as I note from my guidebook that there is plentiful good wine to be had, on account of its numerous vineyards. However, I am also concerned that it is apparently also a reknowned malarious region, with many locals stricken with the ‘ague’. Of course my guidebook from the 1700s may be a little out of date, and the clemency of the weather may have shifted in the meantime. I would be grateful of some more recent information.

  100. Stan
    Posted Aug 14, 2009 at 10:09 AM | Permalink | Reply

    The dog ate their homework again.

  101. Status unknown
    Posted Aug 14, 2009 at 12:08 PM | Permalink | Reply

    Microfiche or punch cards were used back in the60’s, 70′ and 80’s for data storage. More than likely the data is stored on microfiche. Why throw out data that someone was paid to collect?

  102. John Norris
    Posted Aug 14, 2009 at 7:56 PM | Permalink | Reply

    re my request:

    I hereby make a EIR/FOI request in respect to any confidentiality agreements restricting transmission of CRUTEM data to non-academics involving the following countries: BELIZE, BENIN, BERMUDA, BOLIVIA, BURKINA FASO

    I received:

    Pursuant to your rights under section 1(1)(a) of Freedom of Information Act 2000 to be informed whether information is held, this letter is to formally advise you that we do not hold the requested information.

  103. Derek Walton
    Posted Aug 17, 2009 at 1:35 AM | Permalink | Reply

    Hi All

    Sorry for the delay: I have been on holiday.

    I received the standard reply from CRU for Zimbabwe, Zaire, Angola, South Africa and Botswana.

    Derek

  104. Willis Eschenbach
    Posted Aug 19, 2009 at 4:40 PM | Permalink | Reply

    Norway sez:

    Dear Dr. Eschenbach,
    It’s a pleasure for me to inform you that all digitised Norwegian climate
    data now are available free of charge at:

    http://eklima.met.no

    Yours
    Eirik J. Førland
    Director, Climate Department,
    Norwegian Meteorological Institute

    Dang, instant PhD, that’s cool … I wrote back:

    Mr. Førland

    Thank you very kindly for this information, it is very helpful. However, I had asked for something different.

    I am interested in acquiring the original raw temperature station data held by CRU, and used by them to create the HadCRUT dataset. This is very likely different from the current digitized dataset you reference below.

    The CRU say that they cannot release the data which they received from you because of a supposed confidentiality agreement with Norway. What I would like is for you to verify that they can in fact release such station data.

    Many thanks for your assistance in all of this,

    Willis Eschenbach

    Onwards, ever onwards …

    w.

  105. Posted Aug 20, 2009 at 4:13 AM | Permalink | Reply

    I have now had a reply to my FoI request to the Met Office. I asked for copies of the agreement with Phil Jones which prevents further distribution of CRU data. The reply is as follows:

    The Met Office does not hold this information because no written agreement exists. The Met Office received data from Professor Jones on the proviso that it would not be released to any other source. This proviso is based upon a verbal arrangement and a practice which has been adhered to since the early 1980s. We understand from Professor Jones that such an arrangement is necessary because the various organisations contributing to the work of the Climatic Research Unit have imposed restrictions upon the use and disclosure of their specific data. Please see the Climatic Research Unit’s website at http://www.cru.uea.ac.uk/cru/data/availability/ for further information in this regard.
    We are corresponding with the Climatic Research Unit to determine whether it can make further data available subject to obtaining all necessary permissions from the rights holders.

    • Bob Koss
      Posted Aug 20, 2009 at 7:50 AM | Permalink | Reply

      Re: Bishop Hill (#162),
      I see in their response they used “verbal arrangement” rather than “verbal agreement”. Is there a legal distinction?

  106. Willis Eschenbach
    Posted Aug 20, 2009 at 2:53 PM | Permalink | Reply

    Man, I love the Nørwegese …

    One of the reasons that we’ve given free access to our climate data archive is to avoid use of old versions. Most probably there will be no significant differences between the dataset we sent to CRU and the dataset available online at eKlima. However, as the Norwegian Met. Inst. has a very free data policy, we will not maintain the restrictions we put on the dataset that was sent to CRU in 1993. But as mentioned before, we prefer that Norwegian climate data is collected from: http://eklima.met.no

    Yours
    Eirik J. Førland,
    Director, Climate Department
    Norwegian Meteorological Institute,

    I replied:

    Eirik, thank you very kindly for the quick reply. I will forward your information to the relevant parties at CRU.

    I wish that all of the world’s meteorological groups, in particular CRU, were as open and responsive as your organization. You have my profound admiration.

    w.

    So, we can cross Norway off the list of countries with agreements with the CRU … but that was the only (unsigned) agreement that they’ve been able to dig up.

    The beat goes on …

    w.

14 Trackbacks

  1. By Games Without Frontiers « the Air Vent on Aug 11, 2009 at 1:59 PM

    [...] CRU Responds [...]

  2. [...] More from Steve McIntyre, at Climate Audit. [...]

  3. [...] you want to read more on McIntyre and his correspondance with the CRU click here. I highly suggest you do as it contains many more excuses on how the data was lost. Including such [...]

  4. By SteveM Got a Little Press « the Air Vent on Aug 21, 2009 at 10:45 AM

    [...] did, with Ross McKitrick, in deconstructing and undermining the famous “hockey stick” graph), writes on his Climate Audit blog, it appears that the impoverished CRU even lacked filing cabinets in which to store its [...]

  5. [...] did, with Ross McKitrick, in deconstructing and undermining the famous “hockey stick” graph), writes on his Climate Audit blog, it appears that the impoverished CRU even lacked filing cabinets in which to store its [...]

  6. [...] did, with Ross McKitrick, in deconstructing and undermining the famous “hockey stick” graph), writes on his Climate Audit blog, it appears that the impoverished CRU even lacked filing cabinets in which to store its [...]

  7. By Typeboard on Aug 31, 2009 at 1:29 AM

    [...] statistician and blogger Steve McIntyre, who has been asking for the data set for years, says he isn’t impressed by the excuses. McIntyre obtained raw data when it was accidentally left on an FTP server last [...]

  8. [...] than send to anyone.) ja että kun Jonesin olisi lopulta pitänyt luovuttaa alkuperäinen data, se olikin yllättäen kadonnut lopullisesti. Mutta kirjoitin nopeasti enkä muistanut. No, lukekoot [...]

  9. [...] statistician and blogger Steve McIntyre, who has been asking for the data set for years, says he isn’t impressed by the excuses. McIntyre obtained raw data when it was accidentally left on an FTP server last [...]

  10. By The End of CRUTEM? « Climate Audit on Jan 30, 2010 at 4:24 PM

    [...] submitted an even more detailed comment in the same thread here: I have often used the analogy of national Consumer Price Indexes to illustrate the ridiculous [...]

  11. [...] of the global temperature datasets, one of the great scandals of climate science. As described in this comment by Ross McKitrick, posted at Climate Audit on 12 August 2009. I have often used the analogy of [...]

  12. [...] The “small document” is online at CRU here. It was reported at CA on August 11, 2009 here, here and here (among [...]

  13. [...] in late July was published on August 11 here, prompting an immediate response at Climate Audit here. CRU’s webpage did not include ANY confidentiality agreements concerning station data. [...]

  14. [...] in late July was published on August 11 here, prompting an immediate response at Climate Audit here. CRU’s webpage did not include ANY confidentiality agreements concerning station data. [...]

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 3,183 other followers

%d bloggers like this: