CRU has posted up an undated webpage on data availability here , responding to the various recent FOI requests for station data and confidentiality agreements. Here they “list the agreements that we still hold”.
I’m preparing a post on this extraordinary document and am posting this thread as a placeholder for now.
Instalment 1 (Aug 11 8 pm) :
Obviously this is a pretty pathetic combination of excuses and whining. Both CRU and the Met Office should be cringing with embarrassment. Obviously there will be more shoes to drop. But let me reiterate one of my own baseline positions (and one which I do not wish to argue about with readers.) Regardless of how pitiful CRU’s management of data and contracts turns out to be, it is not my position that this is an excuse for delaying climate policy until the original data is found and documented. Neither do I think that any exigencies of the big picture excuse negligence in the small picture.
Lost Data
Surely the most surprising revelation is their confession that they’ve lost all their original data – all they have is their “value added version”. They say:
Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.
I was around in the 1980s and the 1970s. People used filing cabinets back then. You’d have alphabetically arranged files by customer. If you got paper from Andorra or Zambia, you’d put the data in the Andorra or Zambia file. If a decision were made on the handling of an account, you’d put a memo in the file. How is it possible that they don’t have ANY documentation on the construction of their data? This is getting worse and worse.
CRU Excuses
Imagine [you fill in the name] saying something like this:
Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s.
Nobody would take it seriously. Nobody would believe that they were that incompetent. I wonder what would happen if Lonnie Thompson moved offices. Would he lose all his unarchived ice core data?
Or this excuse as to why they can’t get data that any one of us can locate on the internet:
Much climate data are now additionally available through the internet from NMSs, but these are often difficult to use as data series often refer to national numbering systems, which must be related back to WMO Station Identifiers.
Poor babies. Imagine having to do a concordance of CRU numbers to national numbers to enable downloading from NMSs. That would be so boring. After all, they’re climate scientists. Things to do, people to see. When’s the next IPCC authors’ workshop?
But their trials and tribulations get even worse. They report that:
a number of NMSs make homogenized data … available in delayed mode over the internet. Some that provide both raw and homogenized versions, generally do not link the two sets of data together.
Y’mean, that someone somewhere would actually have to inquire as to how to do the links. Hey, Phil, I’ve got an idea. If they won’t tell you, send them an FOI. Or better yet, we’ll save you some trouble. Make a list of all the NMSs that are troubling you and we’ll send FOIs for you. Just have your people contact our people.
And as to why they haven’t documented the source of their data. It’s not their fault – that’s impossible. The reason why they didn’t document anything is that they “never had sufficient resources”.
We are not in a position to supply data for a particular country not covered by the example agreements referred to earlier, as we have never had sufficient resources to keep track of the exact source of each individual monthly value.
Climate Audit 
160 Comments
I have had a reply to my FoI request to the Met Office for a copy of the agreement with Phil Jones which prevents the Met Office disclosing the temperature data that Jones has passed on to them.
Two months to locate a single agreement? I’m speechless.
Re: Bishop Hill (#93),
It might be in a huge government warehouse somewhere, inside of a wooden box just next to the one which holds the Ark of the Covenant. Possibly they are afraid to open the lid early and thus risk having the demons fly out both ahead of schedule and without prior written authorization.
Re: Bishop Hill (#93),
Wow, even if they use offsite data storage facilties this is long time.
Re: Bishop Hill (#1),
Bishop, based on their request for another 30 days to answer your request, they had no direct knowledge of whether there was an actual confidentiality agreement or not, when they used this as an excuse.
We’ve run into problems with the Met Office just making thing up before – John Mitchell’s supposed destruction of his emails; John Mitchell supposedly acting in a “personal capacity” as IPCC Review Editor.
There doesn’t seem to be any downside for them if they mis-speak.
Re: Steve McIntyre (#24), I bet David Palmer is cursing MET right now. He can’t be having fun with this.
Re: Ryan O (#25),
I doubt it. Let’s suppose that they can’t find a confidentiality agreement – probably because there never was one. They will blame us for asking – rather than looking into the mirror and challenging whoever said that there was a confidentiality agreement.
Re: Steve McIntyre (#28), Ah . . . probably true. Perhaps I should have said if I were David Palmer, I’d be cursing MET.
Re: Ryan O (#29),
It looks like Palmer held CRU’s feet to the flame on the “confidentiality agreements” and wouldn’t protect them any further.
Here is the FOI request I sent, and the reply:
The request I sent:
The text of the reply recieved, today:
CRU has responded to one of my 3 requests.
It now has a webpage that lists all of the confidentiality agreements they hold. Reading through that page, it appears that the only confidentiality agreement they currently hold is for BAHRAIN.
CRU Data Availability basically says “No, you’re not going to get raw data, and in some cases we have destroyed it”.
A few key extracts:
Re: Charlie (#170),
Since the Bahrain agreement is with the airport authority/manager, my guess would be that the Bahrain data is solely from the airport there, and thus perhaps freely available from other sources.
The text of CRU’s response to my request was similar to Comment #169, above, but had the following text at the bottom:
Re: Charlie (#171), received my reply today. Same as the reply you received.
Re: Charlie (#171), Same reply as Charlie and Bob Koss.
Hmmm.
He starts out saying “all written agreements”. Should I infer there remain undisclosed verbal agreements they would feel compelled to honor?
Re: Bob Koss (#174),
Yes, on the webpage referenced in the FOIA response, CRU says
I got my reply today too.
So, who holds the details of the relevant agreements with Egypt, Greece, Cuba, and Greenland.
Concerned polar bears want to know.
The way I read this is that they no longer have the raw data. It was the raw data that had the “confidentiality agreement” against it, not the “value added” data. Can this “value added data” have the same confidentiality agreement against it? Does the agreement state that the raw is not to be released, or can it include the post-processing of the data?
If processed data can be lumped in with the agreement, how could ANY scientist release anything about this data without violating the agreement?
Also, if they consider the value-added data they hold to be “quality controlled”, how can they check the quality against data they no longer have?
I got my answer today as well. This is absolutely priceless. Rather than keep the raw data, from which the “quality controlled and homogenized” data could be produced at a moment’s notice, they have thrown away the raw data.
.
The fact that these good folks have the balls to style themselves as “scientists” defies belief.
.
A few other oddities:
.
1) Some of the data which can’t be released is from overseas British territories such as Gibraltar, Stanley Island, and the like …
.
2) Other than the British agreement, only three other countries are on the list, Norway, Bahrain and Spain.
.
3) The Spanish agreement says nothing about passing the data on to third parties. It simply says:
.
.
This means “The person signing below swears to use the requested information for purposes specified in this request, and to cite the origin of the data in whatever publication or work in which it is used”. Nothing about third parties at all. The purpose specified is “Public”, as opposed to “Private”.
.
However, they go on to say:
.
.
Now, I don’t think that “we might have made a confidentiality agreement sometime with somebody from somewhere, but we don’t know who or when” is adequate to shield data from an FOI request … but then, I’m not an FOI lawyer. I’d welcome comments on this. If the FOI allows this as a response, it is useless. Any lawyers want to chime in on this?
.
In any case, the fun goes on. I’ve just sent this response:
.
Gotta love these guys, “science” at its finest …
.
w.
I also had a request into CRU for the agreeements with national met offices which prevent disclosure of land station data supplied to CRU. My request preceded the group effort here by a couple of weeks. This request has now been refused as follows:
The webpage cited is new to me and I’m not sure if the information there is new or not. I’ve been offline today so I haven’t had a chance to look at anything there but there is a page called agreements which I assume is relevant.
OK, I’ve taken a look at the agreements document. It looks to me as if non-disclosure terms have been put in place by
a)Denmark [Steve – nope, this is Norway,]
b)Bahrain
There are also documents filed for:
c) the UK Met Office – there is a document from Mike Hulme requesting British overseas territory data, in which he undertakes not to pass info on to third parties. There is also another document which mprobably a red herring. It looks to be a printout of a website page rather than a formal agreement. It does say that UK Met Office data is commercially valuable and should only be used under the terms agreed.
d) one non-headed document, which appears to be Spain. My spanish is poor but I can see nothing that looks like a non-disclosure clause although there is something which seems to say data should only be used for the agreed purposes which may amount to the same thing. Strangely though I can see nothing where the purposes are defined.
And now I see you’ve been discussing it on a different thread for two hours!
Re: the agreement with Bahrain.
I would think that a true confidentiality agreement would need the signature of the receiver, not the sender. In other words, the person who is to keep the confidence…
Based on the wording of the UKMO memo, it seems clear to me that the disclosure agreements do not prevent the individual NERC Data Centres (of which CRU is one) from releasing the information for scientific/academic purposes. Paragraph 1 states that the NERCs have the authority to release the information to specific individuals for a pre-defined purpose. It is also clear that the computer code used to convert the raw data into a temperature index is not covered by whatever agreements exist with other countries/institutions. As they provided this memo in defense of the non-release of information, I have submitted an additional FOI request for the data and, in keeping with the requirements listed in the UKMO memo, I have stated the scope of the project.
.
The request is for:
.
1. All original station temperature data used to produce the CRU gridded land temperature data set.
.
2. The code used to convert the original station temperature data series into the gridded land temperature set, to include all homogenization adjustments, adjustments to remove non-climatic influences, and any other adjustment to the original station time series. Text format of the code in the native language is acceptable.
.
The scope of the project is:
.
1. Investigation of the effects of changing land use, urban heat island effects, and other non-climatic influences;
.
2. Sensitivity analysis of parameters used to grid and adjust data;
.
3. Sensitivity analysis of methods to calculate monthly means (e.g., min/max, max only, min only);
.
4. Analysis of the source of differences between the CRU land temperature index and other indices (e.g., GISTEMP and NOAA); and,
.
5. Analysis of the effect of using satellite-derived covariance information to interpolate between land stations.
.
#5 is especially because Phil Jones issued the “incorporate satellite data” challenge, and if the data can be released for that purpose, then it should be able to be released to other parties attempting to accomplish the same thing.
.
My read of the information presented at the data availability link is that this particular FOI request fulfills the requirements to release the data (though I would not be permitted to distribute it further). To deny this request, they would have to find something more substantial than what they’ve provided to this point.
Re: Ryan O (#19),
You may still not qualify. From the UKMO memo:
I don’t think blog posts will meet their requirements, so be aware they expect you to sincerely plan to write and publish a research level article using the data in order to qualify for data access.
Re: bluegrue (#42), I sincerely expect to publish a research level article. 😉
The irony is strong with this one.
Steve: Yup. CRU climate scientists have never worried too much about logic.
Re: yodajeez (#21),
Let’s see: they provide 4 documents from 1993-94 pertaining to CRU and a copy of the present webpage setting out the form of agreements between the Met Office and NERC.
The statement above refers to CRU “requests for data from NMSs”. There are only two request documents, one of which is to the Met Office and not to a NMS. The only document demonstrating the form of their requests is the application to Spain, which asks only that Spain be cited.
The other two documents are replies from NMSs of Norway and Babrain. But these do not show the language of the original request.
Just to keep the list up-to-date with regard to responses from David Palmer, today I recieved the following respones to my FOIA request:
(original request):
(Response):
As has been mentioned, the webpage referred to does not provide any confidentiality agreements for the countries I requested. To tell Steve that they can’t provide the data because of pre-existing confidentiality agreements, but then to be incapable of providing them is surely indefensible?
I await Steves post with eager anticipation – looks like we have them by the short and curlies!
I have just sent the following email to David Palmer:
Wow! All these excuses! Why didn’t Dr. Phil explain these “problems” when he was originally asked for the data?
I requested ARUBA, ANTIGUA AND BARBUDA, AFGHANISTAN, ALGERIA, ASCENSION ISLAND.
Its a bit odd that they would phrase it as an affirmative fact: “we do not hold the information” as opposed to saying something like, “we’ve searched our records and can’t find documents that meet your request”. One can only concluded that they know for a fact that they do not have NDAs with these jurisdictions.
Re: mpaul (#30),
I received the same response today – the five I requested were Greenland, Iceland, Ireland, New Zealand, and Thailand.
The letter from the Norwegian Meteorological Institute is surely out of date. The institute is in the lead in Europe when it comes to make data available for free: “From September 2007 weather data has been made available for everyone for free. See more information about our free data policy (in Norwegian only).”
Here you can find the Klimadata Web Services and information in english.
Re: a.s. (#31),
That’s particularly useful in that Norway is almost the only example of a “confidentiality agreement” that CRU could come up.
Oh, what a web we weave, when we seek to retrieve.
BTW I haven’t got an answer to my request yet.
Re: Steve McIntyre (#35),
Me neither…
I got the usual reply for the following countries:
ACORES, ALBANIA, ANDAMAN AND LA, ANTARCTICA, ANTILLES.
For anyone wishing to press CRU please remember station data are obviously “environmental information” exempt from disclosure under the FOIA as is any information about administrative measures, like confidentiality agreements, in connection with it. You should immediately request an Environmental Information Regulations 2004 r.11 reconsideration if you are not satisfied with what CRU say. This has a 40 working day time limit whereas FOIA has none.
EIR has a presumption of disclosure and no absolute exception for confidentiality agreements. Even if disclosure could be shown to be actionable, which is vanishingly unlikely, a public interest test must be made having regard to the public interest and the ICO is unlikely to maintain nondisclosure. I think a Court would give CRU short shrift if they press this issue. Unfortunately as Jones and Mann have shown it is possible to spin this process out for years.
I also received a reply last night, similar to many described above, saying “…we do not hold the requested information.” For the record, my request read as follows:
And for the record, here is the reply:
Data from Spain (Tmax, Tmin, Tmean, Precipitation, etc) for the station network is freely available for the period 1971-2000. I could not find available in the web page the 1961-1990 data requested by CRU to the spanish MET.Office. I do not think the data can be subject to any confidentiality agreement other than not to be used for commercial purposes.
See for example data from various stations in Madrid area here (1971-2000)
best
To repeat: We, therefore, do not hold the original data but only the value-added (i.e. quality controlled and homogenized) data.
In other words, while the dog ate some of the contractual agreements with the sources for the data, more importantly and embarrassedly the dog also devoured the original data. Given that information is it not totally futile to ever expect to see the original data at least from CRU? Is it not also futile to ever expect CRU to go back to the original data and expect them to attempt to improve on any of their adjustment techniques?
One could laboriously attempt a CRU reconstruction of original data from GISS data or going back to the data originators as Steve M has attempted. Meanwhile, as a scientist, one could accept CRU on good faith or say, to hell with your sloppiness, and move onto to other data sources where, at least, the originals are available.
Re: Kenneth Fritsch (#44), I don’t buy it. They have the original data. It might not all be in one, compact package like the value-added data, but they physically possess it. It will take much more than this statement to convince me otherwise.
Re: Ryan O (#45),
Admitting that you have lost your original data for a temperature data base with the prestige of CRU has to be damn near fatal for future use by climate scientists – assumming those scientists care about the apparent sloppiness revealed and what their use of sloppy data would imply about their own work.
My question would be why make such a damaging admission if it were not true. Maybe after years of being without the original data they finally had a Bernie Madoff moment.
Re: Kenneth Fritsch (#44),
I would like to know whether the “value” can be subtracted from the “value added” data. If the adjustments are systematic, they might be reversible, meaning the original RAW might to a degree be approximated. That would be if the original methods, constants and rationales were properly published or available.
Re: John (#96),
One would have to assume that the corrected/adjusted data is “correct” and would thus leave no traces of its former unique problems. Of course, if a record was kept of the exact correction made on the original data and the record of the original data were then lost than your suggestion would work – and no doubt applied by CRU and thereby avoiding their fatal admission.
Not sure Ryan. Compare:
re: confidentiality agreements and:
re: data.
Whilst I agree the actual no. of bytes of info. would have been relatively small; how were they receiving the data in those days? File transfer? Physical media? Written copies for data entry? Transcription by phone? What machine would they have used to perform the “value add” operations? If this were an individual’s PC it is quite possible the raw data never got onto a mainframe for a run and maybe never found its way into an archive and back up system. If they only had physical originals it would not be a big surprise if these went the same way as the confidentiality agreements? More of the dog’s missing dinner perhaps?
Re: curious (#46), Perhaps, but the qualifier they add “not able to keep the multiple sources for some sites” would imply that for others, they did keep them. In any case, this would not prevent them from providing whatever original data they actually possess. Additionally, the advancement of the explanation that data storage availability in the 1980s is the reason means this condition can only apply to a very small subset of the data. And if they actually also lost the handwritten data sheets (along with all of the supposed confidence agreements, then they are truly more incompetent than I had imagined.
Re: curious (#46),
Why don’t you re-read my chronology of CRU versions.
Anything from the 1980s was delivered up to DOE and is available at CDIAC. So even if they made agreements in the 1980s that they can’t find, they broke the agreements and published the data, and can’t now selectively rely on their broken agreements.
Re: Steve McIntyre (#51),
Is this why NOAA has stated that it doesn’t consider the HadCru data to be of value/signifance? 🙂
When do you think it would be appropriate to “out” this sorry lot in a “peer reviewed” publication?
snip – please do not over-editorialize.
Yes, but if “not able to keep…” translates to “we only lost some…” (imagine 1 box of 5 1/4 in. floppies going astray in a move) then they might prefer to just dance around it rather than come clean. Obviously this would go against all previous indicators…. 🙂
Re: curious (#48), My take on this is that they do not have all of the original data (but they have the vast, vast majority of it) and are using semantics to avoid handing out anything.
.
I find this to be the most plausible explanation, as:
.
1. They clearly released data to Peter Webster of Georgia Tech,
.
2. Jones recently published an article finding evidence of UHI at Chinese sites (which requires the original data),
.
3. Steve’s June FOI request was denied with no mention of missing data,
.
4. Jones clearly had the vast majority or all of the underlying data archived as of September, 2002, and was intending to put it on line,
.
5. The MET Office has clearly indicated in several past correspondences that CRU possesses the underlying data, including a written statement from John Kennedy that CRU and Phil Jones were, indeed, in possession of the underlying data,
.
6. The MET Office was apparently so convinced that Jones had the raw data that they even provided a FAQ describing how to request this data from Jones.
.
In short, the statement that CRU (not the MET Office) does not have the original data is a recent development that has appeared out of left field. Therefore, I question the veracity of their claim.
Why doesn’t CRU simply send a notice to any questionable government saying something like:
“Given the general interest in data related to the AGW theory, we intend to release all such data we presently hold. If you have anything in your records regarding a confidentiality agreement between us and you, please forward a copy to us so that we can determine it’s applicability under present circumstances. If we have not received such a document by 1 Oct 2009 we will release the data to public view.”
Once that was done and results obtained, it would be clear the scope of privacy agreements, if any. An entity which didn’t wish to maintain such an agreement could simply not reply while those who came late to the party might or might not have a chance to have their data from public view.
Re: Dave Dardinger (#49),
Doubtless, at one point, they might have said that it was “too much work” to write to the NMSs in question and seek permission. But they’ve shown that they can handle 50 or so FOI requests. So it shouldn’t be that hard for them to send 50 emails to 50 NMSs. Or 200 NMSs.
Re: Steve McIntyre (#50),
In fact I would suggest that are required to. In Annex G of their procedures there are guidelines for contracting with third parties and they have requirements to advise third parties of the potential conflict between confidentiality and FOIA. If they continue to receive data from a third party and have no records of the contractual basis of that transfer then it would appear they have an obligation to clarify matters.
I find the simple direct approach works often enough that it is worth trying.
I just left a message at the online contact form at the Bahrain CAA website.
http://www.caa.gov.bh/contactus.php
Re: Charlie (#53), following on your excellent template, I have sent the following to the Norway DNMI:
Since Bahrain and Norway were the only NMS’s with even a pseudo-agreement, that covers all their bases.
.
Onwards, ever onwards,
.
w.
Re: Willis Eschenbach (#92),
probably want to advise them that CRU have likely violated the agreement ( assuming one existed )by posting the data.
Re: Willis Eschenbach (#92),
The public journal at DNMI seems to contain no reference to your request yet.
Re: Thor (#158), Thanks for following this. You say:
Dunno about that part, I dont read Norwegian. The reply I received, dated the 11th of August, was:
Now, it’s been more than five work days, so I’m expecting a reply very soon. If not, I’ll write back to them.
w.
A key argument made by the CRU is that agreements not to further pass on the data to 3rd parties is part of the normal, accepted way of doing business in the climate science / national met office world, and that even though the CRU has very few copies of such agreements, that virtually all data has been acquired under such understanding, either implicit or explicit.
This of course, ignores the long standing scientific tradition of making published work transparent and reproducible.
I at least two ways around this problem:
1) individually getting each (potential) source of historical station data to waive any confidentiality or restrictions on the use of the data for scientific research, such as Steve McIntyre’s proposed replication of the HADCRU3 gridded data.
2) getting enough public and scientific community and political pressure applied to this issue that the assumptions of what is normal are changed to accomodate the use of the raw station data for scientific purposes such as SM has proposed. Seeking and acquiring such approval for scientific use by a few dozen countries would, in my opinion, go a long ways towards convincing the Met and CRU that they should produce what data they have not yet discarded.
Thoughts? Comments?
Steve: I haven’t proposed personally replicating HadCRU calculations. I don’t have the time or interest to carry out the examination of sites and metadata that would be required in the sort of study that I think needs to be done. Nor for that matter do I think that climate policy should be delayed while this sort of thing is done. But equally I don’t think that any exigencies of the big picture excuse negligence in the small picture.
Re: Charlie (#54),
Absolutely. I urge people to write to NMSs directly. It’s well worth reading WMO REsolution 40 mentioned by Jones.
Re: Steve McIntyre (#56),
.
I read that on your update and seriously laughed out loud. Awesomeness.
.
And we’d do it, too.
.
Hell, we’ll probably do it regardless . . . Charlie’s already boarded that train.
I’ve added some first comments in the thread. It’s hard to know where to begin – this is a very embarrassing document.
I have been reading this website for over a year now. It seems to me that the only court that the obfuscators must respect would be that of the law. How do we prosecute this? How do we shine a light into these dark corners? Are there any readers who share the desire for truth and objectivity who have been called to the bar in the UK and have the passion and resources to help?
Congrats Steve. These documents sound like a confession under duress. Wish someone would teach them how to use an html stylesheet – this plain vanilla html is so 1990s.
“Nor for that matter do I think that climate policy should be delayed while this sort of thing is done.”
What if it’s nonsensical climate policy? I’m not sure that this statement I quoted even makes any sense. You brought it (climate policy) up, BTW.
Andrew
Steve: I also asked readers not to discuss this position – we’ve done so in the past. Any “bad” policy is “bad” not because the dog ate CRU’s homework but for other reasons. If the only thing that stands against a policy is CRU mismanagement, then the policy is probably OK. And I don’t wish to debate this with you or discuss it on this thread.
Re: Andrew (#61), For the record, that wasn’t me.
They’ve had many years to work on obtaining these “rights”. I wonder what concrete steps they’ve taken over the years towards removing any perceived encumbrances? 🙂 Have they ever notified the WMO or IPCC of problems with encumbered data?
I wonder it there are any documents evidencing that they’ve expressed this hope to any third party in the past. 🙂
Eh, yes, I would. During my master’s degree in physics – already over 10 years ago – I worked for an IT company that worked for many mid-sized companies. I was amazed how poorly data archiving was. And on just about everything. From pay-checks, taxes, financials, their inventory, customers, etc. And if backups were made, they were rarely even checked on if the backups actually contained what was the intention.
We should realise, that untill the Hockystick debacle nobody cared or thought about data availability or checking. We are talking about 25+ years of nobody checking anyones data in climate science. People trusted eachother (and possible – except for honest errors – could as the topic wasn’t politics then). So it makes more than sense to me, the data was lost, or only pro-processed data is available.
Never assume wrong intent, when plain stupidity or even more common sloppyness can be blamed.
Much more worrying is actually that only pre-processed data is available. That means assumptions once made, are now final and beyond dispute. Most likely even the exact proprocessing is not available either …
Steve: Please don’t misinterpret my comment. In this case, I am definitely not imputing anything other than incompetence. The incompetence here apparently reaches beyond a lack of care with the data versions, but to the paper trail. As to your statement that prior to the Hockey Stick debacle, nobody “cared or thought” about data availability or checking. Maybe that was true in climate science, but it was definitely not true in the stock market. Such things are routine. I was dumbfounded at the attitudes in climate science and remain amazed.
Re: Armin (#63), The only problem with this position is that, until just now, they had the data. Excuses about limited storage capacity in the 1980s were not present in 2002, when according to Jones himself a complete set of the underlying data was being prepared for posting. So unless they have a time machine and whisked themselves back to the 80’s to pull an Enron on themselves, incompetence is not the answer.
Re: Ryan O (#64),
Ryan, I think that what Jones contemplated archiving (and did archive in Feb 2003) was the “value added” station data – not the “original” data.
Of course, the “confidentiality agreements” didn’t seem to present a problem. Funny how they only became a problem for Climate Audit readers and Warwick Hughes.
Re: Steve McIntyre (#67), Ah. I misunderstood. 😉
Re: Armin (#63),
Sheesh, this is the government we’re talking about. The stock market (theoretically) has watchdogs making them be disciplined. Who watches the watchers? For the government, no one but themselves usually. I ‘ve been associated my whole career with government agencies, including R&D facilities. I would have been surprised if they were able to give you what you asked for, not that they say they can’t.
Andrew,
No it wasn’t you, it was the “bad” Andrew who mismanages his comments. lol 😉
Andrew
What do you call a data product where at least part of the source material cannot be identified back to the source?
It’s not a trick question. I’d really like to know.
It is very odd that an organization in the climate monitoring business should lose its climate data during a move. It is understandable that something could be misplaced in an office move but the thing that is the core of what an organization does is generally not the thing that is misplaced. And something else seems “fishy”, too. Haven’t there been others who have claimed to have used their raw data since then? It is quite possible I am mistaken but I was under the impression that other academics had access to that data since then and only the ones who wanted to “find something wrong with it” had been denied.
They apparently didn’t lose their payroll records, or accounts receivable, or their Rolodexes. That an organization whose primary function is to provide analysis of such data should lose it puts them in a double-bind. They are either being deceptive or are unbelievably incompetent. There is no good way out of this for them.
This is absurd, incompetent, and unscientific. I’ve been involved in data management since the early ‘60s. For source data storage and retention, there was original hard copy, microfilm, initial punched cards, mag. tape and printouts of interim files. Failure to store, index, and assure recoverability of such data (and related legal restrictions) is an admission of incompetence. The assurance of data integrity is incumbent on every scientific endeavor. Without retrievable (and reperformable) base data, calculated conclusions are worthless.
CRU, with their several above responses, is declaring their findings/output to be scientifically unreliable.
If work cannot be replicated, how can it be considered science? I cannot get my mind around that one. I’m baffled that anyone takes work like CRU’s seriously. What joker came up with the expression “value added” to describe CRU’s data?
The fundamental purpose of this exercise must surely be to recover the earliest, least-adjusted records; and then to study the subsequent adjustments for logic, error, assumption etc.
Two years ago I was told that CRU no longer held data I sought. This raises an intrigue as to whether in a legalistic way, CRU has assigned guardianship to another body which needs to be identified then FOI’d. See Kenneth Fritsch at #44.
There are hints that a mass of data in early form exists. I tend to agree with Ryan O at #45 & #52 and Steve at #51.
I wrote to our NMS a week or more ago, asking if we had an agreement of the type mentioned, but have had no response so far. Also, I have posted the copyright statements that now accompany public release of Australian data. I have not succeeded in finding copyright statement versions operating in the early 1990s.
There are several graphs posted in recent weeks where Australian temp data are shown from various sources. So far, no systematics have emerged from the subsets I have examined, though it seems that the “most” original 1993 BOM set often plots higher than some others such as KNMI. It is not uncommon for a year to have a spread of 1 deg C from various current public sources. Presumably, Tmax and Tmin variation can be greater again.
The 1993 BOM set is already homogenised to a degree that I have not examined. It is a lengthy job, station metadata sheets one-by-one for 1,000 or more stations.
Having noted this, it is possible that the gaining of the CRU original data might be a hollow victory for future research, but a severe lesson for the conduct of climate science. The reason is that any “original” set that might be extracted from GB would have to be examined, piece by piece, to see what adjustments had already been made, if any, by the donating parties.
In an overall glum view, if serious science is to be done on the instrumental temperature records, it really needs to be restarted from the ground up, with original data from NMS or country sources including metadata.
We have learned a great deal (thank you Steve) about how the job should be done properly. In the meantime, I have refused to use CRU-Hadley data for any purpose other than to show inconsistency and uncertainty. I do not bother to read papers that use it as a base.
Unfortunately, given the Australian provenance of a good deal of SH information and its prior poorly-understood homogenisation, many other countries will be in a similar position. I cannot recommend any global surface instrumental data set as being adequate for correlation/causation studies in climate science.
I’m not surprised that they didn’t keep the original data back in the 1970’s and 80’s. In their view, I imagine the raw data had no value. The “good” data was that which had been processed. You have to remember just how expensive data storage was back then. All you have to do is look back to the Y2K problem to understand just how far programmers would go to minimize the data stored. Was it shortsighted to not keep the raw data? Absolutely! But seriously who could have predicted the current importance of temperature records from obscure location around the globe from 40 years ago???
Anyway, I don’t consider the raw data to be “lost” but rather a business decision was made that it was unimportant to retain it.
Re: Bill Jamison (#76),
The original data is the heart of the data base and without it how would or could CRU justify the value added part – or ever go back for a Version 2 as USHCN did. GISS rules.
Re: Bill Jamison (#76),
In the 70s and 80s, on line memory was expensive. Off line memory was very cheap. The data could have been placed on t 9-trackape and kept in storage, If someone wanted access to the data, they would send a message to the computer center and ask for the tape to be mounted. In my experience, this took only a few minutes.
There would be nothing unusual about this. Any university would have the facilities to do this. Pictures of computer centers of that era commonly have racks of 9-track tapes shown in them. One tape would have sufficient storage for all temperature data both raw and processed.
Re: TAG (#100),
ditto
Re: Bill Jamison (#76),
“You have to remember just how expensive data storage was back then. All you have to do is look back to the Y2K problem to understand just how far programmers would go to minimize the data stored.”
The problem wasn’t data storage, Bill, but rather the very limited amount of core memory available (RAM) for volatile processing. With a memory space today measured in gigabytes I would have no problem programming a huge, multi-demensional array and retaining and processing it in memory. As recently as the 80’s we were too often restricted to 32 – 64 Kilobytes of RAM: far too little to hold a data array and the instructions to process it. Intermediate structures had to be written back to “temporary” files and then read back in as needed. Even in those days the biggest time constraint to processing was I/O, and jobs running in seconds today took hours back then. Yeah, we economized on dates back then, but it was due to processing time and memory constraints rather than the cost of storage. There is no good excuse for losing scientific data.
* That implies both that they know that they lost data for some sites (which ones?) but not others (which ones?).
* Have they tried to contact the relevant agencies (when?) so they can complete their database for modern analysis?
* Why haven’t they been trying to reanalyze their old data, when GISS and NOAA seem to often reanalyze and adjust their data?
* What policy decisions, studies or research are the basis for their decision to not reanalyze data?
* What are their quality control procedures to ensure that their data is valid or useful?
** This Met Office page mentions ISO 9001 standards, but the page only refers to data they gather from UK locations.
* Have they already studied their data and decided that it is not valid or not useful, or not worth maintaining?
* How much to they spend to maintain this data?
* What are they doing with the data maintenance funds?
BAHRAIN:
There is NO confidentiality agreement with Bahrain, if the posted document is all they have. As James Goneaux [#18], above, noted, Jones and Hulme didn’t sign anything.
More than that, however, it’s clear Jones and Hulme asked for data and it was freely given by the Bahraini agent. The agent didn’t withhold production of the data until he received a written or oral promise of confidentiality.
Only after giving the data did he ASK,
‘Please’ does not create a condition of use.
SPAIN:
There is also NO confidentiality agreement with Spain.
It’s convenient that the portion of the agreement which was attached is now missing
As Willis notes, above [#13], at the location for date and signature [FECHA Y FIRMA], the boilerplate requires data provided to be used for purposes specified and to be cited in all publications. Has this been done ?
Please note that there is NO SIGNATURE at the signature line [FECHA Y FIRMA]. No signature generally = no agreement.
Willis kinda gets it wrong about “public purpose,” however. Hulme describes himself as a member of the public [Empresa: Publica].
The directions for filling out the form tell the applicant to “Briefly describe the use for which you are requesting this data.”
In the portion of the form set aside for Hulme to describe his planned uses, for which he might, hypothetically, be limited to, he states:
So, we have in black and white that Hulme merely stated he wanted the data; he never stated that he intended to use it for any particular purpose. The Spanish gave him the data without any restrictions.
Even IF he had signed the request, there would be nothing to bind him to. He’s free to use it however he wishes.
NORWAY:
There may be SOME confidentiality requirements tied to Norwegian data, but given the age of the data – I doubt it. Bjorne Aune provided both “a report and a disk” to Hulme in 1993. Confidentiality requirements were only attached to the then-unpublished data found on the disk.
DNMI wanted to protect gathered, but unpublished data back in 1993. It is arguably reasonable for a Norwegian public institution to control the use of their data for a short period of time, until publication. The purpose to which DNMI attached confidentiality requirements has long passed. It’s been sixteen years. They’ve had their period of exclusive use to feather their own curricula vitae. They don’t care about confidentiality, anymore.
Stronger still, there’s no signature by Hulme – agreeing to the proposed condition. He likely isn’t bound to the terms of the agreement.
ORAL AGREEMENTS:
There are NO confidentiality agreements based upon oral promises.
One could always raise the question of authority to enter into these agreements, but before getting there, anybody who attempted to enforce one of these agreements would have to deal with the Statute of Frauds , which generally requires certain agreements to be in writing [and signed by the bound party] in order to be enforceable.
Among the agreements covered by the Statute of Frauds are agreements which, by their nature, cannot be completed within a year. An agreement to be a sole supplier or retailer of goods for sixteen years cannot be completed within a year, so it is covered by the Statute of Frauds . The agreement must be in writing or it’s unenforceable.
This will likely be true for confidentiality agreements in the UK. Short ones [11 months] may be oral, long ones [> 1 year] must be written and signed by the party bound [Jones / Hulme].
Any oral agreements were legally void or have long-since expired.
Re: jim edwards (#79),
Jim, note that there’s a bit of a disconnect between the paraphrase of the regulation and the regulation itself.
Palmer’s refusal letter used the terminology:
12(5)(F) doesn’t actually use the term “confidentiality agreement” although it is related. Here’s what 12(5)(f) says:
Interpretation of the exemptions is at http://www.defra.gov.uk/corporate/opengov/eir/guidance/full-guidance/pdf/guidance-7.pdf.
Is the phrase “did not supply it in circumstances such that or any other public authority is entitled apart from these Regulations to disclose it” equivalent to being governed by a confidentiality agreement?
I wonder if there are any cases on this clause.
Re: Steve McIntyre (#87),
Did you notice the case that was linked in footnote #13, at the bottom of page 18 of the the guidelines for interpretation you linked to ?
[* I can’t get the link to work, but the BNRR case is at http://www.birminghamfoe.org.uk/bnrr/judge1.htm%5D
The case involves “environmentally information” provided to the government within the terms of a confidential contract to build and operate a toll road, pursuant to a publicly-granted monopoly.
A couple of environmentalists decided they wanted to know the terms under which this preferential deal was made and requested data, then litigated when it was denied.
The court went through a number of exceptions, focusing most of its efforts on confidentiality of clearly commercially valuable data and practices. [i.e. – not really relevant to this discussion]
At the very end of the case, the court looked at the analogue of 12(5)(f), but in a very confusing way.
The old numbering for EIR 12(5)(f) was, apparently, 4(3)(c).
Consent:
So, assuming that 12(5)(f) applies to the information you’re looking for, a court might look at the inherent likeliness that a private party would withhold consent. Consent seems less likely withheld for 30 year-old weather data than for private accounting statements.
Consideration:
I think another argument can be made that information was provided for consideration, or likely provided for consideration, or pursuant to a pre-existing agreement, or that it cannot be shown that information wasn’t provided for consideration or pursuant to an agreement. Any of these circumstances would cause a legal obligation to arise, and 12(5)(f)(i) would not be met. No exception would arise.
Consideration could include money, as the Norwegians suggest.
Consideration could also include a promise to provide a future benefit, like being named as a coauthor on a paper published in a prestigious journal.
A promise to use one’s data and credit its provenance might also be consideration, where it’s reasonable to expect that being named as an important data source might bring prestige and continued third-party funding.
Promises wouldn’t necessarily have to be reciprocal. It’s possible for one party to fully perform [i.e. – deliver the information] in expectation that Jones or Hulme would fulfill promises in the future.
Ownership:
If I steal/convert information, I don’t magically acquire rights to direct how it may be used. Unlike in the BNRR case, I think there may be genuine issues of ownership present. The regulation doesn’t explicitly state so, but a “person” under 12(5)(f) must be an owner of the information, in order to withhold personal consent.
Assuming that there are agreements to keep publicly-funded information confidential, can Jones show that the parties who gave him the data had the legal right to withhold consent to distribute the information ?
Public Authorities and International Law:
An interesting question to me is whether foreign “public authorities” remain “public authorities” or become “persons” [for the purposes of 12(5)(f)] when they give information to UK “public authorities” under EIR.
At first blush, the interpretational guidelines basically define only UK institutions [minus the Scottish ones… they have their own EIR] as “public institutions.”
But the EIR seems to be a departure from normal UK law, like FOIA. The interpretational guidelines note that the purpose of EIR is to implement the Aarhus treaty. [See, e.g., interpretational guideline 4.12.] I think that there is a reasonable argument that a Finnish meteorological agency doesn’t count as a “person” for the purpose of 12(5)(f). An agent of a foreign “public authority” would, likewise, not be a “person” for this exception, as they aren’t providing information that belongs to themselves, personally.
Non-contractual confidentiality agreements:
Phil Jones may, in fact, have a legitimate moral duty to prevent disclosure of borrowed data, based upon his prior oral commitments to the agents of foreign public authorities. Dr. Jones’ gentleman’s agreements are NOT binding on the UK, however. His personal failings, if any, don’t keep the UK from being “entitled” to disclose data in their possession.
To quote from the BNRR case:
So, after the data has been used and three years have passed, Jones’ gentleman’s agreements may bind him, but it doesn’t follow that the data necessarily can’t be supplied pursuant to a request for publicly-held information. Jones would only be bound if somebody stood up and said. “Hey, that’s my information !” It sounds like he doesn’t even remember his promises, if any.
Re: jim edwards (#79), Your argument regarding the statute of frauds isn’t quite water-tight. In most jurisdictions, the statute of frauds is now regarded with disfavor, which, under the currently popular judicial philosophy (which I call by the inflamatory title, “the imperial judiciary”) means it is construed narrowly–bizzarrely so. In most jurisdictions, the statute of frauds is read to apply only to agreements that cannot possibly be executed in less than a year. This means that an oral agreement can remain ienforceable, theoretically indefinately, as long as it is possible for it to end sooner. In this specific case, for example, if the agreement were to maintain the data in confidence until the source published something with it, that agreement would not fall under the statue of frauds in most jurisdictions, since the publication could possibly happen in less than a year.
Even in jurisdictions that apply a more traditional, broader rule, the issue would be controlled by the intent of the parties. That is, if the parties entered that hypothetical agreement believing that the publication would happen in 9 months or so, then it would be enforceable, even after 16 years, but if they believed publication would take 15 months, the agreement would be unenforceable. (However, even in the former case, after 16 years the contract might be–likely would be, IMO–reformed by a court due to the situation being so far outside what the parties contemplated.)
None of this should be read to contradict your larger conclusion. To the contrary, I agree; I see no evidence of any confidentiality agreement applicable to the present issue, namely public disclosure of the HADCRU raw data.
The admission that confidentiality agreements have been lost means that the confidentiality excuse can’t be used to deny documents under FOI.
All a person has to do is appeal one’s denial and state that you question the existence of any agreements legally limiting the proliferation of the data. The denying agency would have to present the agreements at appeal. It’s up to them to prove that you don’t deserve the data.
No agreements = no proof.
No proof = reversed denial.
Ok, so what was the data which was unearthed by the mole? Steve reported that he wasn’t told the material was not public at one time. And since he wasn’t told to destroy it, it should now be available for examination. So is it part or all of the HADCRUT original data or not? If not, what was the big deal in the first place? If it is original data, does that contradict what CRU now says?
Steve: The “mole”, who appears to have been Phil Jones himself, placed station data online in Feb 2003 (versions from 1996 and 1990 were online as well.) This was the data which, in 2009, was supposed to be subject to “confidentiality agreements”. Up to July 27, 2009, CRU had simply disregarded the supposed agreements.
Is there such a person as an Auditor General there that can be asked to examine their methods?
Standard letter received in respect of CAROLINE ISLANDS, SOLOMON ISLANDS, WALLIS ISLANDS, COOK ISLANDS, NIUE ISLAND
Not quite the same thing. Losing agreements should a lot more difficult than losing data, because agreements should be kept at the attorney’s office. This is doubly incompetent.
Granted all the harrumphing about the loss of data, or the missing confidentiality agreements, but it does make me think of the recent news that the original film of the first moon walk has been lost by NASA. As reported recently in the news, the degrated television video is what we all remember from seeing it over the years. The actual high quality film was put away years ago, and probably thrown out. If it doesn’t occur to anyone at NASA to save the original film of the first humans walking on the freakin’ moon, I can well imagine a British climate agency losing track of sets of temperature data from around the world.
A UK citizen must be able to bowl over the limited and now documented evidential basis for refusing to supply the data based on UK law if Jim Edwards is correct.
Even if it necessitated seeking a release from one or more suppliers or simply excluding those countries.
The interpretation bulletin says:
If there are “no other circumstances that entitle the public authority to disclose it”, then CRU could not have disclosed the data to Peter Webster. So obviously they were circumstances that entitled the public authority to disclose the data.
They also disclosed the data to the Met Office.
They also sent a version to me in 2002.
They also placed versions on their website (taken down only in July 2009.)
While I admire everybody’s persistence in going through channels, in my business (newspaper reporter), if I get the stall and runaround like this, I go to the top.
Not sure who the top would be here. Minister of Defence?
The minister is an MP, right? So he has a top, too. Several, starting with his constituency committee, I believe it is called.
You’re proved that going through channels, although amusing, is not productive.
I’m surprised there has, as yet, been no refusal on cost grounds.
JF
Sounds to me like a serious case of “We don’t want to show you the real data because big chunks of it blow away our findings so we ignored them.”
Steve: As I’ve said over and over, I do not share that opinion.
This is now getting close to an end game. The prosaic truth is far from a Robert Ludlum novel, simply: “we run such a sloppy house that we don’t even know what we have or have not lost”.
The reputation saving obsfucation crumbles as it must. Oh what tangled webs we weave…
Isn’t the term “non-academics” really just a proxy for “commercial”?
Re: Alan Wilkinson (#97),
No. Forbidding non-academic access restricts it to whatever “academic” means. Forbidding commercial access would still allow access by anyone else — cryptologists looking for secret messages which might have been inserted in data streams, artists needing museum selection information, trainspotters looking for rolling stock in good condition, nonprofit global regatta founders.
Re: Eric (#130),
No, the rumor was that NASA had found the lost lunar walk footage. They haven’t, they’re merely restoring TV footage. NASA found records that the original video (the only one stored on a data tape rather than video tape) was recycled for other data. There were two test tapes, which due to being an equipment test were outside the normal record handling, whose status is still unknown.
Regarding routes that could be taken, perhaps this stunning situation should be raised with:
1. The Information Commissioner – it is the responsibility of the IC to ensure that the FOI rules are adhered to
2. The Commons Public Accounts Committee – they have the power to audit the performance of public bodies (but are of course on vacation for weeks to come)
3. The Shadow Environment Secretary (I would not bother with the Government’s ES, go to the Conservative opposition representative)
4. The Daily Mail, The Daily Telegraph, who will get great pleasure in exposing this sham of a government department
5. Channel 4 TV who might well be interested, given that they did commission and broadcast “The Great Global Warming Swindle”
I do sincerely hope that the end game is approaching here. If the UK’s most “prestigious/authoritative” climate organisation cannot/will not find the requested data and agreements then they are either terminally incompetent, or not to be trusted.
Surely even the BBC could get that?
In my experience in the nuclear industry, all data before electronic storage was kept on duplicate microfilm or microfiche, the copies being stored in separate locations (e.g. one copy down a salt mine, the other in an on-site store). No rocket science needed – easy and relatively cheap.
Re: Phillip Bratby (#99),
Agreed,
In 1974 the Australian Government became less than amused by the antics of its intelligence and security sectors and set up a Royal Commission. My mate George was Secretary and did most of the writing for the Judge.
I was reading some of the report today and was struck by the parallels. Here are the words from 1975-6. The records are preserved in their entirety to this day. Of course it could be done properly.
They have the raw data, but don’t want you to have it. You’re getting the bigtime runaround. This raw data must be of very high value if it’s worth ruining the very reputation of the institute entrusted to keep it.
I suppose the possibility exists that the CRU is really completely in shambles, as they say. If so, then it should never be used by or cited in any credible scientific works.
@ Tony B,
It just happens that Nick Herbert, the Shadow Environment Secretary, is my MP. Funnily enough, while reading the Daily Telegraph this morning, the idea of writing an email to him crossed my mind.
snip – overeditorializing
I think I’ll compose an email in the near future but wait until Parliament comes back from its well-deserved holiday. Quite likely Nick will be away on holiday during the coming weeks – probably in a country with a significantly warmer climate.
Chris
They have lost the original data? Those guys are not serious! The CRU temperature record might as well be a series of random number. It should not be used by anyone anymore.
@Lars K – 105
Wasn’t someone else using random numbers too (allegedly…)? Or might as well have been, according to some very useful local detective work.
If I am the gatekeeper for some historical data, and a most unfortunate set of circumstances leads to the most unfortunate loss of the original data…..then could I reasonably make and/or justify any adjustments to the historical (lost) part of the record after the losss of said data?
re: #107
‘after the loss of said data’
sseriously ssad sspelling.
@ Chris Wright – 104
Isn’t the pre-requisite for a successful political career the ability to say the right things at the right time – oh, and that’ll be the same for a climate science career too, I suppose…
With all the problems in the U.S. data sets, you had to know the world-wide data had many issues as well.
snip
Lack of computer space? Please. Even 20,000 stations daily data for 30 years would still fit on a flash drive or an old 9-track tape. I ran simulations in 1980 that generated that much output.
A quick question:
Who holds the original data logs etc. for the majority of the planet, i.e. the oceans? Hopefully it is safe somewhere.
I am not sure, but I believe that Hadley SST data is used in other global data sets. Should there be a traceability issue with this data then it will be inherited by such other datasets.
Also does the IPCC have any standards regarding the traceability of data products used in the construction of its reports?
Steve BTW, I am having real problems posting comments via IE8 (Typing not appearing on the screen for up to a minute. This missive comes to you courtesy of Opera 9.26.
Alexander
Re: Alexander Harvey (#111),
Hadley SST is run by a different operation. ICOADS is the raw material for gridded SST and there is voluminous material online and there seems to be concerted effort to preserve the metadata and information attached to the measurements, not just the number.
The SST people seem to have inherited maritime traditions of care with logbooks etc. Many logbooks from vessels have been preserved for decades.
CRU’s reckless lack of stewardship with land data entrusted to them is totally at odds with the UK maritime tradition of stewardship of maritime data.
If they simply tag new data for a station onto old data, which is what it sounds like they are saying, and that the old data is not the raw but their processed data, then any corrections made by a country to past data for homogeneity or scribal errors etc, will never make it into the CRU database.
Re: Craig Loehle (#112), This is why I believe they have at least a portion of the original data. While they may not maintain a duplicate of the GHCN database the idea that they have no original data for stations outside GHCN does not, in my opinion, pass the common sense test.
first time poster, long-time reader.
As a government employee in the states, I would expect to be “terminated” were I to react in this fashion. It is this type of behavior which makes citizens distrustful of all government employees. “Who’s” hiding “what” and “why” – disgusting.
The loss of the original raw data means the original and subsequent science cannot now be verified.
Human error or a human convenience?
No ready answer, but we can take it that CRU now suffers from a lack of credibility.
Just think of all those papers, journals, books, scientific news-pieces and editorials which referenced CRU value-added data that now have a large ? beside them.
Thanks Steve,
That realy is good news regarding SST data archiving, they have saved two thirds of the planet.
I have to say that just uncovering the loss of the original land data alone, has probably made all your efforts worthwhile.
Ignoring the politics and the amusement that can be had over this, how can this be fixed?
Whatever your point of view and beliefs on climatic change, reliance on the integrity and traceabilily of the temperature record is likely to be a paramount concern.
It seems that they have gone to a lot of effort to produce a very important data product and have let the foundations and perhaps several of the lower floors wash away. Leaving a “Castle in The Sky”. It is a tradegy.
Hopefully someone somewhere has got enough of the original data to at least start to unpick this mess. Hopefully the CRU have made the original data available at sometime to someone and they have kept portions of it.
Question to anyone: Are other data sets, e.g. GISS reliant on CRU value added land temperatures for any regions/countries, or do they hold all the original data to which they add value?
Otherwise I fear that the MET/CRU may be the only data repository for ex-Britih Empire countries prior to independence. Or rather they should have been but neglected to be so.
No joke, I feel sick about this. Not a good day for the UK.
Alexander
Does this exercise really matter? Can we expect the specific data gathering, collation, error checking, and adjustment process to affect the temperature record in a significant way, or are we just chasing ghosts.
To educate myself on the variations between the various temperature products, I googled “giss vs hadcru3 temperatures”.
Interestingly, the 1st result was a Climate Audit post, the 2nd result was Lucia’s Blackboard, and the 3rd was Watts Up With That.
The Climate Audit Feb 2007 post, HadCRU3 versus GISS showed the difference between the HADCRU3 grid and the USHCN-2000 gridded product across the lower 48 USA. An difference in annual temps of about -0.35C in the 1930’s to +0.28C recently.
So it appears that the DIFFERENCE in adjustments between USHCN-2000 and HADCRU3 is on the same order as the observed change in temperature.
We already know from other posts that the total adjustments made by NOAA to the US surface station record are about the same size as the long term changes in temperature, but the above result showing that the differences in adjustment made by two sets of scientists also approach the magnitude of observed warming was somewhat surprising.
The bottom line is that it tells me that this exercise in attempting to replicate HADCRU3 is worth the effort.
Re: Charlie (#118),
It matters for even bigger reasons (for the AGW climate scientists even more than the public). The public has been told that they must defer to the expertise of climate scientists who supposedly have reached a consensus. Any information which casts light on the quality of the science practiced by the “experts” gives the public guidance on whether to acquiesce to the experts’ judgment.
Over the last few years the curtain has been drawn back quite a bit. The sloppiness of so much of what passes for “science” in the climate area has been truly astonishing. The public really deserves to see everything behind the curtain.
Note, for their own good and the public’s, I think it is imperative that those scientists whose scientific practices are top quality be as transparent as possible. The revelations of sloppy science by so many of their contemporaries threatens to reach a tipping point (if it hasn’t already) which will cause the public to lose confidence in all of them. That would be very unfortunate for the public and disastrous for the quality scientists.
Re: Charlie (#118),
“So it appears that the DIFFERENCE in adjustments between USHCN-2000 and HADCRU3 is on the same order as the observed change in temperature.
We already know from other posts that the total adjustments made by NOAA to the US surface station record are about the same size as the long term changes in temperature, but the above result showing that the differences in adjustment made by two sets of scientists also approach the magnitude of observed warming was somewhat surprising.
The bottom line is that it tells me that this exercise in attempting to replicate HADCRU3 is worth the effort.
”
I agree. I think at the risk of being snipped for editorialising, that what Charlie has posted needs to be repeated. Because of various ‘issues’ (e.g. TOBS, UHI, satellite orbital decay, bucket adjustements etc etc) with all FOUR anomaly indices which result in the need to apply various adjustments to the raw data, the adjustments that are made to the ‘raw’ data whether it be ground stations or satellite or radiosonde balloon data etc are of the same magnitude as the claimed warming trend (claimed by some to be unprecedented in the last 1000 years!) in mean global surface temperature over the 20th century! Given the levels of uncertainty/variability in these measurements this means that we CANNOT be certain to any significant level of confidence that the claimed warming is nothing more than an artefact of the adjustments made to the raw data. This IMO is the main conclusion that can be reached from the auditing work that Steve has carried out so far on his blog. This IMO is the main reason why Steve’s request for raw data and details on how the data is adjusted are resisted/opposed by the ‘gatekeepers’.
Since this THE key piece of evidence on which the IPCC claims that man (due to his continued use of fossil fuels) is having an appreciable effect on the earth’s climate, then it is clear that their claim is manifestly unsupportable. In reality because of the ‘issues’ with how we measure the earth’s temperature in the past and present continue the exist then the bottom line is we wil most likely never know to any significant level of confidence whether or not man has had or ever will have an appreciable effect on earth’s climate! Now given even if we could delude ourselves that we are are having an apprciable effect on our climate, the claimed warming trend is so small as it is (its most definitely not unprecedented and is NOT signifacantly greater than recent climatic warming anomalies like the RWP and MWP which were clearly beneficial to man) then why on earth are we introducing legislation to control the level of our emissions of a trace gas (vital to the continued existence of all life on this plant) into our atmosphere?
KevinUK
Whithout attempting to justify the QA/QC (or lack thereof) in this data there is a point which can help explain the obfuscatory nature of *some* of the correspondence.
Consultants routinely insist on a “no reliance by third parties” clause when releasing information or reports to their customers. The intent of the clause is to prevent, particularly in certain large North American countries, torts by persons who relied on the information without understanding the context. Say, in a feasibility study, for example.
Let’s say I prepare a report for a junior exploration firm discussing what their mine might look like and what it might cost. I neglect to add the clause. They post excerpts fromt the report in their quarterly statements. Shares go up. Nothing is there. Shares go down. Consulting firm gets sued out of existance.
The clause also ensures that data that is subject to change and revision is refreshed from the source each time. QA/QC can take effect. Potential customers are required to contact the consultant, develop a relationship, and maybe provide the consultant with more work.
Now – lawers being who they are – this type of language gets more and more complex and people in the company who don’t understand WHY the clause is there pedantically insist on it’s inclusion on anything and everything without regards to its relevance. It can make it look like the consultant won’t stand behind their report. All sorts of terrible things (like losing a customer – worst possible) can happen.
In this case it appears that the clause is being retroactively added to documents and data. Older data sets (pre-80’s) probably didn’t have any such thing as this issue did not exist in its current form then. The folks who reply to FOI requests would seem to be the exact type of staffers who would *assume* that such a “no release to third party” clause would apply to everything.
Perhaps a useful FOI request is to inquire for a copy of the CRU current policy regarding the “no release to third party” clause, the standard text of the clause, and when it was first applied consistently as a policy.
This would be comical if it were not for the fact that databases like CRU are used to formulate Government policy.
With your pernission, Steve, I would like to use the information in this blog to
a) inform some of the less climate-fevered press about these shenanigans
b) make a formal complaint to Ed Miliband M.P. Head of the new (U.K) Climate and Energy Department”
I was told that the cruwlda2.zip data already has a correction for the UHI (presumably 0.05 C/century). By implication, so does newcrustnsall.dat.Z and later versions of the data. Thus, if anyone asks for raw data, they are asking for a data set that is NOT cruwlda2.zip or newcrustnsall.dat.Z. The earlier data may not have been retained, as they say, after the UHI correction. Therefore, they can truthfully deny all FOI requests for the raw data. It is rather tricky way for them to respond.
The FOI request perhaps needs to be more specific, asking for the updated version of newcrustnsall.dat.Z and information on the known differences between this data and the raw data.
After what point in all this do the FOI requests end and the Lawyers are brought in?
UK Met Office
Does this mean only those supporting global warming can get the data?
As a community, I think we tend to be too judgemental. People here are jumping to the conclusion that it was incompetance. I prefer to remain less judgemental and rather assume that many Climate Scientists are simply unfamilliar with many of the methods used in other branches of science. Climate science is a bit insular after all. In that spirit, I’ve begun work of a new article that I intend to submit to the Journal Science: ‘Novel Method of Climate Data Retension and Extraction Through the Use of “Filing Cabinets”‘.
Just think about what the lost raw data means to CRU. They can never modify their adjustments. Ever. If they determine at some point they want to use a different inhomogeneity calculation or adjust UHI differently, they simply can’t.
Just wow.
Clark,
Of course they can modify their adjustments. Who’s going to stop them from making whatever adjustment they want?
Andrew
OT (but interesting) response to #85 regarding NASA losing the original video of the original manned lunar landing. It was lost for 40yrs but they found it.
http://www.neowin.net/news/main/09/07/16/nasa-to-release-high-resolution-film-of-first-moon-walk
Pielke Jr comments here
The response I received confirms no agreements are on file for France, Germany, Italy, Austria and the Ukraine.
.
I have often used the analogy of national Consumer Price Indexes to illustrate the ridiculous situation of the “Global Temperature” data. Each country has large professional staffs at their Stat agencies working on the monthly CPI using international protocols, using transparent methods, with independent academics looking over their shoulders weighing the various aggregation methodologies (e.g. Paasche, Laspeyres, Fisher, Tornqvist etc index number formulae), and with historical archiving rules that allow backward revisions periodically if needed. It’s by no means perfect, but it’s a far cry from the f**king gong show we’re seeing here. The reason CPI data, GDP data, etc. are handled professionally is that a range of policies (such as money supply control, pension indexing, intergovernmental transfers) etc depend on the numbers; also some labour contracts include CPI-based escalator clauses. In other words the numbers matter.
.
By contrast the Global Temperature numbers are coming from a bunch of disorganized academics chipping away at it periodically in their spare time. GISS numbers are handled (on Gavin’s admission) by a single half-time staffer, and the CRU says they’re stumped trying to find their original files back into the 70s and 80s, as well as the agreements under which they obtained the data and which to this day they invoke to prevent independent scrutiny.
.
I take Steve’s point that these issues are not sufficient to change one’s views on any particular policy which could be shown on other grounds to be desirable or otherwise. And this is not the thread (or the site) to debate policy. But bear in mind that the G8 Summit recently issued an undertaking to prevent the Global Temperature from going up by 2C compared to pre-industrial times, and the Copenhagen Agreement will probably contain similar language. That ‘2C’ number refers, in all likelihood, to a few digits that Phil Jones publishes in his spare time using data from files he has lost, etc. etc. If that kind of quality control is good enough for the purposes of G8 agreements and international protocols, then let’s not waste so much money running national statistical agencies to produce numbers that are used for such trivialities as central bank policy, national pensions and labour contracts. We’ll just ask some professors to cobble together a monthly number in their spare time, we’ll let them do historical revisions any time they want without notice or external scrutiny, and we’ll promise them that nobody will ever check their work.
.
And if the CRU staff want to plead that we are being too harsh on them because they “did the best they could” with such limited means and minuscule staffs, etc, fine. I’ll be the first to join in such a defence: the minute they put a big red disclaimer on their data products saying “Warning: These numbers are offered on an as-is basis and should not be used for any important public purposes.” But they can’t have it both ways, claiming their data products are good enough for policy purposes, while also claiming they cannot be expected to meet the quality expectations that apply to the other data series we use for policy purposes.
OK, no arguments. I’ve only been an occasional lurker lately, so if this has been covered in more detail elsewhere, feel free ignore, but, whatever do you mean by
As in to develop policy? or to implement policy (whatever it is)? or, Something Must Be Done!?
(note that I did not ask, “Policy based on what?” No arguments here, not even by Socratic interrogation!)
.
I can almost pity the poor folk at CRU – they must feel like they’re under siege by a swarm of auditors (but then, they could have deferred on the super-duper computer until they had their house in order).
.
Even given the unbelievable sloppiness at CRU, I’m a little bothered by Steve’s harsh, heavy-handed sarcasm. The power of CA has always been its cooly analytical approach, leavened by the occasional zinger. I hope this thread is just a holiday.
Steve’s battle is now mentioned in Nature Magazine
http://www.nature.com/news/2009/090812/full/460787a.html
Re: Leif Svalgaard (#136),
Thanks for the heads up. I was expecting something tomorrow and keeping it as a little surprise.
I guess we can now add another debating point when discussing with the Other Side. It used to be “we have good non-proxy data from at least 1850”. This will now be, what, 1990 or so?
re: #133 above
Isn’t 2C the margin of error on tree ring proxies?
In view of the wholesale “adjustments” made by those of the Hansen persuasion to data of various origins, though perhaps especially of data originating in the USA, I have to wonder what sort of effect the CRU manipulations, if any, or indeed misplacements, might have on our knowledge of large scale climate effects. It seems to be accepted quite widely that climatology is at best an approximate science in which opinion and methods of working can have a large influence on the outcome of a learned paper. It is easy to think of an example! However, I hazard that the loss/modification of original information on the scale that might have happened at CRU is unlikely to have such profound and global political effects as the methods adopted in MBH98 and ’99 have had.
Perhaps the CRU data is still adequately wide-ranging and reliable enough for the general purpose of the goals of open-minded investigators to remain largely unaffected. If so we should be thankful for that. I recognise however, that this cannot be fully checked unless the original information surfaces somewhere.
Nevertheless I fear that over-tenacious pursuit of some missing information may be counter-productive in the long run.
Ha ha , they have been pulling your legs. This isn’t real, it’s an episode of “The Men from the Ministry”.
You can listen to episodes on BBC 7:
http://www.bbc.co.uk/programmes/b009smq5
From my reading of this it seems like CRU have lost some, but not all, raw data and that some, but not all, of the raw data they have may be covered by a confidentiality agreement. It therefore seems sensible to request that they publish such raw data as exists and is not subject to a confidentiality agreement and to document such raw data that is known to be lost and to list the information about such data that exists but is under a confidentiality agreement.
I believe I see an FOI request coming up
Re: FrancisT (#142),
And here it is
Just wanted to say congratulations all who sent in their FOI requests. Do you think the University Admin that has to deal with this will be pleased with the way Phil Jones has behaved over the years? My guess is no and that they will demand greater rigour in the future. This will eventually be beneficial for climate science.
Reporting In: I Received the standard notice (Ref. FOI_09-104) from David Palmer for QATAR, REUNION, ROMANIA, SAO-TOME-AND, TAIWAN.
Had a private sector company said, “oh gee, we seem to have lost our ERP data, but we’re quite sure our earnings are properly stated”, there would be hell to pay. The government would impose huge fines and there would be calls for the execs to do jail time. But for some reason we hold the public sector to a lower standard.
snip – policy
Maybe HADley has LOSTly the original data, but they haven’t, surely, lost the algorithms for adjustment and normalisation, HADTHEYly?
In addition to requesting info about confidentiality agreements, I had requested info about person or organizations to whom CRU had supplied data in the last 5 years. The response came back today. My questions in italics. CRU response in blockquote.
.
.
1. Please supply a list of persons and/or organizations to whom the CRUTEM data set has been provided to in the last 5 years. If personal privacy concerns prevent you from identifying which specific persons have been provided such data, then it would be sufficient to simply supply the organizational affilliation of those persons. For example, it has come to my attention that most likely this data set has been provided to Peter Webster of the Georgia Institute of Technology in the USA. If you are unable to confirm that such data has been sent to Peter Webster, then it would meet my needs if you were to simply list Georgia Institute of Technology as the recipient.
.
.
2. Please supply a copy of any agreements between your organization and the recipients or the institutional recipients of such data that impose any conditions of confidentiality, or which prohibit further transmission of such CRUTEM data sets, or which prohibit the public posting of such data sets on the internet.
.
.
3. Please supply a copy of any internal regulations or personnel regulations or rules regarding the supplying of such “raw” temperature data products to persons outside your organization; and any internal regulations, rules, personnel guidelines regarding the confidentiality of CRUTEM data records
He apparently misunderstood my question #2, which was for any agreements between CRU and the recipients (such as Peter Webster) of data from CRU. He answered with info about agreements between CRU and the raw station data providers. I will request clarification.
I poorly phrased my question #1, as it could easily be construed to inquire only about persons/organizations to whom the entire dataset has been supplied. In any future FOI or EIR requests like this I will of course include, .. or portion thereof …..
I’ve been considering holidaying in Kent, as I note from my guidebook that there is plentiful good wine to be had, on account of its numerous vineyards. However, I am also concerned that it is apparently also a reknowned malarious region, with many locals stricken with the ‘ague’. Of course my guidebook from the 1700s may be a little out of date, and the clemency of the weather may have shifted in the meantime. I would be grateful of some more recent information.
The dog ate their homework again.
Microfiche or punch cards were used back in the60’s, 70′ and 80’s for data storage. More than likely the data is stored on microfiche. Why throw out data that someone was paid to collect?
re my request:
I received:
Hi All
Sorry for the delay: I have been on holiday.
I received the standard reply from CRU for Zimbabwe, Zaire, Angola, South Africa and Botswana.
Derek
Norway sez:
Dang, instant PhD, that’s cool … I wrote back:
Onwards, ever onwards …
w.
I have now had a reply to my FoI request to the Met Office. I asked for copies of the agreement with Phil Jones which prevents further distribution of CRU data. The reply is as follows:
Re: Bishop Hill (#162),
I see in their response they used “verbal arrangement” rather than “verbal agreement”. Is there a legal distinction?
Man, I love the Nørwegese …
I replied:
So, we can cross Norway off the list of countries with agreements with the CRU … but that was the only (unsigned) agreement that they’ve been able to dig up.
The beat goes on …
w.
14 Trackbacks
[…] CRU Responds […]
[…] More from Steve McIntyre, at Climate Audit. […]
[…] you want to read more on McIntyre and his correspondance with the CRU click here. I highly suggest you do as it contains many more excuses on how the data was lost. Including such […]
[…] did, with Ross McKitrick, in deconstructing and undermining the famous “hockey stick” graph), writes on his Climate Audit blog, it appears that the impoverished CRU even lacked filing cabinets in which to store its […]
[…] did, with Ross McKitrick, in deconstructing and undermining the famous “hockey stick” graph), writes on his Climate Audit blog, it appears that the impoverished CRU even lacked filing cabinets in which to store its […]
[…] did, with Ross McKitrick, in deconstructing and undermining the famous “hockey stick” graph), writes on his Climate Audit blog, it appears that the impoverished CRU even lacked filing cabinets in which to store its […]
[…] statistician and blogger Steve McIntyre, who has been asking for the data set for years, says he isn’t impressed by the excuses. McIntyre obtained raw data when it was accidentally left on an FTP server last […]
[…] than send to anyone.) ja että kun Jonesin olisi lopulta pitänyt luovuttaa alkuperäinen data, se olikin yllättäen kadonnut lopullisesti. Mutta kirjoitin nopeasti enkä muistanut. No, lukekoot […]
[…] statistician and blogger Steve McIntyre, who has been asking for the data set for years, says he isn’t impressed by the excuses. McIntyre obtained raw data when it was accidentally left on an FTP server last […]
[…] submitted an even more detailed comment in the same thread here: I have often used the analogy of national Consumer Price Indexes to illustrate the ridiculous […]
[…] of the global temperature datasets, one of the great scandals of climate science. As described in this comment by Ross McKitrick, posted at Climate Audit on 12 August 2009. I have often used the analogy of […]
[…] The “small document” is online at CRU here. It was reported at CA on August 11, 2009 here, here and here (among […]
[…] in late July was published on August 11 here, prompting an immediate response at Climate Audit here. CRU’s webpage did not include ANY confidentiality agreements concerning station data. […]
[…] in late July was published on August 11 here, prompting an immediate response at Climate Audit here. CRU’s webpage did not include ANY confidentiality agreements concerning station data. […]