“Community” climate scientists (e.g. Nature, Realclimate) have been quick to accept the idea that CRU was prevented from releasing station data because of confidentiality agreements with originating meteorological services.
Something that Nature and Realclimate didn’t discuss or even seem to notice – and the blind spot is symptomatic – is that Jones delivered “confidential” data to his friends. In a Climategate Letter in early 2005, Jones told Mann that he had just sent “loads of station data to Scott [Rutherford]“, Mann’s close associate.
I guess that it didn’t occur to Nature or to Gavin Schmidt that confidentiality agreements also apply to masters of the universe. If the data is as confidential as Nature and Realclimate say, then the distribution of confidential data to associates and other third parties in breach of the confidentiality agreements may warrant just as much attention as Jones’ effort to obstruct FOI requests.
CRU publicly articulated their position on the supposed confidentiality agreements with national meteorological services (NMSs) in summer 2009 both in interviews with Nature and in a short statement on their website posted in response to numerous FOI requests for confidentiality agreements. (These FOI requests constitute the vast majority of all FOI requests.)
On Aug 13, 2009, Olive Heffernan reported in Nature (v. 460, p. 787):
Jones says he can’t fulfil the requests because of confidentiality agreements signed in the 1990s with some nations, including Spain, Germany, Bahrain and Norway, that restrict the data to academic use. In some cases, says Jones, the agreements were made verbally, and in others the written records were mislaid during a move.
In an associated blog post, Heffernan added:
Why won’t Jones give McIntyre the data? Jones says that he tried to help when he first received data requests from McIntyre back in 2002, but says that he soon became inundated with requests that he could not fulfill, or that he did not have the time to respond to. He says that, in some cases, he simply couldn’t hand over entire data sets because of long-standing confidentiality agreements with other nations that restrict their use.
(In passing, none of these statements should be taken at face value. For example, Jones obviously did not become “inundated with requests” “soon” after my 2002 request.)
Jones also posted a webpage on the matter at almost the same time as the Nature article:
Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties.
As most readers know by now, in response to multiple requests for the supposed confidentiality agreements with national meteorological services, CRU produced only four “agreements” (UK Met Office, Spain, Norway, Bahrain). Obviously, these agreements would not have encumbered the entire data set for very long, as waivers from Spain, Norway and the Met Office could presumably have been obtained in a day or so.
Faced with these few embarrassing scraps, CRU argued that there were many other agreements that they had unfortunately lost or never committed to writing. Indeed, they didn’t even know for sure who they had made agreements with. CRU:
Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s. Some date back at least 20 years. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period.
They stated that these agreements contained language that was representative of the period and of what was used in the “lost” agreements, noting in particular that the examples showed that they “would not make the data available to third parties”:
In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.
The four agreements were placed online here.
The agreement with the Met Office (respecting data from various colonies and ex-colonies) is the clearest agreement. It does indeed include language that the data would not be made available “to third parties”. But it did more than that. It said that the data would only be used by the Climatic Research Unit and only for the “specific project” of “construction of a gridded baseline climatology for global land areas” and would “not be used unauthorized for any other project” or “passed on to third parties”:
These data are to be used by the Climatic Research Unit for a specific research project sponsored by the NERC, namely the construction of a gridded baseline climatology for global land areas. The project falls in the area of Global Environmental Change which makes this data request subject to the IACGEC Framework for Data Exchange to which the Met. Office is a signatory. The data will not be used unauthorised for any other project and will not be passed onto any third party.
The CRU “agreements” included a recent NERC form which clearly proscribed use of data for academic work outside of the original consented program:
UKMO data/software so obtained may be used solely for the purpose for which they were supplied. They may not be used for any other projects unless specific prior permission has been obtained in writing from the UKMO by a NERC Data Centre. Note that this applies even for other bona fide academic work.
The NERC form also stated that any “infringement” of confidentiality, whether intentional or merely negligent, would be taken very seriously – language that is a little ironic when one considers that CRU had left versions of the confidential station data on their FTP site for many years (see CA, July 2009, A Mole?)
NERC recognises that some data holdings supplied by UKMO under the arrangements are commercially valuable : the recipients of data are under an obligation to respect the terms and conditions of data supply, and to have regard to the security of datasets entrusted to them. Any infringement, whether by deliberate abuse or negligence, will be regarded extremely seriously by NERC, as endangering Council’s own reputation and the integrity of the NERC supported scientific community. The introduction of sanctions against individuals or Departments may be considered if breaches occur.
“Loads of station data to Scott”
For the sake of analysis, let’s stipulate the position of CRU, Nature and Realclimate – that the station data was restricted by confidentiality agreements.
Despite such agreements, on Feb 2, 2005, Jones emailed Mann notifying him that he had just sent “loads of station data to Scott [Rutherford]“, Mann’s associate. This email has previously attracted attention for Jones’ threat to delete the data rather than send it to “MM” and for Jones’ disdainful attitude towards UEA officials who would dare argue that the Freedom of Information might apply even to Jones. However, commentators have thus far not paid attention to the equally interesting information that Jones had just sent “loads of station data” to Mann’s close associate, data that would be denied to Warwick Hughes a few weeks later, to Willis Eschenbach in 2006, and to me, McKitrick, Pielke Jr, McCulloch and others in 2009. Jones wrote Mann as follows:
Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? – our does ! The UK works on precedents, so the first request will test it.
We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it – thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who’ll say we must adhere to it!...
Another distribution of confidential station data to a third party occurred last summer when Jones delivered station data to a professor at Georgia Tech, who, unlike Mann, was not a close associate of Jones, and who had innocently asked for the data, implicitly accepting the quid pro quo that Jones would be granted a coauthorship on any subsequent article. Olive Heffernan reported at Nature (2009, 460, 787) as follows:
Webster says his team was given the station data for a very specific request that will result in a joint publication with Jones.
At the time of his initial inquiry, Webster would have had no reason to presume that the CRU station data was encumbered by confidentiality agreements and nothing here is a criticism of Webster’s behavior, which has been scrupulous at all times. However, Jones knew of the confidentiality agreements and it was Jones’ obligation to honor the supposed confidentiality agreements. Nevertheless, Jones delivered the “confidential” data to Webster in the expectation of coauthorship – a coauthorship that would incrementally increase Jones’ H-Index.
The Climategate Letters show that Jones’ H-Index was important to him: Jones wanted to be an AGU Fellow and looked to Michael Mann to organize the nomination. In one letter, Jones and Mann had an unsavory discussion in which Mann said that he would report a false higher value of Jones’ H-Index in connection with Jones’ nomination as AGU Fellow. (Because Jones is a common surname, Jones advised Mann that an online algorithm for the H-index yielded an higher H-Index that was too high because it included articles by an unrelated biologist Phil Jones. Mann advised Jones that he would use the inflated value in the nomination form anyway.)
In subsequent discussion at the Nature blog, Heffernan was unable to see any inconsistency between Jones’ delivery of confidential data and any obligations to the NMSs to keep his data confidential. Jones’ desire for coauthorship may be understandable in academic terms, but it is totally irrelevant to his obligations under the NMS confidentiality agreements. The interest of the NMSs would be in the confidentiality agreement, not Jones’ H-Index. An increase in Jones’ H-Index is not a valid reason to breach the confidentiality agreements. Even Nature should get this.
There was an even more remarkable breach of the confidentiality agreements 20 years ago. The collection of station data by CRU was financed by a unit of the US Department of Energy (the Oak Ridges Nuclear Lab) – DOE financing continued for over 25 years. Notwithstanding confidentiality agreements in effect in 1990 (and CRU says that many agreements date from the 1980s), Jones delivered the data set to the Oak Ridges Nuclear Lab (Carbon Dioxide Information Center), who placed the data set online. This version is available to this day ( http://cdiac.ornl.gov/ftp/ndp020/ ).
The language said to be standard in CRU confidentiality agreements stated that the data was to be used only by CRU and only for the construction of a gridded temperature index and not to be used for other purposes and not to be passed on to third parties.
Jones flouted these terms. He passed confidential data to the Oak Ridges Nuclear Lab in the US, who had funded the collection of the data. He passed confidential data to his close associates, Michael Mann and Scott Rutherford. He passed confidential data on to a relative stranger in circumstances where he expected to increase his personal H-Index.
The reaction of the Community to these revelations is just as important.
Jones’ violations of the confidentiality agreements didn’t bother Gavin Schmidt, who said that there was nothing-to-see:
CRU data accessibility. From the date of the first FOI request to CRU (in 2007), it has been made abundantly clear that the main impediment to releasing the whole CRU archive is the small % of it that was given to CRU on
Similarly, Nature said that there was nothng-to-see-move-along:
But for much crucial information the reality is very different. Researchers are barred from publicly releasing meteorological data from many countries owing to contractual restrictions.
The idea that masters of the universe should be constrained by confidentiality agreements doesn’t seem to have occurred to them. It should have.
There’s an interesting example of this attitude in one of the Climategate Letters that arose out of the very first post in the current FOI sequence. On May 11, 2009, as mentioned in a CA post here, I had noticed that the UK Met Office webpage on CRUTEM stated:
To obtain the archive of raw land surface temperature observations used to create CRUTEM3, you will need to contact Phil Jones at the Climate Research Unit at the University of East Anglia.
No mention of restrictions due to confidential agreements. On May 12, 2009, Peter Thorne of the UK Met Office wrote to Phil Jones ( see Climategate Letter):
Mr. Fraudit never goes away does he? How often has he been told that we don’t have permission? Ho hum. Oh, I heard that fraudit’s Santer et al comment got rejected. That’ll brighten your day at least a teensy bit?
The Team has such an elegant rhetorical style.
Thorne was correct that our IJC comment had been rejected, but, at that point, I hadn’t commented on the status of our Santer comment. On May 28, 2009 (CA here), I stated:
Ross and I submitted a comment on this topic to the International Journal of Climatology, which we’ve also posted on arxiv.org. I’m not going to comment right now on the status of this submission.
It appears that Santer co-author Thorne was a reviewer of our submission, a reviewer who recommended rejection and who then raced to tittle-tattle about the rejection to his confidante, Phil Jones.