In today’s post, I continue my re-appraisal of various untrue statements made by the University of East Anglia in order to avoid disclosure of CRUTEM station data. I do not consider motives at this time. Also see preceding posts here, here.
In East Anglia’s response to July 2009 FOI requests for alleged confidentiality agreements (here) , CRU stated that, since the 1980s, they had entered into confidentiality agrements that prohibited them from providing station data to third parties, but were unable to “locate” any such agreements noting that they had “moved offices several times during the 1980s”:
Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties. Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s. Some date back at least 20 years. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period. In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.
In East Anglia’s submission opposing appeals by Jonathan Jones and Don Keiller (e.g. here), they made the even stronger claim that national meteorological services(NMSs) “invariably” released information only under licences that prohibited transfer to third parties:
It was invariably the case that most NMSs only released information under licences, both written and verbal, that prohibited the further transfer of the information. There was no standard form for such licences but they were all similar in that they prohibit the onward transmission of the data to third parties.
CRU’s longstanding relationship with the US Department of Energy and its transmission of station data to the US Department of Energy (which placed CRU’s station data online (e.g. here) is inconsistent with their claims that they entered into binding confidentiality agreements in the 1980s. In my 2009 appeal, I confronted East Anglia with this relationship (a matter that they discussed in email 2929).
Their refusal of my appeal was dated Nov 12, 2009 (one day before the last email in the Climategate dossier) and was sent to me on Nov 18, 2009. While the dates of these refusals have attracted some commentary, relatively little attention has been paid to the content of the refusal (reported at CA on Nov 21, 2009 here) in which East Anglia stated that the “restrictions” applying to the station data only arose after CRU provided data to the US Department of Energy (along the lines of a scenario that I contemplated in a CA post of Aug 4, 2009 entitled Dr Phil, Confidential Agent):
In regards the information provided to the US Department of Energy, my investigation has revealed that this was done in the early 1990s prior to the imposition of the restrictions now pertaining to the data pursuant to a contractual obligation at the time. Therefore, the analogy you are drawing does not apply to the data that is the subject of this request.
Despite this explicit admission, CRU’s claims that they have been subject to longstanding confidentiality agreements have been accepted by Nature and the climate science community, which has, in turn, denounced CRU critics for not believing in the existence of actionable CRU confidentiality agreements from the 1980s.
In preparing today’s post, I considered not merely the (limited but relevant) new information from Climategate 2.0, but also re-examined contemporary (1980s, 1990s) statements about provenance of station data. Prior to 2004, there is little to no evidence of the commitment to secrecy said by East Anglia to have been “invariably the case”. Quite the contrary. Prior to 2004, there is considerable evidence of a longstanding practice of making station data publicly available (both by GHCN and CRU).
World Weather Records
World Weather Records was (and is) a longstanding international effort in which station data (and metadata) have been published and, when the technology became available, digitally archived.
TD9644 , published in 1991, described its origin as follows:
In 1923, the International Meteorological Conference Committee, convened in Utrecht, Netherlands offered initial justification for the creation of WWR:
“…the conference thinks that publication of long and homogeneous series of observations in the form of monthly means of pressure, temperature and rainfall would be of the highest importance for the study of the general circulation of the atmosphere.”
In response to the Conference’s findings, the Smithsonian Institution of the United States sponsored the first series of WWR, edited by H. Helm Clayton and published in a single volume (1196 pages) in 1927. The publication included full period of record through 1920 of monthly means of temperature, pressure (station and sea-level) and precipitation for selected global stations.
According to TD9644, data was publicly available in 1991:
All available data are both published in the volume sets and digitally archived through NCDC’s tape library under DSI-9644.
Original World Weather Records report (some of which have been scanned and are available online) commendably describe the provenance of each record (something later said by CRU to be impracticable), e.g. the following:
The station names, elevations, and coordinates given in the table headings were supplied by the General Directory of Meteorology of Uruguay and were in use at the end of 1970.
World Weather Records continue to be published and are online at UCAR here.
Jones et al 1985-1991 and CRUTEM1 (1994)
Jones’ original compilation of station data, funded by the US Department of Energy, was described in three technical reports: Bradley et al 1985 (out of print and not online other than an excerpt here), TR022 (Northern Hemisphere) online here and TR027 Southern Hemisphere (online here), and two less-informative academic articles. The original version of NH data (ndp012) is no longer online, but a 1991 update is online here.
These documents indicate almost total reliance on prior World Weather Records compilations, plus archives at the UK Met Office, all of which had been made publicly available (contradicting later CRU claims of a longstanding practice of confidentiality agerements.) Jones et al 1985 (TR022) described the provenance of Northern Hemisphere station data as follows:
Most studies of global or hemispheric temperature fluctuations have relied upon the compilations of station data in World Weather Records… Bradley et al (1985) have added considerably to the WWR data using material available in published and manuscript form in meteorological archives, particularly those of the UK Meteorological Office… Full details of these improvements in station coverage are given in Bradley et al 1985. The most important improvements in coverage occur over parts of the Soviet Union and northern Europe particularly before 1881…Further improvements in coverage have also been made for the twentieth century particularly over northern Africa before 1940 and over the Peoples Republic of China…Details of each of the 2666 stations in the data bank are documented in Appendices using the formats described in Goodess et al 1985 and Bradley et al 1985.
TR027 provided similar statements about Southern Hemisphere data. It reported the compilation of 610 station records of which 293 were used in the grid. Only one meteorological service (Peru) was mentioned. In passing, two of the authors cited (Pittock 1980; Salinger 1981) turn up many years later as leaders of the campaign to censure Climate Research for the publication of Soon and Baliunas 2003.
The basic source of station air temperature data for the Southern Hemisphere land masses is the set of volumes of World Weather Records (WWR) (Smithsonian Institution, 1927, 1934, 1947, and U.S Weather Bureau, 1959–1982; available in digitized form from the National Center for Atmospheric Research (NCAR), Jenne, 1975). A considerable amount of additional temperature data for Argentina and Chile for the years 1931-60 has recently been added to this set. In WWR, these countries only have data available from 1951 (see Pittock, 1980, for further details).
Searches for data in archives as part of the present project yielded additional data for Indonesia and Australia and for some Pacific Islands, particularly Tahiti. Additional data for New Zealand was found in Salinger (1981). For Peru, the Peruvian Meteorological Service supplied information for about 10 stations covering the 1940s and 1950s. Additional data for Australia was provided by their Bureau of Meteorology. All of these sources are gratefully acknowledged.
In 1991, an update to the gridded data set (ndp020r1) re-iterated this description of provenance of station data, noting that all station data was available online.
The primary sources of these data are the World Weather Records (WWR), published by the Smithsonian Institution and the U.S. Weather Bureau, the archives of the United Kingdom Meteorological Office, and the Monthly Climatic Data for the World, published by the National Climatic Data Center (Asheville, North Carolina). Additional sources are described in Bradley et al. (1985) and in Jones et al. (1985 – TR022, 1986a – JCAM, 1986c – TR027, 1986d – JACM). The present updated version of this data set is identical to the earlier version (Jones et al. 1986b) for all records from 1851 through 1978. For the period 1979-1984, the present data set corrects erroneous data using satellite data for some sites and appends data for other sites by adding previously unavailable station data (Jones et al. 1988). The present package also adds monthly surface air temperature anomalies for the period 1985-1990, Antarctic monthly surface air temperature anomalies for the period 1957-1990, as well as the monthly mean temperature records for individual stations (Antarctic stations excluded) that were used to generate the set of gridded anomalies. Individual station data for the Antarctic (stations south of 62.5S) are not presented in this package but are given in Jones and Limbert (1989 -NDP032) and may be obtained free of charge from the Carbon Dioxide Information Analysis Center.
In Climategate 2.0 email 2929 (Sept 8, 2009), Jones said that it was a “requirement” of CRU’s contract with DOE that the data be placed online:
I don’t recall all the facts from that long ago. There is or was a version on a US Dept of Energy website from about 1990. This was a contract requirement at the time.
In 1994, Jones expanded the 1873-station data set to 2891 stations, described in Jones et al 1994 (later CRUTEM1), ascribing the provenance of new data primarily to the projects described in Karl et al 1993 (all of which had been placed online.)
In 1996, CRU placed the CRUTEM1 station data online (/projects/advance10k/cruwlda2.zip), where it remained online until CRU’s removal of the data from its website on or about July 29-30, 2009 (4270). Peterson and Vose (1997), in an article introducing the second version of GHCN, described the 2891-station CRUTEM1 dataset as “widely used”. Doug Hoyt reported that he had downloaded the file in 1999. Simister (2002) and Simister and van de Vliert (2005) report that they downloaded the data in November 2001. I downloaded the file cruwlda2 as late as July 25, 2009 (an event discussed by CRU in Climategate 2.0 4270). In 2002, Jones sent me (then unknown) the cruwlda2 dataset, mentioning that an updated dataset would be published early in 2003.
Jones et al 1999 continued use of CRUTEM1:
Here we use the land station data set developed by Jones [1994]. All 2891 station time series used have been assessed for homogeneity by subjective interstation comparisons performed on a local basis. Many stations were adjusted and some omitted because of anomalous warming trends and/or numerous nonclimatic jumps (complete details are given by Jones et al. [1985, 1986c]).
In 2009, Jones told (2929) East Anglia administrators that the cruwlda2 version wasn’t “complete” and that it was placed on an ftp site because it was “easier” to do this than to send out disks. (Jones did not explain how this related to compliance with supposed “confidentiality” agreements.)
The 1996 version (cruwlda2) wasn’t a complete version and was something we developed for a number of people in EU projects to use. We made these available to people on these projects via our ftp site, as it was easier to do this than sending disks at that time (email attachments were smaller then).
Whether the data was provided by email or ftp was obviously irrelevant to compliance with alleged confidentiality agreements. In addition, contrary to Jones’ claim to his administration, the 1996 version (CRUTEM1 or cruwlda2) was complete.
GHCN
Important light is shed on practices of the period in articles introducing the two GHCN versions.
GHCN-v1 was introduced in 1992 and was described in Vose et al 1992 NDP041. Interestingly, the 1891-station Jones network (ndp020r1) was incorporated as one of the sources for GHCN. ( The “raw” GHCN dataset lists several scribal versions. I’ve looked at a few stations in GHCN-2 and can trace one of the scribal versions to the CRU version. There’s an interesting twist here that I’ll discuss on another occasion.)
In 1997, Peterson and Vose released a second version of GHCN (the one in use until recently). Contrary to the regime of secrecy described by CRU, Peterson and Vose (1997) said that national meteorological organizations were “cooperative and enthusiastic” about contributing data to GHCN:
Because numerous institutions operate weather stations and because no single repository archives all of the data for all stations, we employed five acquisition strategies to maximize the available pool of data: 1) contacting data centers, 2) exploiting personal contacts, 3) tapping related projects, 4) conducting literature searches, and 5) distributing miscellaneous requests. In general, most parties were cooperative and enthusiastic about donating their data to the GHCN initiative, particularly since GHCN is a World Meteorological Organization (WMO) Global Baseline Data Set. As a result, GHCN version 2 contains data from 31 diverse sources (Table 1).
Whereas “personal contacts” between CRU and meteorological organizations were said to be fraught with secrecy, Peterson and Vose (1997) reported that their “personal contacts” had resulted in fresh contributions to public archives:
We also exploited personal contacts by contacting colleagues in the search for potential data sources. For example, scientists who visit or work in conjunction with the authors’ respective institutions often either have data themselves or are able to facilitate the acquisition of data from another party (e.g., by putting the authors in contact with potential sources). This was another extremely productive means of acquiring data, which yielded approximately 10 new datasets.
They reported that these efforts had yielded a much larger data set than the “widely used Jones (1994 “2961-station” dataset (a comment that further confirms the non-secrecy of the Jones 1994 station dataset.
With 7280 stations, GHCN is over twice as large as the widely used Jones (1994) 2961-station mean temperature dataset.
CRUTEM2 (Jones and Moberg 2003)
As noted above, in East Anglia’s rejection of my appeal in November 2009, they said that “restrictions” applying to the data arose after the data had been sent to the US Department of Energy. Here East Anglia was attempting to rationalize Jones’ admission (2929), in which, after admitting that he had sent station data to the US Department of Energy under a “contractual requirement”, claimed that the “restrictions” arose in the mid-to-late 1990s:
There is or was a version on a US Dept of Energy website from about 1990. This was a contract requirement at the time. Much extra data has been added since then, and this is what the restrictions refer to from the mid-to-late 1990s.
Needless to say, this position is totally inconsistent with their previous claim that they had entered into agreements in the 1980s and had lost the agreements during office moves during the 1980s.
It also raises other interesting issues that have not been examined to date: if the only relevant “restrictions” arose in the mid-to-late 1990s, this is well after the “office moves” of the 1980s. It is implausible, to say the least, that CRU should have no record of any communications with NMSs in the late 1990s relating to the confidentiality of station data. It’s hard to believe that data arising from such agreements would not have been transmitted electronically.
Jones and Moberg 2003, which introduced CRUTEM2, briefly discussed CRU’s contact with NMSs (discussed in a prescient August 4, 2009 CA post Dr Phil, Confidential Agent). Jones and Moberg 2003 gave no indication of that it had commenced a new practice of using confidential data. It complimented meteorological organizations for improvements in the quality and quantity of data exchanged and made available “for all to use”:
The majority of the world’s countries have endorsed the initiatives of the Global Climate Observing System (GCOS) to improve the quality and quantity of monthly climate data routinely exchanged. …
The second reason is that several national and other initiatives have also dramatically improved the quantity and quality of monthly mean temperature data available. Several countries have extensively homogenized their entire national holdings, releasing the results for all to use. CRU over the last eight years has also received several national and other temperature datasets.
Jones and Moberg 2003 listed these other datasets without mentioning any confidentiality issues. Whereas earlier editions of CRUTEM had not mentioned direct CRU contact with NMSs, Jones and Moberg reported that CRU had obtained station data through direct contact with some difficult or then pariah regimes: Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan.
CRU has collected a number of temperature records through direct contacts with the NMSs in Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan. Many of these records cover only the period 1961–90, but others extend over the entire twentieth century.
Jones and Moberg (2003) also reported the acquisition of data from less colorful regimes (that presumably attorned to WMO Resolution 40): NORDKLIM data from Denmark, Sweden, Finland, Norway, Iceland, Ireland, the Netherlands, Belgium and the UK itself, as well as additional data from Canada and Australia.
Accordingly, despite the elaborate charade by CRU and the UK Met Office purporting to seek consent from every NMS in the world, there were only a relatively small number of countries (Algeria, Croatia, Iran, Israel, South Africa, Syria and Taiwan) from which “restrictions” from the “mid-to-late 1990s” could have arisen.
2005
Mosher and Fuller (CRUTape Letters) concluded that Jones’ attitude towards provision of data was transformed by Mann’s unrelenting “animus”. They state:
Jones, guided by Mann’s example of animus toward McIntyre and others who question the work of climate scientists, and guided by Mann’s relentless appeals to motive is transformed from a researcher who once promised to share data even in cases where it appeared there might be legal cause to withhold it, to a researcher who cares first about motive, and second about his own reputation and who in the end will use every legal and bureaucratic means to obstruct the release of temperature data and threaten its destruction.
Climategate 1.0 (and especially) 2.0 emails contain an enormous dossier of spite following the publication of MMM2003 in October 2003, exacerbated by the publication of our MM2005a,b in late January 2005. Jones’ emails, as Mosher and Fuller observed, now become even more partisan. In an email in which Jones reported the sending of station data to Mann and Rutherford (despite supposed prohibitions on sending the data to third parties), Jones threatened to delete station data rather than provide it to us.
Just sent loads of station data to Scott [Rutherford]. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs [McIntyre and Mckitrick] have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone.
Mann replied (apparently alluding to my finding the CENSORED directory):
Yes, we’ve learned our lesson about FTP. We’re going to be very careful in the future what gets put there. Scott really screwed up big time when he established that directory so that Tim could access the data
A few weeks later, Jones made his notorious email to Warwick Hughes (not in the Climategate 1.0 dossier, but known long before Climategate):
Hans Teunisson will reply. He’ll tell you which other people should reply. Hans is “Hans Teunissen” . I should warn you that some data we have we are not supposed top pass on to others. We can pass on the gridded data – which we do. Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. There is IPR to consider.
The email is included as a trailer to a Climategate 2.0 email from Jones to Teunissen (1299- Feb 21, 2005) in which Jones poisoned the well with Teunissen (who never replied to Hughes)
Hans,
This guy pesters me from time to time. I’ve given him your name at WMO/GCOS as someone who’ll reply. We can discuss the merits of my stance in April ! He wants to find fault with some of our station data and by default that the world isn’t warming. If you can just tell him some wmo email addresses that might respond. Why I’m helping him with emails is beyond me ! He wants to discredit what I’ve done. Why the gridded data isn’t good enough is beyond me.
Cheers
Phil
To the extent that Climategate 2.0 emails clarify the issue, in my opinion, they somewhat support Mosher and Fuller’s contention that Jones’ attitude towards data availability changed in response to Mann’s “animus” following publication of McIntyre and McKitrick (2003), though other factors are hardly precluded.
2009
Briefly re-capping 2009 events in light of the above.
In Dr Phil Confidential Agent on August 4, 2009, I had surmised that any governing “restrictions” – if they existed – arose in Jones’ dealings with Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan, and were very limited in application. From the Climategate emails, we know that CRU was closely monitoring Climate Audit and thus were aware of the questions raised in this post.
Nonetheless, in their “small” webpage purporting to respond to FOI requests for confidentiality agreements, East Anglia made the claim (reported above) that they were bound by confidentiality agreements from the 1980s that had been lost during office moves in the late 1980s, a claim that East Anglia resiled from in their November 2009 ruling on my appeal.
A point not discussed at the time, but a loose end: CRU’s misdirection towards the 1980s diverted attention from their unresponsiveness in respect to their communications with Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan in the “mid-to-late 1990s” – communications that would not have been lost in “office moves in the 1980s” and which can hardly have been entirely “verbal”. (The transmission of data, for example, must have been digital.)
On Sep 2, 2009, in my appeal of CRU’s rejection of my FOI request, as noted above, I confronted East Anglia with the inconsistency of their supplying data to the US Department of Energy with their claims that they had entered into binding confidentiality agreements in the 1980s (as well as CRU’s later online publication of CRUTEM1 “cruwlda2”). Jones’ answer in email 2929, as noted above, was both incoherent and unresponsiveness, but did introduce the new claim that the supposed restrictions arose subsequent to the supply of data to DOE i.e. in the mid-to-late 1990s. The latter position was adopted in East Anglia’s refusal of my appeal in November 2009.
Despite all of the above, East Anglia’s representations to the ICO in connection with appeals by Don Keiller and Jonathan Jones reverted to the claim that supply of station data “invariably” was accompanied by a restriction prohibiting the data being sent to a third party. The ICO decision did not address East Anglia’s inconsistent stories, deciding the issue on alternate grounds (that East Anglia had failed to demonstrate an “adverse impact” as set out in the FOI/EIR regulations.)
At this time, CRU and East Anglia have made so many different and inconsistent stories that it is impossible to draw any conclusions on the existence and language of their confidential agreements without a competent investigation to clear the air. It’s too bad that there hasn’t been one.
Finally, although East Anglia conceded to me that there were no “restrictions” arising from agreements from the 1980s and early 1990s, its webpage on data availability stated (and continues to state) the opposite. These untrue statements by the university have contributed to widespread misunderstanding within the climate science “community” of the role of the alleged (and still unseen) “confidentiality agreements” in Jones’ obstruction of both ordinary requests for data and of requests for data under FOI.
Climate Audit 
62 Comments
“Invariably the case that most NMSs”??
That’s like, “It is invariably the case that most days, I am at work on time.” That’s really smooth, UEA.
Thanks again, Steve.
RTF
Links missing first para and last para “World Weather Records”
No links.
All the more reason to impeach Mr Mann’s emails.
…
Oh! what a tangled web we weave
When first we practice to deceive!
A Palmer too! No wonder why
I felt rebuked beneath his eye.
Sir Walter Scott
But, oh, how we improve our style,
Once we have practiced for awhile.
H/t Emily Preyer
=========
http://www.cru.uea.ac.uk/cru/research/grants.htm
The dendroclimatic divergence phenomenon: reassessment of causes and implications for climate reconstruction.
£231,441.00
Jones used the same tactic in 2007 to rebuff French researcher Courtillot:
#5241, Jones to Andrew Wallard, 29/10/2007:
“Andrew,
My French isn’t up to much, but if you go here you can get the gridded temperature data we produce in combination with the Hadley Centre.
[1]http://hadobs.metoffice.com/
then you click on datasets such as HadCRUT3 or CRUTEM3
What I think Courtillot wants is our raw station temperature database. We have entered into agreements to get this data with many National Met Services and also scientists around the world (dating back to the 1980s). We can use the data in the gridded products (see above) but not make the raw station data available. You can get much of the raw data from a web site in the US (NCDC Asheville) if you know what you’re doing.
This isn’t an issue for almost all climate scientists around the world. They are happy with the products we put together. We’ve started getting requests from people in the last few years asking for the raw station data. We’ve always not made the raw data available.
As I said – I am assuming this is what he is talking about.
Cheers
Phil”
Jones’ meassage to Palmer on July 28, 2009 (#1577) is similar:
“Dear All,
Here are a few other thoughts. From looking at Climate Audit every few days, these people are not doing what I would call academic research. Also from looking they will not stop with the data, but will continue to ask for the original unadjusted data (which we don’t have) and then move onto the software used to produce the gridded datasets (the ones we do release). CRU is considered by the climate community as a data centre, but we don’t have any resources to undertake this work. Any work we have done in the past is done on the back of the research grants we get – and has to be well hidden. I’ve discussed this with the main funder (US Dept of Energy) in the past and they are happy about not releasing the original station data.
We are currently trying to do some more work with other datasets, which will get released (as gridded datasets) through the British Atmospheric Data Centre (BADC). This will involve more than just station temperature data.
Perhaps we should consider setting up something like this agreement below
[1]http://badc.nerc.ac.uk/data/surface/met-nerc_agreement.html
I just want these orchestrated requests to stop. I also don’t want to give away years of hard effort within CRU. Many of the agreements were made in the late 1980s and early 1990s and I don’t have copies to hand. I also don’t want to waste my time looking for them. Even if I were to find them all, it is likely that the people we dealt with are no longer in the same positions. These requests over the last 2.5 years have wasted much time for me, others in CRU and for Dave and Michael. Some of you may not know, but the dataset has been sent by someone at the Met Office to McIntyre. The Met Office are trying to find out who did this. I’ve ascertained it most likely came from there, as I’m the only one who knows where the files are here.
See you all later.
Phil”
With Jones’ propensity to state the raw truth on other topics in what he thought were private E-mails, it is hard to establish the “confidentiality agreements from the 1980s” as a fabrication in the CG E-mails.
As I observe in the post, CRU’s stories are inconsistent. If the confidentiality agreements in the 1980s weren’t a fabrication, then they would have applied to the provision of data by CRU to the US Department of Energy. Thus, East Anglia’s claim in November 2009 that the “restrictions” applied only for agreements from the mid-to-late 1990s was fabricated.
One or the other claims is untrue.
I don’t think that you can read a great deal into Jones’ claims about the confidentiality agreements after 2005 – even internally. By his own statement, his recollection of earlier events is not precise and it seems to me that these much later memories are somewhat tainted by the story that’s developed by now. I’d place more weight on contemporary documents from the 1980s and 1990s.
Too bad that there weren’t any adequate investigations.
Steve,
For what it is worth, I do not dispute any of your statements and totally agree that “one or the other claim is untrue.” My guess is that there really were a few confidentiality agreements from the 80s, but that those agreements were (1) probably tangential to raw temp data and had more to do with the coordinates of measuring stations, (2) ignored when the data was supplied to the US DOE, and (3) puffed up and used willy-nilly since then to keep competitors (and auditors!) a step behind.
Steve
As I am sure that you probably already know, if these confidentiality agreements cannot be produced by one party, they can be produced by the other party. If neither of them can, then by law, they are not in force.
The issue of confidentiality agreements is just a red herring.
Whether they exist or not makes no difference, Jones ain’t giving out the data.
With Jones’ propensity to state the raw truth on other topics in what he thought were private E-mails
I’m not sure that is entirely the case. I noticed in this email dated 2/20/2007 that Jones advises correspondents that he has the requested data on disc:
http://di2.nu/foia/foia2011/mail/0715.txt
Yet barely 2 months later in an email dated 4/21/2007 he tells others that he doesn’t have the data.
http://di2.nu/foia/1177158252.txt
So to summarize what I’m seeing…
They blame office moves in the ’80s for losing confidentiality agreements from the ’90s and beyond?
Don’t you hate it when that happens? Like when you got a dog in high school and he ate your homework in 5th grade?
More like the dog you had when you were 5 that died 10 years later eats your homework in college 15 years later.
John, John, John. Johnny John John. John John. John. Please. As Steve has taught us.
That’s climate science.
Oh, and from some of the most repsected professionals in the field.
Some of them need to be dissected and not repsected. Professionally, of course.
Wed Jan 16 09:23:52 2008
I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.
http://di2.nu/foia/1200493432.txt
Steve, you certainly seem to figure into decisions about data release.
“…I took a decision ages ago not to release our station data, mainly because of McIntyre” also seems to imply that he could decide to release it if he wished, and thus was not constrained by any other agreements?
Temporal teleconnections. It sure is easier to not start down that path of publicly lying about stuff…so easy to get tripped up.
1990 seems a long time ago ! By the way, I do have the data from the study on disk! I was wise even when Steve McIntyre first requested the data many years ago. I think I could replicate the study if I had that rare commodity – time.
http://di2.nu/foia/foia2011/mail/0715.txt
He certainly did not want to share data with you, Steve.
If you know the names of the CRU stations and their
data, you can audit this with archived US or Canadian
or Norwegian Temp data independent of CRU. Maybe they
were trying to hide the really small number of stations
used, or the lack of any UHI adjustments (remember in
the small number of stations used in CRUTEM a higher
portion are large cities making UHI blossom), or the
fact that to get a temperature grid of say, Fresno, CA
you have to interpolate among SFX, Reno, Las Vegas
and LAX. Which would be idiotic, right?
Steve:
You would have made a fine attorney. I don’t know what the FOI process is in the the UK, but essentially any legal process requires all parties to act in good faith, which includes being truthful in communications under the process. If you can show (or reasonably assert) bad faith on the part of UEA — such as knowing and willful false submissions, I suspect it will be grounds to reopen the request, notwithstanding any expiration periods provided by the statute.
The argument would be the UEA claimed an exemption to which they were not entitled, given a reasonable interpretation of the facts, and they made knowing and willful false statements to support that exemption. The bad faith response should void any expiration period for appeals and reopen the request (hopefully under some more penetrating supervision).
Because the FOI process is a requirement of national law, at some point someone at UEA becomes liable for at least civil if not criminal penalties. I gather that up to this point your interactions have not gone higher than the FOI compliance officer at UEA. I believe the information in Climagetegate 2.0 provides the basis to identify that officer as a willing participant in the bad faith response.
I’m not an attorney, so I’m certainly not your attorney, and I have no right to suggest how you should spend your time. But I firmly believe that research upon which significant public policy is based needs to be completely transparent, which of course includes access to the raw data, the methods and algorithms, and the code.
Anyway, I really do enjoy your posts. If I ever need a consulting statistician for a mining endeavor, I know where I would go.
I can’t help thinking (though Steve plays this down) that there might have been standard copyright agreements and that these are the “agreements” that CRU are hiding behind.
Also see this extract from an email that PD Jones sent me March 28 2006. It helps augment Steve’s points about the ready flow of data between conutries and groups, that suddenly turned into the wall of silence from CRU.
“We searched the literature in the 1980s, but more importantly searched a number of data archives (proper archives such as libraries), particularly that at the UK Met Office to digitise additional data. We also managed to get some from national met agencies. We continue to do this and add in new datasets as and when we get the data. For example, Canada has a project to homogenize all their station and precipitation (daily and monthly) series. They sent us all the final adjusted series and we include them in our database – about 250 series. We got the data through scientific contacts at meetings in the late 1990s. All the adjusted data are not in the GHCN archive in Asheville, partly because they (Asheville) have never asked for them. If you ask for Canadian data now you get the adjusted data, but only from Environment Canada. NCDC (Asheville) and GISS still use the unadjusted Canadian data. We do the best we can, but we’re not some global archive. I guess NCDC is the official archive as recognized by WMO and GCOS, but they don’t have resources to fund anyone chasing countries for improved past data.”
Some stories are not fully consistent with others.
Does it matter? Too right, it does. As more people access more raw data and examine adjustments, more doubts arise. For example, should past temperature proxy studies be revised by addition of later temperature data, which is much more level in trend? The proxy reconstruction might change with modern data added. Therefore the extent of global warming might be changed and a whole industry based on global warming might look suspect.
RE: “I can’t help thinking (though Steve plays this down) that there might have been standard copyright agreements and that these are the “agreements” that CRU are hiding behind.”
As I posted in the previous thread, Mr. Palmer first suggested using Copyright as a way to avoid FOI/EIR from Mr. McIntyre with the hope of avoiding the expected appeal.
#0780
http://foia2011.org/index.php?id=730
Dave says (edited):
‘Copyright’ in the contents of a database would require some personal creative input by ourselves to the data or database that would render it different from preceding external versions and ‘original’.
To which Phil responds, “Piece of cake.” or words to that effect:
“Your idea about ‘copyright’ of the database is an excellent one. Over the last 30 years we have reworked the database on a number of occasions – every 3-5 years, so re-extending for another 15 years is fine.”
I don’t believe copyright was ever established to be the ‘agreements’ they could hide behind . . . but then again I haven’t gone thru all the CG2 emails on this front.
Matt,
If the raw temperature data would support your claims, would you not gladly hand it over?
They did hand it over, but to the US DOE and not to Steve. Suppose that there are/were a few countries that only have temperature stations at government facilities, and those countries were worried about their adversaries knowing the exact latitude and longitude of those facilities. Back in the 80s that stuff mattered; now you can look on Google maps. Those countries would be OK with releasing gridded data that lacks specific coordinates, but would want the raw data with coordinates kept close. Right or wrong, Jones might have felt he could slip the data surreptitiously to the DOE to fulfill a grant requirement (perhaps envisioning something akin to the warehouse at the end of Raiders of the Lost Ark…”if you know where to look”), but giving it to other researchers such as Steve would be like splashing it across the internet. If so, the rationale for not giving it away most likely morphed over time from legitimate concern of repercussions, to defending the edifice. I am only trying to understand, not defend, this behavior. I think if there were major skeletons in this closet there would be snark about it in the CG E-mails, but I can think of reasons why that might not be right either.
Matt,
So let’s get this right…..some countries were depending upon Phil Jones and UEA
to guard the whereabouts of their most cherished………..weather stations?
the security concern that someone MIGHT express would be this.
1. we dont want to give people historical weather data because that could
make attacking us easier ( weather planning)
2. weather stations are located at airports in some cases and we want to
keep the coordinates of instruments secret
That MIGHT be what somebody would argue. However, the locations of airports are well known. The FAA used to publish a public database until the war on terror.
So, somebody might have made this security argument. ….However,
Nobody but commenters and defenders of the indefensible on blogs would suggest, absent evidence, that countries did IN FACT require these agreements for this reason.
arm waving.
I suppose that when I offer possible scenarios in blog posts, sooner or later someone will accuse me of storytelling and arm waving, which is not much fun. Guess I will find a different approach for expressing scenario-based logic. It is simply this: weather data is often produced by government entities. Governments routinely apply blanket nondisclosure policies to information produced by government offices. The reason they do that is because they do not want worker bees making the decision whether something is sensitive or not. After awhile, these things become routine and no one thinks about them much. Old meaningless forms pile up and get thrown away. Now I am not making this up, OK? I am applying personal experience in interpreting what Jones said in the second block quote in Steve’s original post (and yes, “invariably” is overwrought and inaccurate, as Geoff S. has confirmed above). I do not fault Steve at all for following the law, but I do believe that he unintentionally put Jones in a difficult situation that I would not want to be in myself. There is plenty of evidence of reprehensible behavior by Jones, I just don’t see this nondisclosure stuff adding much to that. Geoff S wrote:
“There remains as well the fundamental question of whether CRU or Jones had the authority to withhold or release much of the data, given that it was not “owned” by them, but merely held in custody by the good graces of the real owners/compilers.”
None of us know the answer to Geoff’s question. What Jones should have done was inform Palmer (and Hughes, and Courtillot, and Steve) that he had no idea whether he had the authority to release the raw data because various agreements and policies made it a muddled mess, and just let Palmer do his job.
Expect they can’t produce these ‘confidently’ agreements that stop they releasing the data , in fact they had be releasing the same data to other but only those they ‘approved off’ , in other words it was pure BS from Jones used to cover his poor behavior and who can forget the comedy gold words “why should I send you the data if only want to find something wrong with it “
As I understand it, the problem wasn’t that station data wasn’t available, but that non team members didn’t know which stations were used in crutem, or which time frames. Were the entire series for each available site used? That’s why this is important.
There is no evidence that Jones concern about giving the data to steve was related to fear that steve would re distribute it. Were that his concern he could have addressed it by having steve sign a agreement not to re distribute.
Were that his concern you would expect him to voice that concern in other mails.
Were that his concern he would not have shared data with rutherford and mann who managed to leave other data in the clear for folks to take. Were that his concern he would have taken more care at CRU and not left data laying around on the ftp site. There is no evidence that it was his concern. You are making things up out of whole cloth to obscure a much more plausible explanation that has documentary evidence to support it.
Jones shared the data up until feb 21 2005. In his mail to mcIntyre in 2002 he shows that he was aware of some countries restrictions and not persuaded by their arguments.
Jones was persuaded, it seems to me, by the example mann set. An example covered in the press at the very time and on the very day he decided to thwart warwick hughes request for data: feb 21 2005.
MM 2005 made their life difficult. Published just as the ZOD was being worked on it was clear that briffa’s job would be made more difficult. Jones joined the fight against Mac full force and from all angles.
Warwick Hughes’ requests started in 2004 and Jones obstructed them. Consider the under-commented incident in early 2004 when, as a reviewer of a submission by Mann et al to Schneider’s CLimatic Change, I asked for data and code in order to do the review. This caused a commotion among the editorial board of Climatic Change – later misrepresented by Schneider in his book. Jones and Santer were on the editorial board of Climatic Change and worked hard to oppose Climatic Change establishing policies that required authors to provide data and especially code (coordinating confidentially with Mann).
Added to the database. It was clearly a habit for you to ask for data and code as a reviewer. That this alone:
1. marked you out as different from everyone else
2. was considered dangerous
3. was fought by such underhand means at the editorial board level
says everything one could ever want to know about climate science.
Excellent point, Richard.
Viewed from their vantage point, their thought process, on learning of Steve’s requests for data and code, was (perhaps in rapid succession):
1. McIntyre is asking for the data and the code.
2. This is different from everyone else.
3. This is dangerous.
4. This must be prevented by any means available.
Perhaps not unheard of in research.
But, when you add factor in the presence of government backing of the scientists, with all the implicit reserve powers that go along with that … you have the makings of something especially unsettling.
RTF
What blows me away is the insouciance with which they admit to just ‘losing’ stuff all the time – data, confidentiality agreements – did Phil have to buy a new coffee mug every week as well?
What a bunch of cowboys. I have worked in several establishments where legal agreements with individuals and organisations are used, and they are logged, and kept, in a safe in the legal department if there is one. If there isn’t a stand alone legal unit, they are logged, and kept, in a safe by a very senior executive. Quite likely copies would also be retained by the organisation’s legal advisers as a backup.
Things do get mislaid during office moves, but ‘lost’ usually means ‘thrown out’. And, there is no way that legal agreements, or critical data, would just be ‘lost’ without anyone noticing and then moving heaven and earth to find it asap.
Apart from Steve’s incisive question about which of their statements was true in this instance – as they can’t both be – more generally, there is a pattern of sloppiness and indifference to record keeping that permeates the whole sorry history of Jones’ tenure at UEA. And, not only does he not care about the integrity of his unit’s records, the University seems to share his lack of concern about such pesky and trivial matters. It is just mind-bogglingly unprofessional, and highlights that UEA as a whole contributed to, indeed hosted, the culture which made this egregious behaviour acceptable and commonplace.
This suggests to me that even the awful attitude to Steve McIntyre etc can (in part – there is more) be seen as an extension of this pattern of sloppy record-keeping. Not bothering to check Steve’s genuine expertise and the formidable backup of expertise available at CA, and accepting without check, the AGW labelling of all challengers as in the paypockets of Big XX.
Wherever I’ve worked the legal department keep the legal agreements, which have to be signed by both sides. There may be legal agreements but I suspect it was an informal agreement not to pass on the data, with no real legal agreement to support it.
One of the problems is that CRU’s answer to a question sometimes depends on who is asking. UEA appears to have had policies that prohibit unauthorized staff from entering into confidentiality agreements – such agreements had to be signed by specialist university administration personnel. When these specialists were asked (see a post about two posts ago), Jones told them that his agreements with met organizations weren’t actual confidentiality agreements.
However, when refusing FOI requests, CRU took an entirely inconsistent position – that they had entered into binding agreement and that failure to adhere to these agreements would impact international relations.
Had there been an actual “investigation” at the University of East Anglia, some facts might have been established.
I’ve not seen email #3465 mentioned, but it discusses the merits of ‘exempted information’ under section 27 of the UK act regarding “international relations”.
Specifically 27.3 appears to exempt “confidential” information if
“…the circumstances in which it [confidential information] was obtained make it reasonable for the State, organisation or court to expect that it will be so held.”
-Thus if the recepients think the circumstances justified an “expectation” of confidentiality then that is good enough. No written agreements necessary?
It’s also an interesting email in that it reveals Met Office thinking as to how they can claim that they do not “hold” information that they do physically hold, but do not “own”.
Steve
Looking back through the late 2004-2005 mails, especially feb 2005 might yield
folks some interesting insights into jones /mann
I looked for just a second and lots was going on during that critical period.
Steven (and Steve),
Not quite between Jones and Mann, but have you read 1667? David Ritson basically rips the team a new one, aiming specifically for The Hockey Stick, but also has some specific criticism of “M+M” (he thinks the short calibration period was a red herring and there were far bigger problems). Unfortunately I lack the stats background to fully understand it all myself, but I am sure you guys will have no problem. I liked this ending:
“Frankly I am apalled by an apparent poverty of mechanisms in the climate field
to resolve such problems, or alternatively to classify them as irresolvable?”
I also note that emeritus physics professors from Stanford don’t get the same treatment as junior dendrochronologists!
There remains as well the fundamental question of whether CRU or Jones had the authority to withhold or release much of the data, given that it was not “owned” by them, but merely held in custody by the good graces of the real owners/compilers.
The potential for big damage comes when the data held in custody are adjusted in ways unstated.
Suppose I were to have someone’s property (data) with use rights. Suppose there is a confidentiality / nondisclosure agreement. As long as I somehow derive a second massaged data set and don’t disclose how it was done (so the original data can’t be reconstructed from the derived set), I can destroy the original data and not worry about protecting it. The massaged set is my IP.
I’ve seen a case where there was no initial claim to confidentiality, but the university later had their lawyers send a letter claiming confidentiality and to cease use. The IP in question was cleaned off of all databases and any physical representations (this was a design) were destroyed at significant cost.
Even coming up with plausible scenarios (we’re just really confused and disorganized, we were forced to wipe the data, etc), the FOI responses seem bogus.
Steven,
You are the first to mention redistribution. I’m not sure who you are arguing against.
At this point I am confused as to who asked for the truly raw data, as opposed to the adjusted data, and which statements refer to which. It only matters when trying to deconvolute the E-mails.
Assuming the “you”= me, I am not trying to obscure anything, nor am I disputing anything, I just have a different opinion. I think Jones really did have fuzzy agreements against disclosure, but under the harsh light of day, they were not legally binding, putting him in a difficult situation when it came to a very public release to Steve. I do agree that much of the stonewalling, especially later in the game, stemmed from a simple vendetta against Steve.
Matt, If it helps, I have year 2006 letters from both UEA and Australia’s Bureau of Meteorology that leave Australia out of your hypothetical about the prior existence of agreements. So that is one fairly large data set for which there were no agreements (apart from ordinary copyright in recent years). The BOM letter can also be construed as meaning that there were no secondary agreements allowing the adjustment & distribution of data as covered by copyright, or at all.
Part of the problem is finding out from CRU what they call “raw data” as opposed to “adjusted” “homogenised” “gridded” “interpolated” and so on. There seems to be an answer somewhere, but it’s a trial to find it. It remains possible that CRU are confused about similar classifications of data received from various countries, leading to some corrections being done more than once.
Given the importance placed on global temperature, the housekeeping of data is a global disgrace and i, for one, simply do not trust it as a basis for further analysis.
Can you tell me of every prior adjustment to GISS data in the form used by authors today?
Can you mount a case for continued non-disclosure?
Can you agree that FOI was abused?
Can you tell is if climate science was improved by non-disclosure?
Geoff,
For what my opinions are worth, data and code behind published papers should always be free and available, and paper reviewers should routinely request it whether they use it or not. On the other hand, I think most folks would agree that if one has spent years painstakingly negotiating with various entities to build up a database and plan to publish on it someday, the question of whether it is fair for someone to say “now that you have it all give it to me” is more complicated (disclaimer: hypothetical situation specific to Geoff’s general question!). I believe there is a “working papers” exclusion in FOI for that, but am not an expert. I lack the knowledge to answer your questions directly.
Sorry but couldn’t resist. From BBC Live Football:
“ 2150: GOALFLASH Newcastle 3-0 Man Utd (Phil Jones OG)
Phil Jones is in two minds as to whether to deal with it or let it bounce …. Talk about soft. “
His excuse is quite simple – only 19 years old, and it was windy.
Dear Admin, it’s urgent. Please get in touch with me via email as soon as possible – thanks!
Steve: The “legal agreements” that CRU posted don’t look much like legal agreements I have seen. Hopefully one of your readers is a real attorney who will take the time to comment on them. A binding legal agreement would normally be signed by an authorized official from CRU or the UAE and state specifically what CRU promised to do or not do with the data. It is no wonder that CRU lost most of these “agreements” and chose to ignore them when convenient by sending data to other favored academics and the US Dept of Energy.
The letter to the UKMO signed by Hulme says: “Please let me know about the availability and terms of supplying the above data.” The form from the UKMO/NERC states their terms, but there is no signature from CRU or the UEA on this form to indicate that they agreed to the terms they received. This is not surprising since Hulme’s letter is dated 1994 and the NERC form was “last modified in 2003”! (These environmental records presumably are now subject to EIR.)
The document in Spanish doesn’t clearly say what CRU agreed to do and not do with the data. “El abajo firmante se compromete a utilizar la informacion requerida para 10s fines especificado en este solicitud y citara siempre el origen de la misma en cualquier publicacion o trabajo en que se use.” Google translation: “The undersigned agrees to use the information required to 10s purposes specified in this application and always cite the source of it in publication or any work that is used.” If the term “10s purposes” were defined in this document or on an attached page and if there was a signature from CRU, a legally binding agreement would exist.
The documents from Bahrain and Norway make a request of CRU, but weren’t signed by anyone from CRU to show that they agreed with the terms.
Matt, you need more reading. For example, who dominated who?
Climategate 3341. 3 Dec 2008.
When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions – one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school – the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian – who deals with appeals. The VC is also aware of what is going on –
Phil Jones to Tom Wigley, CRU.
Can someone please help me?
I have a file called “all.zip” which I put in a folder I named “CRU-all”. It has folders numbered from 1 to 98, though not all numbers are represented. Each folder has numbered files of temperature for different periods but none later than October 2009.
I seem to recall that at one stage the CRU put online the data from countries for which they had no confidentiality agreement. The problem is I can’t remember where I got it from. Does anyone recognise it?
There’s an all.7z zip file in FOIA 2011 and it’s the folder with 250,000 emails nobody so far has opened as it apparently needed a pass phrase. If that’s what you’ve got, man, you have hot material.
http://www.metoffice.gov.uk/research/climate/climate-monitoring/land-and-atmosphere/surface-station-records ?
Steve Mosher re reasons for requiring confidentiality agreements.
I believe the reason for requiring such agreements is much simpler. Many meteorological services (incluing the UK Met Office) charge for supplying data and realise that allowing it to be put on line would end that lucrative market. In many cases the charges can be hundreds of pounds for a decade or so of monthly data for 4 or 5 parameters.
JONESSH.DAT is uncorrected, but is the gridded SHEM1990.ANO corrected? NDP022R2.DAT seems to include marine data, is the land-only result somewhere?
If they insist on using the “dog ate my homework” defense, examine the dog. Request a copy of the non-disclosure agreement from the data source. If neither party to the nondisclosure agreement can produce the document, then move to compel in court.
Steve,
For completeness, I think there’s yet another (!) version of Jones’ description of NMS/confidentiality agreements that I don’t see in this post, i.e., Keenan’s May 2007 correspondence with Jones.
I just saw it while reading your 2008 summary of CRU correspondence: https://climateaudit.files.wordpress.com/2008/05/cru.correspondence.pdf
———
May 25, 2007, Keenan Letter to Jones
According to Steve McIntyre, some of your raw station data was obtained from National Met Services who asked you to not disclose that data. Is this correct? If so, will you send me a list of the countries with which CRU/UEA has such non-disclosure
agreements? Or, if it is easier, direct me to an appropriate person to ask for this?
———
May 31, 2007 Jones Reply to Keenan
Dear Doug,
I have done some searching in files – all from the period 1990-1998. This is the time when we were in contact with a number of NMSs. We have also got datasets from fellow scientists and other institutes around the world. All supplied data (eventually and sometimes at cost), but we were asked not to pass on the raw data to third parties, but we could use the data to develop products (our datasets) and use the data in scientific papers.
It is likely that some of the NMSs and Institutes have changed their policies now – and that the people we were corresponding with (all by regular mail or fax) are no longer there or are in different sections. The lists below don’t refer to all the stations within these countries, nor to all periods, but to some of the data for some of the time.
The NMSs
Germany, Bahrain, Oman, Algeria, Japan, Slovakia and Syria
Scientists/Institutes (data for these countries)
Mali, India, Pakistan, Poland, Indonesia, Democratic Republic of the Congo (was Zaire), Sudan and some Caribbean Islands
These are the only ones I can find evidence for. I’m sure there were a few others during
the 1980s, but we have moved buildings twice since 1980.
Not sure how you will use this data.
Phil Jones
———
One Trackback
[…] Dr Phil, Confidential Agent: Re-visited […]