In today’s post, I continue my re-appraisal of various untrue statements made by the University of East Anglia in order to avoid disclosure of CRUTEM station data. I do not consider motives at this time. Also see preceding posts here, here.
In East Anglia’s response to July 2009 FOI requests for alleged confidentiality agreements (here) , CRU stated that, since the 1980s, they had entered into confidentiality agrements that prohibited them from providing station data to third parties, but were unable to “locate” any such agreements noting that they had “moved offices several times during the 1980s”:
Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties. Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s. Some date back at least 20 years. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period. In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.
In East Anglia’s submission opposing appeals by Jonathan Jones and Don Keiller (e.g. here), they made the even stronger claim that national meteorological services(NMSs) “invariably” released information only under licences that prohibited transfer to third parties:
It was invariably the case that most NMSs only released information under licences, both written and verbal, that prohibited the further transfer of the information. There was no standard form for such licences but they were all similar in that they prohibit the onward transmission of the data to third parties.
CRU’s longstanding relationship with the US Department of Energy and its transmission of station data to the US Department of Energy (which placed CRU’s station data online (e.g. here) is inconsistent with their claims that they entered into binding confidentiality agreements in the 1980s. In my 2009 appeal, I confronted East Anglia with this relationship (a matter that they discussed in email 2929).
Their refusal of my appeal was dated Nov 12, 2009 (one day before the last email in the Climategate dossier) and was sent to me on Nov 18, 2009. While the dates of these refusals have attracted some commentary, relatively little attention has been paid to the content of the refusal (reported at CA on Nov 21, 2009 here) in which East Anglia stated that the “restrictions” applying to the station data only arose after CRU provided data to the US Department of Energy (along the lines of a scenario that I contemplated in a CA post of Aug 4, 2009 entitled Dr Phil, Confidential Agent):
In regards the information provided to the US Department of Energy, my investigation has revealed that this was done in the early 1990s prior to the imposition of the restrictions now pertaining to the data pursuant to a contractual obligation at the time. Therefore, the analogy you are drawing does not apply to the data that is the subject of this request.
Despite this explicit admission, CRU’s claims that they have been subject to longstanding confidentiality agreements have been accepted by Nature and the climate science community, which has, in turn, denounced CRU critics for not believing in the existence of actionable CRU confidentiality agreements from the 1980s.
In preparing today’s post, I considered not merely the (limited but relevant) new information from Climategate 2.0, but also re-examined contemporary (1980s, 1990s) statements about provenance of station data. Prior to 2004, there is little to no evidence of the commitment to secrecy said by East Anglia to have been “invariably the case”. Quite the contrary. Prior to 2004, there is considerable evidence of a longstanding practice of making station data publicly available (both by GHCN and CRU).
World Weather Records
World Weather Records was (and is) a longstanding international effort in which station data (and metadata) have been published and, when the technology became available, digitally archived.
TD9644 , published in 1991, described its origin as follows:
In 1923, the International Meteorological Conference Committee, convened in Utrecht, Netherlands offered initial justification for the creation of WWR:
“…the conference thinks that publication of long and homogeneous series of observations in the form of monthly means of pressure, temperature and rainfall would be of the highest importance for the study of the general circulation of the atmosphere.”
In response to the Conference’s findings, the Smithsonian Institution of the United States sponsored the first series of WWR, edited by H. Helm Clayton and published in a single volume (1196 pages) in 1927. The publication included full period of record through 1920 of monthly means of temperature, pressure (station and sea-level) and precipitation for selected global stations.
According to TD9644, data was publicly available in 1991:
All available data are both published in the volume sets and digitally archived through NCDC’s tape library under DSI-9644.
Original World Weather Records report (some of which have been scanned and are available online) commendably describe the provenance of each record (something later said by CRU to be impracticable), e.g. the following:
The station names, elevations, and coordinates given in the table headings were supplied by the General Directory of Meteorology of Uruguay and were in use at the end of 1970.
World Weather Records continue to be published and are online at UCAR here.
Jones et al 1985-1991 and CRUTEM1 (1994)
Jones’ original compilation of station data, funded by the US Department of Energy, was described in three technical reports: Bradley et al 1985 (out of print and not online other than an excerpt here), TR022 (Northern Hemisphere) online here and TR027 Southern Hemisphere (online here), and two less-informative academic articles. The original version of NH data (ndp012) is no longer online, but a 1991 update is online here.
These documents indicate almost total reliance on prior World Weather Records compilations, plus archives at the UK Met Office, all of which had been made publicly available (contradicting later CRU claims of a longstanding practice of confidentiality agerements.) Jones et al 1985 (TR022) described the provenance of Northern Hemisphere station data as follows:
Most studies of global or hemispheric temperature fluctuations have relied upon the compilations of station data in World Weather Records… Bradley et al (1985) have added considerably to the WWR data using material available in published and manuscript form in meteorological archives, particularly those of the UK Meteorological Office… Full details of these improvements in station coverage are given in Bradley et al 1985. The most important improvements in coverage occur over parts of the Soviet Union and northern Europe particularly before 1881…Further improvements in coverage have also been made for the twentieth century particularly over northern Africa before 1940 and over the Peoples Republic of China…Details of each of the 2666 stations in the data bank are documented in Appendices using the formats described in Goodess et al 1985 and Bradley et al 1985.
TR027 provided similar statements about Southern Hemisphere data. It reported the compilation of 610 station records of which 293 were used in the grid. Only one meteorological service (Peru) was mentioned. In passing, two of the authors cited (Pittock 1980; Salinger 1981) turn up many years later as leaders of the campaign to censure Climate Research for the publication of Soon and Baliunas 2003.
The basic source of station air temperature data for the Southern Hemisphere land masses is the set of volumes of World Weather Records (WWR) (Smithsonian Institution, 1927, 1934, 1947, and U.S Weather Bureau, 1959–1982; available in digitized form from the National Center for Atmospheric Research (NCAR), Jenne, 1975). A considerable amount of additional temperature data for Argentina and Chile for the years 1931-60 has recently been added to this set. In WWR, these countries only have data available from 1951 (see Pittock, 1980, for further details).
Searches for data in archives as part of the present project yielded additional data for Indonesia and Australia and for some Pacific Islands, particularly Tahiti. Additional data for New Zealand was found in Salinger (1981). For Peru, the Peruvian Meteorological Service supplied information for about 10 stations covering the 1940s and 1950s. Additional data for Australia was provided by their Bureau of Meteorology. All of these sources are gratefully acknowledged.
In 1991, an update to the gridded data set (ndp020r1) re-iterated this description of provenance of station data, noting that all station data was available online.
The primary sources of these data are the World Weather Records (WWR), published by the Smithsonian Institution and the U.S. Weather Bureau, the archives of the United Kingdom Meteorological Office, and the Monthly Climatic Data for the World, published by the National Climatic Data Center (Asheville, North Carolina). Additional sources are described in Bradley et al. (1985) and in Jones et al. (1985 – TR022, 1986a – JCAM, 1986c – TR027, 1986d – JACM). The present updated version of this data set is identical to the earlier version (Jones et al. 1986b) for all records from 1851 through 1978. For the period 1979-1984, the present data set corrects erroneous data using satellite data for some sites and appends data for other sites by adding previously unavailable station data (Jones et al. 1988). The present package also adds monthly surface air temperature anomalies for the period 1985-1990, Antarctic monthly surface air temperature anomalies for the period 1957-1990, as well as the monthly mean temperature records for individual stations (Antarctic stations excluded) that were used to generate the set of gridded anomalies. Individual station data for the Antarctic (stations south of 62.5S) are not presented in this package but are given in Jones and Limbert (1989 -NDP032) and may be obtained free of charge from the Carbon Dioxide Information Analysis Center.
In Climategate 2.0 email 2929 (Sept 8, 2009), Jones said that it was a “requirement” of CRU’s contract with DOE that the data be placed online:
I don’t recall all the facts from that long ago. There is or was a version on a US Dept of Energy website from about 1990. This was a contract requirement at the time.
In 1994, Jones expanded the 1873-station data set to 2891 stations, described in Jones et al 1994 (later CRUTEM1), ascribing the provenance of new data primarily to the projects described in Karl et al 1993 (all of which had been placed online.)
In 1996, CRU placed the CRUTEM1 station data online (/projects/advance10k/cruwlda2.zip), where it remained online until CRU’s removal of the data from its website on or about July 29-30, 2009 (4270). Peterson and Vose (1997), in an article introducing the second version of GHCN, described the 2891-station CRUTEM1 dataset as “widely used”. Doug Hoyt reported that he had downloaded the file in 1999. Simister (2002) and Simister and van de Vliert (2005) report that they downloaded the data in November 2001. I downloaded the file cruwlda2 as late as July 25, 2009 (an event discussed by CRU in Climategate 2.0 4270). In 2002, Jones sent me (then unknown) the cruwlda2 dataset, mentioning that an updated dataset would be published early in 2003.
Jones et al 1999 continued use of CRUTEM1:
Here we use the land station data set developed by Jones . All 2891 station time series used have been assessed for homogeneity by subjective interstation comparisons performed on a local basis. Many stations were adjusted and some omitted because of anomalous warming trends and/or numerous nonclimatic jumps (complete details are given by Jones et al. [1985, 1986c]).
In 2009, Jones told (2929) East Anglia administrators that the cruwlda2 version wasn’t “complete” and that it was placed on an ftp site because it was “easier” to do this than to send out disks. (Jones did not explain how this related to compliance with supposed “confidentiality” agreements.)
The 1996 version (cruwlda2) wasn’t a complete version and was something we developed for a number of people in EU projects to use. We made these available to people on these projects via our ftp site, as it was easier to do this than sending disks at that time (email attachments were smaller then).
Whether the data was provided by email or ftp was obviously irrelevant to compliance with alleged confidentiality agreements. In addition, contrary to Jones’ claim to his administration, the 1996 version (CRUTEM1 or cruwlda2) was complete.
Important light is shed on practices of the period in articles introducing the two GHCN versions.
GHCN-v1 was introduced in 1992 and was described in Vose et al 1992 NDP041. Interestingly, the 1891-station Jones network (ndp020r1) was incorporated as one of the sources for GHCN. ( The “raw” GHCN dataset lists several scribal versions. I’ve looked at a few stations in GHCN-2 and can trace one of the scribal versions to the CRU version. There’s an interesting twist here that I’ll discuss on another occasion.)
In 1997, Peterson and Vose released a second version of GHCN (the one in use until recently). Contrary to the regime of secrecy described by CRU, Peterson and Vose (1997) said that national meteorological organizations were “cooperative and enthusiastic” about contributing data to GHCN:
Because numerous institutions operate weather stations and because no single repository archives all of the data for all stations, we employed five acquisition strategies to maximize the available pool of data: 1) contacting data centers, 2) exploiting personal contacts, 3) tapping related projects, 4) conducting literature searches, and 5) distributing miscellaneous requests. In general, most parties were cooperative and enthusiastic about donating their data to the GHCN initiative, particularly since GHCN is a World Meteorological Organization (WMO) Global Baseline Data Set. As a result, GHCN version 2 contains data from 31 diverse sources (Table 1).
Whereas “personal contacts” between CRU and meteorological organizations were said to be fraught with secrecy, Peterson and Vose (1997) reported that their “personal contacts” had resulted in fresh contributions to public archives:
We also exploited personal contacts by contacting colleagues in the search for potential data sources. For example, scientists who visit or work in conjunction with the authors’ respective institutions often either have data themselves or are able to facilitate the acquisition of data from another party (e.g., by putting the authors in contact with potential sources). This was another extremely productive means of acquiring data, which yielded approximately 10 new datasets.
They reported that these efforts had yielded a much larger data set than the “widely used Jones (1994 “2961-station” dataset (a comment that further confirms the non-secrecy of the Jones 1994 station dataset.
With 7280 stations, GHCN is over twice as large as the widely used Jones (1994) 2961-station mean temperature dataset.
CRUTEM2 (Jones and Moberg 2003)
As noted above, in East Anglia’s rejection of my appeal in November 2009, they said that “restrictions” applying to the data arose after the data had been sent to the US Department of Energy. Here East Anglia was attempting to rationalize Jones’ admission (2929), in which, after admitting that he had sent station data to the US Department of Energy under a “contractual requirement”, claimed that the “restrictions” arose in the mid-to-late 1990s:
There is or was a version on a US Dept of Energy website from about 1990. This was a contract requirement at the time. Much extra data has been added since then, and this is what the restrictions refer to from the mid-to-late 1990s.
Needless to say, this position is totally inconsistent with their previous claim that they had entered into agreements in the 1980s and had lost the agreements during office moves during the 1980s.
It also raises other interesting issues that have not been examined to date: if the only relevant “restrictions” arose in the mid-to-late 1990s, this is well after the “office moves” of the 1980s. It is implausible, to say the least, that CRU should have no record of any communications with NMSs in the late 1990s relating to the confidentiality of station data. It’s hard to believe that data arising from such agreements would not have been transmitted electronically.
Jones and Moberg 2003, which introduced CRUTEM2, briefly discussed CRU’s contact with NMSs (discussed in a prescient August 4, 2009 CA post Dr Phil, Confidential Agent). Jones and Moberg 2003 gave no indication of that it had commenced a new practice of using confidential data. It complimented meteorological organizations for improvements in the quality and quantity of data exchanged and made available “for all to use”:
The majority of the world’s countries have endorsed the initiatives of the Global Climate Observing System (GCOS) to improve the quality and quantity of monthly climate data routinely exchanged. …
The second reason is that several national and other initiatives have also dramatically improved the quantity and quality of monthly mean temperature data available. Several countries have extensively homogenized their entire national holdings, releasing the results for all to use. CRU over the last eight years has also received several national and other temperature datasets.
Jones and Moberg 2003 listed these other datasets without mentioning any confidentiality issues. Whereas earlier editions of CRUTEM had not mentioned direct CRU contact with NMSs, Jones and Moberg reported that CRU had obtained station data through direct contact with some difficult or then pariah regimes: Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan.
CRU has collected a number of temperature records through direct contacts with the NMSs in Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan. Many of these records cover only the period 1961–90, but others extend over the entire twentieth century.
Jones and Moberg (2003) also reported the acquisition of data from less colorful regimes (that presumably attorned to WMO Resolution 40): NORDKLIM data from Denmark, Sweden, Finland, Norway, Iceland, Ireland, the Netherlands, Belgium and the UK itself, as well as additional data from Canada and Australia.
Accordingly, despite the elaborate charade by CRU and the UK Met Office purporting to seek consent from every NMS in the world, there were only a relatively small number of countries (Algeria, Croatia, Iran, Israel, South Africa, Syria and Taiwan) from which “restrictions” from the “mid-to-late 1990s” could have arisen.
Mosher and Fuller (CRUTape Letters) concluded that Jones’ attitude towards provision of data was transformed by Mann’s unrelenting “animus”. They state:
Jones, guided by Mann’s example of animus toward McIntyre and others who question the work of climate scientists, and guided by Mann’s relentless appeals to motive is transformed from a researcher who once promised to share data even in cases where it appeared there might be legal cause to withhold it, to a researcher who cares first about motive, and second about his own reputation and who in the end will use every legal and bureaucratic means to obstruct the release of temperature data and threaten its destruction.
Climategate 1.0 (and especially) 2.0 emails contain an enormous dossier of spite following the publication of MMM2003 in October 2003, exacerbated by the publication of our MM2005a,b in late January 2005. Jones’ emails, as Mosher and Fuller observed, now become even more partisan. In an email in which Jones reported the sending of station data to Mann and Rutherford (despite supposed prohibitions on sending the data to third parties), Jones threatened to delete station data rather than provide it to us.
Just sent loads of station data to Scott [Rutherford]. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs [McIntyre and Mckitrick] have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone.
Mann replied (apparently alluding to my finding the CENSORED directory):
Yes, we’ve learned our lesson about FTP. We’re going to be very careful in the future what gets put there. Scott really screwed up big time when he established that directory so that Tim could access the data
A few weeks later, Jones made his notorious email to Warwick Hughes (not in the Climategate 1.0 dossier, but known long before Climategate):
Hans Teunisson will reply. He’ll tell you which other people should reply. Hans is “Hans Teunissen” . I should warn you that some data we have we are not supposed top pass on to others. We can pass on the gridded data – which we do. Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. There is IPR to consider.
The email is included as a trailer to a Climategate 2.0 email from Jones to Teunissen (1299- Feb 21, 2005) in which Jones poisoned the well with Teunissen (who never replied to Hughes)
This guy pesters me from time to time. I’ve given him your name at WMO/GCOS as someone who’ll reply. We can discuss the merits of my stance in April ! He wants to find fault with some of our station data and by default that the world isn’t warming. If you can just tell him some wmo email addresses that might respond. Why I’m helping him with emails is beyond me ! He wants to discredit what I’ve done. Why the gridded data isn’t good enough is beyond me.
To the extent that Climategate 2.0 emails clarify the issue, in my opinion, they somewhat support Mosher and Fuller’s contention that Jones’ attitude towards data availability changed in response to Mann’s “animus” following publication of McIntyre and McKitrick (2003), though other factors are hardly precluded.
Briefly re-capping 2009 events in light of the above.
In Dr Phil Confidential Agent on August 4, 2009, I had surmised that any governing “restrictions” – if they existed – arose in Jones’ dealings with Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan, and were very limited in application. From the Climategate emails, we know that CRU was closely monitoring Climate Audit and thus were aware of the questions raised in this post.
Nonetheless, in their “small” webpage purporting to respond to FOI requests for confidentiality agreements, East Anglia made the claim (reported above) that they were bound by confidentiality agreements from the 1980s that had been lost during office moves in the late 1980s, a claim that East Anglia resiled from in their November 2009 ruling on my appeal.
A point not discussed at the time, but a loose end: CRU’s misdirection towards the 1980s diverted attention from their unresponsiveness in respect to their communications with Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan in the “mid-to-late 1990s” – communications that would not have been lost in “office moves in the 1980s” and which can hardly have been entirely “verbal”. (The transmission of data, for example, must have been digital.)
On Sep 2, 2009, in my appeal of CRU’s rejection of my FOI request, as noted above, I confronted East Anglia with the inconsistency of their supplying data to the US Department of Energy with their claims that they had entered into binding confidentiality agreements in the 1980s (as well as CRU’s later online publication of CRUTEM1 “cruwlda2”). Jones’ answer in email 2929, as noted above, was both incoherent and unresponsiveness, but did introduce the new claim that the supposed restrictions arose subsequent to the supply of data to DOE i.e. in the mid-to-late 1990s. The latter position was adopted in East Anglia’s refusal of my appeal in November 2009.
Despite all of the above, East Anglia’s representations to the ICO in connection with appeals by Don Keiller and Jonathan Jones reverted to the claim that supply of station data “invariably” was accompanied by a restriction prohibiting the data being sent to a third party. The ICO decision did not address East Anglia’s inconsistent stories, deciding the issue on alternate grounds (that East Anglia had failed to demonstrate an “adverse impact” as set out in the FOI/EIR regulations.)
At this time, CRU and East Anglia have made so many different and inconsistent stories that it is impossible to draw any conclusions on the existence and language of their confidential agreements without a competent investigation to clear the air. It’s too bad that there hasn’t been one.
Finally, although East Anglia conceded to me that there were no “restrictions” arising from agreements from the 1980s and early 1990s, its webpage on data availability stated (and continues to state) the opposite. These untrue statements by the university have contributed to widespread misunderstanding within the climate science “community” of the role of the alleged (and still unseen) “confidentiality agreements” in Jones’ obstruction of both ordinary requests for data and of requests for data under FOI.