Recently, Philip Jones of CRU (Climatic Research Unit) claimed to have entered into a variety of confidentiality agreements with national meteorological services that prevent him from publicly archiving the land temperature data relied upon by IPCC.
Unfortunately, Jones seems to have lost or destroyed the confidentiality agreements in question and, according to the Met Office, can’t even remember who the confidentiality agreements were with. This doesn’t seem to bother the Met Office or anyone in the climate “Community”. This sits less well with Climate Audit readers, many of whom have made FOI requests for agreements between CRU and other countries throughout the world.
Because Jones is having so much trouble remembering who he made confidentiality agreements, we here at Climate Audit, always eager to assist climate scientists, are happy to do what we can to help Jones’ memory. I’ve spent some time reviewing Jones’ publications on the construction of the CRU_Tar index – in particular, Jones et al (1985); Jones (1994); Jones and Moberg (2003) and Brohan et al (2006). These contain interesting and relevant information on the provenance of Jones’ information and provide helpful clues on potential confidential agreements.
Jones insists on the distinction between academics and “non-academics” being scrupulously observed. I honored Jones’ demand that this distinction be observed by using his full academic title, Dr Phil, in the title, but, in the rest of the post, for the most part, I will refer to him more informally merely as Jones.
CDIAC
Jones has spent much of his academic career as a sort of temperature accountant. Commencing in the early 1980s, he collected station data and compiled averages – a useful enterprise, but surely no more than accounting.
In 1982 following the Three Mile Island incident and rising anti-nuclear sentiment, the US Department of Energy under the Reagan administration established the Carbon Dioxide Information Analysis Center (CDIAC), described by Wikipedia, as “the organization within the United States Department of Energy that has the primary responsibility for providing the US government and research community with global warming data and analysis as it pertains to energy issues.”
Wikipedia observes that CDIAC’s “present offices are located within the Environmental Sciences Division of Oak Ridge National Laboratory”. In fact, CDIAC’s offices have been at Oak Ridges either since its foundation (or closely so.) Oak Ridges’ main business is nuclear.
One of CDIAC’s first enterprises was its support of a world temperature index. As agents for this enterprise, they contracted Raymond Bradley, Thomas Wigley and Jones, all of whom become prominent figures in the IPCC movement. CDIAC published a “climatic data bank” of station temperatures in 1985 (Bradley et al 1985), the construction of which was described in Jones et al 1985a, 1985b, which described the data as “DOE” data. The first version was made available in print and diskette form, but a version from the late 1980s remains online at the CDIAC website. Jones et al 1985 descibed a data set with 1873 stations (1584 – NH; 289 – SH; 16- Antarctic); the ndp020r1 online version (circa 1990) has 1881 stations.
Here is a short clip of a contemporary confidential agent going to his work place, perhaps CDIAC, perhaps a front organization.
If Jones, as confidential agent for CDIAC in the 1980s, entered into confidentiality agreements with shadowy meteorological organizations and shady temperature data brokers around the world, he immediately breached such agreements by turning the data over to CDIAC, who published the station data both in print in 1985 and online. None of the shadowy meteorological organizations seems to have objected to any breach of contract by Jones.
More likely, there were no confidentiality agreements in this period. Jones et al 1985 does not describe any provenance of data from mysterious NMSs (national meterological services). Instead, it describes far more mundane provenance: by digitizing various editions of the Smithsonian’s World Weather Records and similar data from NCAR.
Jones(1994)
The first major update of the Jones data set was described in Jones (1994). Jones(1994) reported the continued financing of the CRU temperature index by the U.S. Department of Energy. The connection to the U.S. nuclear laboratories was less overt, with financing now attributed to the Atmospheric and Climate Research Division (Grant DE-FG02-86ER60397), attenuating the connection to the nuclear laboratories, though the archiving of gridded data at CDIAC, Oak Ridges was noted.
Jones (1994) referred to the use of 2961 stations; the next update (Jones and Moberg 2003) provided the additional information that the Jones (1994) dataset contained over 3900 stations, “of which 2961 stations were used in the gridding”. (The difference presumably arising from the availability of enough measurements in the reference period to qualify the series.)
A version of the Jones (1994) data set, vintage 1996, was made available at the CRU website (entitled cruwlda2.zip) and was publicly available from that time until July 31, 2009, when CRU purged their public data directories. For most of this period, there was a link from the CRU Advance-10Kwebpage (E.U. project number ENV4-CT95-0127) to cruwlda2.zip, evidencing that the data was intentionally placed in a public directory. On a few occasions, downloading of cruwlda2.zip has been reported by third parties.
Thus, if Jones entered into confidentiality agreements in respect to the Jones (1994) data set, once again, these agreements were promptly ignored.
It is more likely that there were no relevant confidentiality agreements from this period. Jones (1994) stated that the new stations were said to arise partly from additional information from stations in the extended network providing enough data to create a 1961-90 reference period (and thereby qualify the station for inclusion in gridding) and partly from “a number of projects e.g. Karl et al 1993 (BAMS)”, the latter including the Russia, China and Australia collections collated into GHCN. Jones (1994) did not mention the direct receipt of data from national meteorological organizations or international temperature data brokers/traders, shadowy or otherwise.
Jones and Moberg 2003
The next major update is described in Jones and Moberg 2003. Like the earlier versions, it was funded by the the U.S. Dept. of Energy (Grant DE-FG02-98ER62601). This time, Oak Ridges wasn’t mentioned anywhere.
Jones and Moberg 2003 reported the use of 5159 station records, of which 4167 had enough 1961-90 data to provide a reference period. Whereas the 1985 and 1994 editions contained no reference to direct dealings with national meteorological organizations (indeed they describe nothing much more than digitization of existing paper records or collation of GHCN sources), Jones and Moberg 2003 reported direct dealing with 7 national meteorological organizations:
CRU has collected a number of temperature records through direct contacts with the NMSs in Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan. Many of these records cover only the period 1961–90, but others extend over the entire twentieth century. Data for the whole of Australia for 1991–2000 have also been collected through direct contacts.
and later (also see Table 1):
CRU has received data from a number of countries through direct contacts with the respective NMSs. Records for 54 stations that were not represented in Jones were now included. Data for an additional 19 stations already represented in Jones were merged with priority for the NMS source. Most of these NMS records came from Iran, Algeria, Taiwan, Croatia, Israel, South Africa, and Syria. The merging of the NMS data was made after those from NHJ had been merged. As some stations were represented both in the NMS and NHJ data, the priority during merging was sometimes given to the data from NHJ rather than to data originating from Jones.
It’s an interesting list of NMSs. In other cases (Syria, Taiwan), CRU has some stations not represented in GHCN. In other cases (Iran, Algeria, Croatia, South Africa), there is an almost total overlap with GHCN IDs; however, for a number of stations, the Jones version is more extensive than the GHCN version, especially for Iran where CRU has an especially complete record. In this instance, the totalitarian regimes of CRU and the Ayatollah were apparently in complete agreement on the need to avoid public scrutiny. As to the confidentiality agreement between CRU and the Ayatollah, I guess that it was Burn after Reading.
Jones and Moberg (2003) also reported the acquisition of data on presumably conventional terms from less colorful regimes. They reported the use of new NORDKLIM data developed by the non-totalitarian regimes of Denmark, Sweden, Finland, Norway, Iceland, the United
Kingdom, Ireland, the Netherlands, and Belgium, as well as new datasets from non-totalitarian regimes in Canada and Australia.
Regardless of any possible confidentiality agreements, once again, CRU archived a near-contemporary version of the Jones and Moberg station data in early 2003, shortly after publication of Jones and Moberg 2003. The datasets newcrustnsall.dat and newcruextus.dat provide a combined total of 5070 stations, within 2% of the reported count of 5139 – pretty close when the Team is counting.
This dataset remained online until July 27, 2009, when it was purged by CRU. Unlike cruwlda2.zip, there were no links from CRU webpages to these datasets. Indeed, as we’ve discussed on previous occasions, both Dr Phil and the FOI officers at CRU and the Met Office on the one hand and people seeking copies of the Jones and Moberg data on the other seem to have been unaware of the existence of this data set until the connection of newcrustnsall.dat and newcruextus.dat to the long sought CRU station was reported at Climate Audit last month.
Brohan et al 2006
Brohan et al 2006, the next edition, reported that the development of the land station data set had been supported for the past 27 years by the US Department of Energy:
This work was funded by the Public Met. Service Research and Development Contract; and by the Department for Environment, Food and Rural Affairs, under Contract PECD/7/12/37. The development of the basic land station dataset has been supported over the last 27 years by the Office of science (BER), U.S. Department of Energy; most recently by grant DE-FG02-98ER62601.
They reported that the Jones and Moberg data set has “been expanded and improved for use in the new dataset”. It described the use of 4349 stations – I presume that this count corresponds to the 4167 in Jones and Moberg 2003 which had enough 1961-90 data to provide a reference period, rather than the 5159 overall total. As a result of FOI actions, a list of stations was provided in 2007 here, which reduced the count by 211 from 4349 to 4138. The associated webpage explained the reduction as follows:
The 4138 total is lower than the 4349 value given as the starting point for Brohan et al. (2006) and used in the latest IPCC Report. A small number of stations have been removed during Brohan et al. (2006) because of the presence of duplicate data and insufficient coverage for the period 1961-90.
“Duplicate” data mostly covers the inclusion of the same station in two different sources. For example, Jones 1994 included many USHCN stations under WMO numbers. Jones and Moberg 2003 added in the USHCN network with only a cursory effort to check for duplication – something that is a bit laborious, but only a few days work. As a result, there was extensive duplication. Most of the 211 duplicates removed for the webpage are USHCN stations. However, despite this belated check for duplicates carried out for the FOI response, quite a few duplicates still exist.
Although Brohan et al 2006 reported an “expanded” network, the new total of 4138 is lower than the count of Jones and Moberg 2003 stations in use (4167), though I presume that the Jones and Moberg count is inflated by 211 duplicates removed in the subsequent count. Brohan reported new stations were added for Mali, the Democratic Republic of Congo, Switzerland and Austria. Reported counts of 29 Mali stations and 33 Congo stations cannot be reconciled on existing information with much lower counts in the available listing.
While Brohan et al 2006 did not specifically attribute the provenance of the new data to direct contact by CRU with national meteorological organizations, this seems possible.
Confidentiality Policy
If one is trying to narrow down the search for the lost CRU confidentiality agreements, it seems to me that the logical place to start is with the following nine countries: Iran, Algeria, Taiwan, Croatia, Israel, South Africa, Syria, Mali, Congo. Did Dr Phil enter into confidentiality agreements with all or some of these nine countries? If so, what are the terms? And why are these agreements “lost”?
And exactly when were these agreements entered into? Our analysis shows that these agreements are not hoary old agreements from early Oak Ridges-CDIAC days, but were made well within the IPCC period. As a long-time IPCC hand, Jones knew of the IPCC commitment to “openness and transparency”.
Did Jones seek approval from an advisory board or some other form of independent oversight prior to unilaterally entering into confidentiality agreements with various totalitarian (and other) regimes? If these few confidentiality agreements are sufficient to poison public disclosure of the entire database (as CRU and the Met Office now argue), how was Phil Jones able to poison the public availability of the database through a few seemingly unsupervised confidentiality agreements?
UPDATE: As a closing thought (h/t reader below), recent CRU efforts to sanitize their public directories were perhaps anticipated by another confidential agent’s Cone of Silence:
0.47 (not significant at all).
Climate Audit 