Recently, Philip Jones of CRU (Climatic Research Unit) claimed to have entered into a variety of confidentiality agreements with national meteorological services that prevent him from publicly archiving the land temperature data relied upon by IPCC.
Unfortunately, Jones seems to have lost or destroyed the confidentiality agreements in question and, according to the Met Office, can’t even remember who the confidentiality agreements were with. This doesn’t seem to bother the Met Office or anyone in the climate “Community”. This sits less well with Climate Audit readers, many of whom have made FOI requests for agreements between CRU and other countries throughout the world.
Because Jones is having so much trouble remembering who he made confidentiality agreements, we here at Climate Audit, always eager to assist climate scientists, are happy to do what we can to help Jones’ memory. I’ve spent some time reviewing Jones’ publications on the construction of the CRU_Tar index – in particular, Jones et al (1985); Jones (1994); Jones and Moberg (2003) and Brohan et al (2006). These contain interesting and relevant information on the provenance of Jones’ information and provide helpful clues on potential confidential agreements.
Jones insists on the distinction between academics and “non-academics” being scrupulously observed. I honored Jones’ demand that this distinction be observed by using his full academic title, Dr Phil, in the title, but, in the rest of the post, for the most part, I will refer to him more informally merely as Jones.
CDIAC
Jones has spent much of his academic career as a sort of temperature accountant. Commencing in the early 1980s, he collected station data and compiled averages – a useful enterprise, but surely no more than accounting.
In 1982 following the Three Mile Island incident and rising anti-nuclear sentiment, the US Department of Energy under the Reagan administration established the Carbon Dioxide Information Analysis Center (CDIAC), described by Wikipedia, as “the organization within the United States Department of Energy that has the primary responsibility for providing the US government and research community with global warming data and analysis as it pertains to energy issues.”
Wikipedia observes that CDIAC’s “present offices are located within the Environmental Sciences Division of Oak Ridge National Laboratory”. In fact, CDIAC’s offices have been at Oak Ridges either since its foundation (or closely so.) Oak Ridges’ main business is nuclear.
One of CDIAC’s first enterprises was its support of a world temperature index. As agents for this enterprise, they contracted Raymond Bradley, Thomas Wigley and Jones, all of whom become prominent figures in the IPCC movement. CDIAC published a “climatic data bank” of station temperatures in 1985 (Bradley et al 1985), the construction of which was described in Jones et al 1985a, 1985b, which described the data as “DOE” data. The first version was made available in print and diskette form, but a version from the late 1980s remains online at the CDIAC website. Jones et al 1985 descibed a data set with 1873 stations (1584 – NH; 289 – SH; 16- Antarctic); the ndp020r1 online version (circa 1990) has 1881 stations.
Here is a short clip of a contemporary confidential agent going to his work place, perhaps CDIAC, perhaps a front organization.
If Jones, as confidential agent for CDIAC in the 1980s, entered into confidentiality agreements with shadowy meteorological organizations and shady temperature data brokers around the world, he immediately breached such agreements by turning the data over to CDIAC, who published the station data both in print in 1985 and online. None of the shadowy meteorological organizations seems to have objected to any breach of contract by Jones.
More likely, there were no confidentiality agreements in this period. Jones et al 1985 does not describe any provenance of data from mysterious NMSs (national meterological services). Instead, it describes far more mundane provenance: by digitizing various editions of the Smithsonian’s World Weather Records and similar data from NCAR.
Jones(1994)
The first major update of the Jones data set was described in Jones (1994). Jones(1994) reported the continued financing of the CRU temperature index by the U.S. Department of Energy. The connection to the U.S. nuclear laboratories was less overt, with financing now attributed to the Atmospheric and Climate Research Division (Grant DE-FG02-86ER60397), attenuating the connection to the nuclear laboratories, though the archiving of gridded data at CDIAC, Oak Ridges was noted.
Jones (1994) referred to the use of 2961 stations; the next update (Jones and Moberg 2003) provided the additional information that the Jones (1994) dataset contained over 3900 stations, “of which 2961 stations were used in the gridding”. (The difference presumably arising from the availability of enough measurements in the reference period to qualify the series.)
A version of the Jones (1994) data set, vintage 1996, was made available at the CRU website (entitled cruwlda2.zip) and was publicly available from that time until July 31, 2009, when CRU purged their public data directories. For most of this period, there was a link from the CRU Advance-10Kwebpage (E.U. project number ENV4-CT95-0127) to cruwlda2.zip, evidencing that the data was intentionally placed in a public directory. On a few occasions, downloading of cruwlda2.zip has been reported by third parties.
Thus, if Jones entered into confidentiality agreements in respect to the Jones (1994) data set, once again, these agreements were promptly ignored.
It is more likely that there were no relevant confidentiality agreements from this period. Jones (1994) stated that the new stations were said to arise partly from additional information from stations in the extended network providing enough data to create a 1961-90 reference period (and thereby qualify the station for inclusion in gridding) and partly from “a number of projects e.g. Karl et al 1993 (BAMS)”, the latter including the Russia, China and Australia collections collated into GHCN. Jones (1994) did not mention the direct receipt of data from national meteorological organizations or international temperature data brokers/traders, shadowy or otherwise.
Jones and Moberg 2003
The next major update is described in Jones and Moberg 2003. Like the earlier versions, it was funded by the the U.S. Dept. of Energy (Grant DE-FG02-98ER62601). This time, Oak Ridges wasn’t mentioned anywhere.
Jones and Moberg 2003 reported the use of 5159 station records, of which 4167 had enough 1961-90 data to provide a reference period. Whereas the 1985 and 1994 editions contained no reference to direct dealings with national meteorological organizations (indeed they describe nothing much more than digitization of existing paper records or collation of GHCN sources), Jones and Moberg 2003 reported direct dealing with 7 national meteorological organizations:
CRU has collected a number of temperature records through direct contacts with the NMSs in Algeria, Croatia, Iran, Israel, South Africa, Syria, and Taiwan. Many of these records cover only the period 1961–90, but others extend over the entire twentieth century. Data for the whole of Australia for 1991–2000 have also been collected through direct contacts.
and later (also see Table 1):
CRU has received data from a number of countries through direct contacts with the respective NMSs. Records for 54 stations that were not represented in Jones were now included. Data for an additional 19 stations already represented in Jones were merged with priority for the NMS source. Most of these NMS records came from Iran, Algeria, Taiwan, Croatia, Israel, South Africa, and Syria. The merging of the NMS data was made after those from NHJ had been merged. As some stations were represented both in the NMS and NHJ data, the priority during merging was sometimes given to the data from NHJ rather than to data originating from Jones.
It’s an interesting list of NMSs. In other cases (Syria, Taiwan), CRU has some stations not represented in GHCN. In other cases (Iran, Algeria, Croatia, South Africa), there is an almost total overlap with GHCN IDs; however, for a number of stations, the Jones version is more extensive than the GHCN version, especially for Iran where CRU has an especially complete record. In this instance, the totalitarian regimes of CRU and the Ayatollah were apparently in complete agreement on the need to avoid public scrutiny. As to the confidentiality agreement between CRU and the Ayatollah, I guess that it was Burn after Reading.
Jones and Moberg (2003) also reported the acquisition of data on presumably conventional terms from less colorful regimes. They reported the use of new NORDKLIM data developed by the non-totalitarian regimes of Denmark, Sweden, Finland, Norway, Iceland, the United
Kingdom, Ireland, the Netherlands, and Belgium, as well as new datasets from non-totalitarian regimes in Canada and Australia.
Regardless of any possible confidentiality agreements, once again, CRU archived a near-contemporary version of the Jones and Moberg station data in early 2003, shortly after publication of Jones and Moberg 2003. The datasets newcrustnsall.dat and newcruextus.dat provide a combined total of 5070 stations, within 2% of the reported count of 5139 – pretty close when the Team is counting.
This dataset remained online until July 27, 2009, when it was purged by CRU. Unlike cruwlda2.zip, there were no links from CRU webpages to these datasets. Indeed, as we’ve discussed on previous occasions, both Dr Phil and the FOI officers at CRU and the Met Office on the one hand and people seeking copies of the Jones and Moberg data on the other seem to have been unaware of the existence of this data set until the connection of newcrustnsall.dat and newcruextus.dat to the long sought CRU station was reported at Climate Audit last month.
Brohan et al 2006
Brohan et al 2006, the next edition, reported that the development of the land station data set had been supported for the past 27 years by the US Department of Energy:
This work was funded by the Public Met. Service Research and Development Contract; and by the Department for Environment, Food and Rural Affairs, under Contract PECD/7/12/37. The development of the basic land station dataset has been supported over the last 27 years by the Office of science (BER), U.S. Department of Energy; most recently by grant DE-FG02-98ER62601.
They reported that the Jones and Moberg data set has “been expanded and improved for use in the new dataset”. It described the use of 4349 stations – I presume that this count corresponds to the 4167 in Jones and Moberg 2003 which had enough 1961-90 data to provide a reference period, rather than the 5159 overall total. As a result of FOI actions, a list of stations was provided in 2007 here, which reduced the count by 211 from 4349 to 4138. The associated webpage explained the reduction as follows:
The 4138 total is lower than the 4349 value given as the starting point for Brohan et al. (2006) and used in the latest IPCC Report. A small number of stations have been removed during Brohan et al. (2006) because of the presence of duplicate data and insufficient coverage for the period 1961-90.
“Duplicate” data mostly covers the inclusion of the same station in two different sources. For example, Jones 1994 included many USHCN stations under WMO numbers. Jones and Moberg 2003 added in the USHCN network with only a cursory effort to check for duplication – something that is a bit laborious, but only a few days work. As a result, there was extensive duplication. Most of the 211 duplicates removed for the webpage are USHCN stations. However, despite this belated check for duplicates carried out for the FOI response, quite a few duplicates still exist.
Although Brohan et al 2006 reported an “expanded” network, the new total of 4138 is lower than the count of Jones and Moberg 2003 stations in use (4167), though I presume that the Jones and Moberg count is inflated by 211 duplicates removed in the subsequent count. Brohan reported new stations were added for Mali, the Democratic Republic of Congo, Switzerland and Austria. Reported counts of 29 Mali stations and 33 Congo stations cannot be reconciled on existing information with much lower counts in the available listing.
While Brohan et al 2006 did not specifically attribute the provenance of the new data to direct contact by CRU with national meteorological organizations, this seems possible.
Confidentiality Policy
If one is trying to narrow down the search for the lost CRU confidentiality agreements, it seems to me that the logical place to start is with the following nine countries: Iran, Algeria, Taiwan, Croatia, Israel, South Africa, Syria, Mali, Congo. Did Dr Phil enter into confidentiality agreements with all or some of these nine countries? If so, what are the terms? And why are these agreements “lost”?
And exactly when were these agreements entered into? Our analysis shows that these agreements are not hoary old agreements from early Oak Ridges-CDIAC days, but were made well within the IPCC period. As a long-time IPCC hand, Jones knew of the IPCC commitment to “openness and transparency”.
Did Jones seek approval from an advisory board or some other form of independent oversight prior to unilaterally entering into confidentiality agreements with various totalitarian (and other) regimes? If these few confidentiality agreements are sufficient to poison public disclosure of the entire database (as CRU and the Met Office now argue), how was Phil Jones able to poison the public availability of the database through a few seemingly unsupervised confidentiality agreements?
UPDATE: As a closing thought (h/t reader below), recent CRU efforts to sanitize their public directories were perhaps anticipated by another confidential agent’s Cone of Silence:
Climate Audit 
91 Comments
As a party to many Confidentiality Agreements, I would immediately terminate an agreement with any organization so incompetent as to lose the agreement, and I would immediately demand return of all confidential information.
I’d be interested in information from DOE on grant DE-FG02-98ER62601 if any readers track down information.
Re: Steve McIntyre (#2),
This turns out to be a good question. Google and Google Scholar return many hits for which this grant is acknowledged as a funding source. The time period and number of acknowledgements by different authors seems to indicate that it was used as some kind of generic, general-purpose, boiler-plate Grant for many years, and for many and various institutions.
Can’t find anything about it at doe.gov or osti,gov.
Re: Steve McIntyre (#2),
The Energy Citations Database maintained by the US DOE / Office of Science and Technical Information (OSTI) lists only the following report for a search on: DE-FG02-98ER62601:
Identification of Anthropogenic Climate Change Using a Second-Generation Reanalysis
Ben Santer, Tom Wiglet, Adrian Simmons, Per Kallberg,Graeme Kelly, Sakari Uppala, Caspar Ammann, James Boyle, Wolfgang Bruggemann, Charles Doutriaux, Mike Fiorino, Carl Mears, Gerald Meehl, Robert Sausen, Karl Taylor, Warren Washington, Michael Wehner, Frank Wentz
Lawrence Livermore Laboratory Report # UCRL-JRNL-204468, 2004 Jun 02
Re: Steve McIntyre (#2),
Previous comment #66 provides a direct link to all details regarding DOE grant DE-FG02-98ER62601: http://www.osti.gov/oberabstracts/detail.jsp?projectSerial=2002&query_id=5&searchpage=search.easy.jsp
The full alphanumeric designator of the grant number is not spelled out but the information on the web page makes it clear that this is the same grant (first awarded 10/1/97, which is fiscal year 98 for DOE; the same publications that noted they were funded by grant DE-FG02-98ER62601 are mentioned on the web page; it is to Phil Jones; etc.).
Now that it is established that DOE funded data collection and analysis, Dr. Jones’ refusals to release data become exceedingly puzzling as DOE has some strict guidelines for data sharing. Everywhere you go this site it states: “Funding of projects by the program is contingent on adherence to BER’s data sharing policy.” with BER being the Office of Biological & Environmental Research that provides the funding.
On webpage http://www.science.doe.gov/ober/CCRD/per.html it states:
I’m sure you would qualify as a member of “the interested public”. Perhaps you could try requesting that Dr. Jones specifically provide all “modeling methods or results, a copy of model code, parameter values, and/or any input dataset(s)” generated as a result of DOE grants from Office of Biological & Environmental Research. Given BER’s data sharing policies I’m sure that the Program Manager would not take kindly to a “why should I give you the data…” type of response.
If for some reason this does not work there is always the option of a FOIA to DOE. As you found out by requesting information regarding Santer at one of the DOE labs, DOE takes FOIA seriously.
Additionally if you search http://www.osti.gov/oberabstracts for “jones” you will find Phil D. Jones of University of East Anglia received multiple other grants including current ones.
A person in a high ranking position loses ALL the agreements? Take note that assistants to parliamentary Ministers in Canada who misplaced confidential papers were fired or resigned. Are the British MP’s reading this? Hello.
Steve again does the reading that writers won’t do.
And as cruwlda2.zip was published data, someone should be in trouble.
If IPCC really professes such a policy, and CRUTAR is in fact closed and opaque, doesn’t this automatically disqualify it from inclusion in AR5?
My question is rhetorical, of course, since it hasn’t stopped them in the past, but still I think it makes a good talking point.
[As a long-time IPCC hand, Jones knew of the IPCC commitment to “openness and transparency”.]
A bit of O&T from CRU would be welcome at this moment.
CA allows me the opportunity to gently hyperventilate.
Thanks, Steve
Jack
All of the .Z and .zip files on the “Climate data for Advance10K” page were deleted from CRU’s FTP server. Advance10K was a Eurasian tree ring study. So now they’re blocking replication of all the studies based upon the Advance10K tree ring data.
Re: AnonyMoose (#7),
The link in your comment appears to point to a different directory than the one in mine (copied from Steve’s lead post. On the surface, they appear to be the same page.
Remember, Taiwan is officially the Republic of China. FYI if asking for records.
None of the files on the CRU Advance-10K webpage do not seem to be accessible. Sporadically, I get a response which asks me to log in. When I try to do so anonymously (with a correct e-mail address as a password), my login is refused. ???!
Steve:
That was more fun than reading Spencer Weart’s book!!
Once again, Steve, a great story, complete with many laughs. Much of Climate Science is very funny.
I hope the data becomes public sometime in the next year and is impartially evaluated by someone with the intellect and skills (need not be a “Climate Scientist — or even a “Scientist” — though a Scientist is one by virtue of what he/she accomplishes — not by virtue of a PhD).
The only paper previously consumed by dogs was homework but canine appetites have apparently broadened to include legal documents.
Re: Person of Choler (#14),
I am not so sure. This may be the first case of a homework/document eating dog dying of starvation.
It seems like Jones has been operating for nigh on 30 years without any quality control. I cannot find anything on the CRU website about their quality management or if they are ISI-9001 accredited. The Met Office holds ISO 9001-2000 accreditation and, wherever practical, requires that its suppliers also hold such accreditation.
The CRU website says:
“The area of CRU’s work that has probably had the largest international impact was started in 1978 and continues through to the present-day: the production of the world’s land-based, gridded (currently using 5° by 5° latitude/longitude boxes) temperature data set. This involved many person-years of painstaking data collection, checking and homogenization. In 1986, this analysis was extended to the marine sector (in co-operation with the Hadley Centre, Met Office from 1989)”.
Note the ‘many person-years of painstaking data collection and checking’.
In the history of CRU it says:
“CRU’s work in these early years played a major part in navigating the study of climate change out of an academic backwater and started to set the agenda for the major research effort in, and political preoccupation with, climate research since.”
Note the ‘political preoccupation with climate research.’
The CRU describes itself on its website thus:
“The Climatic Research Unit is widely recognised as one of the world’s leading institutions concerned with the study of natural and anthropogenic climate change.
Consisting of a staff of around thirty research scientists and students, the Unit has developed a number of the data sets widely used in climate research, including the global temperature record used to monitor the state of the climate system, as well as statistical software packages and climate models.
The aim of the Climatic Research Unit is to improve scientific understanding in three areas:
* past climate history and its impact on humanity;
* the course and causes of climate change during the present century;
* prospects for the future.
The Unit undertakes both pure and applied research, sponsored almost entirely by external contracts and grant from academic funding councils, government departments, intergovernmental agencies, charitable foundations, non-governmental organisations, commerce and industry.
Alongside its research activities, the Unit has an educational role through its contribution to formal teaching with the School of Environmental Sciences (most notably, the MSc in Climate Change) and various forms of in-service training including postgraduate education. It is regarded as an authoritative source of information on both the science and policy aspects of climate change by the media and maintains a high public profile.
The staff of the Unit have an enviable publication record, contributing to both peer-review and popular journals as well as editing various newsletters and bulletins.
The Climatic Research Unit is part of the School of Environmental Sciences with close links to other research groups within the department such as the Centre for Social and Economic Research on the Global Environment. The Unit undertakes collaborative research with institutes throughout the world on a diverse range of topics and is coordinating or contributing to a number of networking activities.”
Re: Phillip Bratby (#15),
Interesting point. “The Met Office holds ISO 9001-2000 accreditation and, wherever practical, requires that its suppliers also hold such accreditation.”
One would think that quality control should apply to Met’s data and not only its brochure paper.
* “In order to meet best-practice standards as an employer, supplier and an environmental agency, the Met Office must maintain accreditation under ISO 9001 and 14001 and Investor in People (IiP).”
** Well, the lawyers think ISO 9001 is important to what the Met supplies.
* “A corporate risk management process is one of the key processes of the Met Office’s ISO 9001 Business Management System.”
** I wonder if corruption of their data is written in their risk books.
* “The Met Office is ISO 9001 accredited and the STFS operating instructions and methodology are under constant review as an ongoing part of this process.”
** Good, they review their tidal warning “operating instructions and methodology”.
Summary:
– CRU purges data, suddenly & without explanation
– Requests for data are denied under oddly asserted “confidentiality agreement[s]” restriction[s]
– Available historical record indicates a pattern inconsisent with the need for such “confidentiality agreements”
snip – too much inappropriate speculation
If those confidentiality agreements do not exist at present, they probably never existed. It is an excuse made up by Phil Jones not to have his data and methods checked by independent researchers.
snip -please do not extrapolate beyond the specifics
Has anyone ever uncovered any evidence that there have been any “confidentiality agreements” at all? Given his stonewalling, “dog ate my homework” excuses, and the reluctance of CRU (or anyone at the University of East Anglia) to press him for information, I’m beginning to think that only some sort of lawsuit has any chance of prying out the truth of the matter.
I’ve never understood why so few climate scientists will stand up and call foul over nonsense like this. They have to know that such behavior tarnishes more than just Phil Jones.
Re: MikeU (#18),
“I’ve never understood why so few climate scientists will stand up and call foul over nonsense like this.”
To avoid confusing Mr Mug Q Public, of course.
I’ve seen quite a few Letters to the Editor in populist newspapers wherein a Professor of Astrophysics and other such disciplines have stated that these kerfuffles and data perturbations are simply inconsequential.
It is astonishing that there never seem to be any consequences or ramifications for the myriad failings by “respected” scientists so often documented here and in other truth-seeking quarters. Here’s hoping we’re reaching a genuine “tipping point”, after which the wider world will begin to acknowledge these emperors have no clothes.
OK, OK, I think I know what is going on here: It’s all very convoluted (natch), but such is the way of the world (of climate science). The key is that there is still one more level of convolution, namely that there exists a confidentiality agreement with respect to the confidentiality agreements. That explains it. Yes, Dr. Phil would poison his own well. Don’t be naive, Mr. McIntyre: He includes the poison pill as a “suicide out”; it just has to be swallowed. I’m quite sure that, if you care to look closely, you will find the uber-confidentiality agreement under Mali (one of your narrowed-down suspects). Note the pun of it! What does “mal” mean in French? Huh? Huh?
snip – no need to vent
Gerald – that clinches it! I’m hitting the tip jar on this one 🙂 Good stuff Steve.
Why am I reminded of the character Corporal Jones from the BBC sitcom Dad’s Army?
Folks, please resist the temptation to make piling on comments. I have no reason to believe that the stonewalling is evidence of anything more than prima donna behavior or the data deletion from public directories is anything more than a temper tantrum.
If I do not have that evidence, neither, in all likelihood, do individual readers.
You can think whatever you want, but editorially please stay away from angry speculations about motives.
Re: Steve McIntyre (#25),
If you do not have the evidence then you are engaging in speculation. This is getting to be like a cat playing with a mouse. Either kill it or let it go. My comment above was made only in response to a reader introducing discussion of motives despite my repeated requests to stay away form such speculation.
Steve: Nor do I speculate on motives in this post. I see little purpose in such speculations and even less on speculations involving lurid motives.
Re: Patrick M. (#29),
Should read:
My point is that it seems premature to be posting on this stuff if you don’t have the goods on Jones. What’s the point? It seems a bit shrewish.
Just my $0.02
Steve: Seems to me that Jones has activated the CRU Cone of Silence:
Steve: Quite so. I’ve adopted this insightful observation in my post.
snip –
Steve: please do not speculate on why the U.S. nuclear industry was interested in carbon dioxide. I think that the reasons are obvious and not unintelligent.
“Folks, please resist the temptation to make piling on comments. I have no reason to believe that the stonewalling is evidence of anything more than prima donna behavior or the data deletion from public directories is anything more than a temper tantrum.
If I do not have that evidence, neither, in all likelihood, do individual readers.
You can think whatever you want, but editorially please stay away from angry speculations about motives.
”
Your whole article was one of sneering contempt, then you wonder why you stir up such passions.
Re: slownewsday (#32),
Readers are welcome to place their own labels on the conduct of Jones and CRU. I understand why you would apply the term “sneering contempt” to describe the behavior of Jones and CRU towards the public, though this phrase is stronger rhetorically than the terms that I myself normally use or used in the article. I described the deletion of files and FOI refusals in more ironic language, but perhaps stronger language of the type that you urge here is justified after all.
The obstruction of “openness and transparency” for an important issue such as climate change is something that should arouse passion and I’m glad that you are so inspired. I hope that you support our campaign for disclosure of the various confidentiality agreements that supposedly encumber this important data set.
If there are any errors of fact or material misinterpretations in this post, I’d appreciate it if you would draw them to my attention so that I can correct them.
Re: Steve McIntyre (#32), I think he was referring to you by that comment, not Mr Jones et al.
Good find re the very limited possibilities of nations who may have tied Mr Jones’ hands, and how recent that must have been.
Sounds like some USA FOI’ing is in order now too. Your Senator on the environment committee should also be advised of this progress.
Small correction:
“Oak Ridge”, not “Oak Ridges”
Great, great detective work Steve.
Dr. Phil. Don`t tarp me dude.
I just want my DOE funded data!
I want to point out that if Jones really had a ‘confidentiality agreement’ this would almost certainly have been executed through a Materials Transfer Agreement (MTA). EVERY academic organization with which I have been associated has had STRICT rules about any transfer of materials (including information) that place ANY sort of restrictions or liability limitations on the material. As a ‘lone’ professor, I have absolutely no authority to sign one of these MTAs on my own. They MUST be routed through the University legal Dept which reads over (and often rejects) the legal language in the MTA. Indeed, I have had to abandon many lines of research because the lawyers simply wouldn’t sign something they didn’t understand or accept, and the material donors had no interest in rewriting their standard MTA form.
If Jones executed any agreements with third parties, there would almost certainly be a record of such in the legal Dept of his home institution. That would be the place for an effective FOI.
Here’s Dr. Phil’s song:
Let’s see if I am computer literate enough to have done this right…
DaveCF
Re: DaveCF (#39),
Why do people have such a flair for the melodramatic?
Phil Jones is just being a tremendous dick. I don’t think that Steve is likely to find Jimmy Hoffa’s corpse in there. I don’t even think that the CRU algorithm will be much more than weighted averages.
It would be nice to solve the puzzle as to exactly what Jones is doing with this highly influential dataset and why he believes it will help his cientific reputation to behave like a petulant child, but its not the be-all and end-all of climate science at this point.
It’s just unsurprisingly normal behavior for the Team
Re: John A (#42),
“Phil Jones is just being a tremendous dick. I don’t think that Steve is likely to find Jimmy Hoffa’s corpse in there. I don’t even think that the CRU algorithm will be much more than weighted averages.”
I agree John A. One of the great things about Steve M is that despite the fact that he has low expectations for the outcome of most of the auditing work he does, he nonetheless persists in his auditing. This IMO is a very admirable quality and demonstrates just how much more Steve M deserves to be called a ‘scientist’ then do the poor excuses for scientists that make up ‘The Team’.
In this case (i.e. the failure of CRU to release the station data) it’s much more about holding CRU to task than finding any huge mistakes in CRUTAR(TEM). The Wizard of EAU set himself up for all this agro when he so arrogant told Warwick Hughes that he would never release the data to him because as far as ‘he’ (the loud voice behind the curtain) was concerned, Warwick Hughes only wanted to find fault with it. This may well have been true for Warwick Hughes at the time but certainly is not the case for Steve M at the moment. Steve M doesn’t have any agenda in this respect. Plain and simply, he just wants the data released so that it can be audited. After having previously audited GISS, RSS and UAH, he just now wants to complete the ‘full set’ so to speak.
Now we can all perhaps speculate what the outcome may be of Steve’s auditing of CRUTAR(TEM) but IMO the balance of probablity (as I’ve personally concluded from the results from his auditing of the other 3 indices) is that his audit will likely show that the claimed ‘warming trend’ towards the end of the 20th century is likely to be largely as a consequence of the ‘adjustments’ made to the raw station data and/or due to the methods used to analyse the raw data. If this turns out to be the case then the ‘take home message’ for me will be that all FOUR indices are problematic, that none of them can and so should NOT be relied upon (by policy makers) as snip – editorializing
KevinUK
well so much for the argument that the agreements must have predated the enactment of FOIA.
From the Telegraph:
Weather records are a State Secret.
Main post:
Does that duplication have any significant effect on the final gridded product?
Does it effectively give double weighting to the USHCN network?
Or is all of this also opaque because of lack of disclosure of processing methods?
Which country lays claim the oceans and so could have made confidentiality agreements over 71% of the Earth’s surface?
Which of the 7 applicant countries has had its claim to Sovereignty over the parts of the Antarctic approved by the rest of the World? (None) How could an agreement with CRU be made over the Antarctic?
Was agent 99 measured in Fahrenheit – or millimeters circumference?
Does this Australian agreement apply? It comes with Bureau of Meteorology data which are protected by copyright law and has been expressed for some years in this or similar form.
I might be remiss as I have made very minor modifications (in-fills of missing data) without the permission of the Director. It was my feeling that this was fair use and has been noted. The tiny volume would simply be an annoyance to the Director to handle formally.
The Director currently has a formal letter asking if there exists an Australian agreement with CRU or related bodies, covering not only this copyright statement, but also commercial use by CRU in general. This possible CRU use is more than an infrequent, tiny use and might be more than an annoyance.
My diagnosis: academic “its mine” disease, which afflicts many many researchers who have an investment in either collecting data or in being an expert for some topic. In other words, territorial behavior.
My prediction as to what Steve would find if data released: duplicate stations, egregious errors undetected, odd data processing methods, failure to update stations whose data are easily accessible.
Browsing the CRU website, I see they provide a link to “make your voice heard” on Copenhagen. Proceeding to the Copenhagen site, there are forums and wiki available for public comment. I suggest we make use of these…
As for Mr. Jones, I shall quote Andreas Schliefer, who said “Without data, you are just another person with an opinion.” Unless CRU becomes open about their data, they are just another group of people with an opinion, and thus are just an advocacy group…
What is the basis for this statement? As I understand it the FOI requests regarding the confidentiality agreements are still pending, and the 20 working days for a response will expire roughly August 21? Perhaps I missed something on an earlier thread.
Also, he is Prof Phil!
Steve: On an earlier thread, the Met Office refusal letter stated: “Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released and it cannot be determined which countries or stations data were given in confidence as records were not kept.”
Just wanted to note that it is Oak Ridge National Laboratory in Oak Ridge, Tennessee. You have Oak Ridges repeatedly in the article. Otherwise great article. Hopefully sanity will prevail with FOI being granted.
Steve, thanks for clarifying this – I have found it now – and apologies for not keeping up!
If the data was ever confidential, it is (mostly) not anymore. I did a search on the suspect countries Iran, Algeria, Taiwan, Croatia, Israel, South Africa, and Syria in Mathematica and found the 20 closest stations to the country centers (though I can’t verify that all these are inside the countries in question). Checking Syria first, the 20 closest stations to country center are
{“HLLS”, “WMO62259”, “WMO62200”, “WMO62131”, “WMO62018”, “WMO62161”, \
“HLKF”, “WMO62121”, “WMO62120”, “WMO62019”, “HLGT”, “WMO62055”, \
“DAAJ”, “WMO62014”, “DAUZ”, “FTTY”, “DRRI”, “WMO62016”, “HLLB”, \
“DAAP”}
but the distances to country center (km) are:
{339.832, 425.022, 458.599, 469.965, 632.448, 637.588, 643.482, \
643.931, 687.286, 691.392, 691.506, 708.533, 766.505, 804.338, \
807.467, 810.858, 819.885, 847.266, 852.104, 876.436}
which makes it seem maybe these are outside the country–I don’t know how to check these.
Algeria seems better but some of the distances are also large:
{“DAUI”, “WMO60630”, “DAUE”, “DAUT”, “DAUA”, “WMO60603”, “DAUG”, \
“DAUU”, “DAUH”, “DAFH”, “WMO60563”, “WMO60602”, “DAAP”, “DAAT”, \
“WMO60680”, “WMO60560”, “DAUL”, “DAUK”, “DAOR”, “DAUZ”}
{96.2728, 98.6019, 286.525, 299.682, 313.286, 408.205, 493.962, \
494.212, 508.968, 549.62, 549.809, 555.911, 561.43, 627.788, 628.385, \
633.218, 641.682, 643.844, 646.04, 652.723}
taiwan seems plausible
{“WMO59158”, “WMO59562”, “WMO59358”, “RCKH”, “C9014”, “WMO59559”, \
“WMO59567”, “RCTP”, “WMO58968”, “RCSS”, “C3721”, “WMO58944”, \
“WMO58974”, “WMO98132”, “ZSAM”, “ROIG”, “ZSFZ”, “WMO98134”, \
“WMO58931”, “WMO58926”}
{79.2318, 84.8901, 97.5269, 122.437, 158.011, 168.94, 172.788, \
177.246, 178.535, 183.421, 191.398, 256.139, 260.859, 313.075, \
316.193, 335.342, 335.825, 355.219, 383.448, 416.315}
All the other countries look like there are weather stations within their borders.
Re: Craig Loehle (#50),
You have to distinguish historical data from current data.
In Iran, it looks like Dr Phil obtained a tranche of historical data ending in 1990 or so that is not at GHCN.
Re: Craig Loehle (#50),
Syria’s longest internal distance is about 760 km, Algeria 2,100 km and Taiwan about 370 km, but Taiwan has airports at places like Orchid Island some 70 km E of the Sth tip and these are candidates for weather stations.
In Taiwan Code RCKH is Kaohsiung Intrnational Airport.
In Syria, DAAP is Nejrab airport.
In Algeria DAUZ is In Amenas airport, DAUK is Touggourt Airport.
You should not have to go through agony to find these out. I used Google Earth.
From Steve’s post:
It strikes me as very odd that data on global warming is managed not by something like NOAA or NCAR or NWS, but by an agency devoted to “Carbon Dioxide Information”. CDIAC is actually the official repository (and final adjustor) of USHCN data. There may or may not be a link between CO2 and warming, and perhaps the DOE should have an agency to study this link. But why is their agency in charge of the weather data itself?
Perhaps CDIAC stands for Carbon Dioxide Information and Alarmism Center??
My personal opinion is that if any scientist whether climate or some other field, withholds data and the computer code that manipulates that data, then his work should be disregarded as hearsay and opinion.
A part of a persons word is their work and if we cannot see their work then their word is worth little.
Oak Ridge National Lab in Oak Ridge, Tenn. is run by the US Dept of Energy. Oak Ridge was home to a group of related facilities (ORNL, Y-12, K-25, X-10) that were involved in the development of nuclear weapons. All were turned over to the DOE and have been downsized substantially. Their missions have been broadened considerably to begin branching out in other areas of energy beside nuclear.
A reference to DOE may very well be the same as one to Oak Ridge.
Do not reject as an urban myth the story that Britons are prone to mislay secrets by leaving them on the train. I once found pages of hard copy secrets on a London to Cambridge train. How did I know that they were secrets? It said so on most pages.
A question: in assessing science should we be seeking replicability or reproducibility? It strikes me that whats being chased by the participants in this blog replicability. But surely science advances best by seeking reproducibility. (This is not just my idea, there’s some interesting discussions about the nuances of the difference. For example, see http://www.site.uottawa.ca/~cdrummon/pubs/ICMLws09.pdf)
It strikes me that if we spent more time and on reproducibility independent of someone else’s methods/code, rather than policing the science of others, then science would be the big winner.
This is quite aside from the interesting debate on whether science should or should not require full disclosure of code and data.
Re: berbmit (#56), Ah, yes, but in this case we have a government agency holding the world’s climate data, and if you try to go get your own copies of it it’s the “wrong” data. So since what CRU puts out is the “official” trend, it seems critical to audit that.
berbmit:
housekeeping: your link picked up the closing parentheses after .pdf, better is:
Click to access ICMLws09.pdf
Apples and oranges. A model is not an experimental method. The model is the result. Examining its “fit” to the real world tells others something about our understanding (or lack thereof) of a science. Also helps identify a discipline’s gaps in the micro data.
I’m pretty sure this blog is not about replication of results. The model itself is what is important to the discipline and should be shared. There’s quite a few finacial firms wondering what happened to them even though they bet the farm on the results of proprietary econometric models.
I don’t know what common practice is in the UK, but in most organizations that I’m familiar with, copies of legally binding contracts are kept by the attorneys, no? I’ve never heard of the original copy of anything legally binding being held by the signatory. That’s what duplicate copies are for.
So not only did Dr. Phil lose all of the several agreements, but they were all originals, and his attorney doesn’t have copies? If he says so, I suppose…
I work in health records, and ALL of our privacy agreements are kept under lock and key (or scanned and kept on secure servers), with various paper and electronic tables spelling out where they are.
Legally, there is no agreement unless there is a signed copy, by both parties, FOR each party.
It is simply unbelievable that someone can claim that an agreement is in place if he cannot provide a copy. Generally, if there is no record, there is no agreement. I think what we may be seeing here is a “memory” file or policy that quite simply doesn’t exist anywhere on paper.
I would like to be an auditor over there right now. This would be the easy investigation in history. “Show your work” and “prove it” are two questions we are always asked (see first paragraph for my answer).
BTW, does anyone else see a reflection of the “Dead Sea Scrolls” publishing controversy years ago? In that case, a small group of researchers kept the data close to their chests for DECADES, with the excuse being that “non-experts” didn’t know anything, and that experts who may have had a differing opinion simply weren’t as qualified. Eventually, the academic arrogance led to so many leaks, the wall was breached.
Here’s hoping…
snip – conspiracy is a forbidden word here.
This may have been mentioned before but I haven’t seen a reference to it. The Lord Chancellor has issued guidelines for the various exemptions to the FOI Act. There is specific guidance on using confidentiality as a defence given in:
http://www.justice.gov.uk/guidance/docs/foi-exemption-s41.pdf
Thre are other guidance notes for other exemptions. Now the Lord Chancellor’s guidelines aren’t law – but it would be a brave lawyer or judge and an extremely brave civil servant who would want to be caught defying them.
My quick reading suggests, it applies if CRU is a public body. Well I assume they are as they have taken previous FOI requests and either dealt with them or tried to find an excuse for not dealing with them. The Lord Chancellor seems very specific, that information can only be treated confidential – if breaching it would be actionable. It’s not good enough to say, the data was provided in confidence. The civil servant has to be able to show that if they revealed it – the public body might get sued (I’m simplifying things here).
I’m trying to imagine whether the Met. Office of a foreign country is really likely to start legal action against the CRU. Seems unlikely. And if they do, chances are, someone who reads Climate Audit will simply do an FOI or equivalent in that country. Or would the UK courts really want to end up suppressing scientific information in the UK because of a complaint from a country like Iran?
There is also a public interest argument mentioned in the guidance notes. This can only come into play, if the information was being kept confidential – because to break that would be actionable. The notes say that the motive for making the FOI request don’t matter. It doesn’t matter if Steve wants the information ‘to find something wrong with it’ or to provide interesting examples of statistical techniques -snip – – I’m saying it doesn’t matter about his motive).
What does matter is the public interest. However, it seems to me that to explain the public interest might require making stronger allegations than might otherwise be the case. The public interest is that science should be done properly. With science being open so that results of experiments and research can be independently verified or refuted. But that’s a bit vague and woolly and might not quite win the day.
snip
Re Craig Loehle #45″: “Its mine Disease”
Looks like that to me. A man starts a project 30 years ago, spends a whole career gathering data and publishing reports based on it. He probably had to “sell” his mission to numerous sponsors over time to keep his job alive.
About 25 years into the venture, his field gets top billing in the science world. There behind him sits a career of the level of work produced when the product is an intangible with no commercial feedback: It is as good as his personal drive compelled him to make it, and it has been steered by his funding sources – not maliciously or for the purpose of deception, but in the way a practical man approaches a career.
It seems to me that releasing his data would:
1) Eliminate a lot of his enterprise value.
2) Open his older work to increased criticism and attack.
I would not want a lot of my engineering work revieved by others. Sometimes projects just don’t work out for a while, sometimes my wife needed me to be less than 100% on the job because the kids drove her up a wall, and sometimes I just didn’t give a s***. If you’re reading Steve’s blog from work right now, you’re no better.
Dr Phil is not Dr Evil until he actually deletes the data, or tampers with the data to cover past errors or advance an agensa.
Oak Ridge National Lab was basically built around the X-10 Graphite Reactor. (X-10 was decommissioned in 1963 and is now a national historical site on the lab grounds.) ORNL reports primarily to the DoE Office of Science, though it also does contract work for other federal research agencies. There’s been various flavors of environmental research at ORNL since the mid ’60s, so it’s not entirely surprising that CDIAC landed there.
BTW, the Y-12 National Security Complex, which originally did electromagentic U235 separation for the Manhattan project, is still in operation, although I don’t know if they still do e-mag separation or not. Y-12 is part of the National Nuclear Security Administration, which is also part of DoE but separate from the Office of Science. Y-12 and ORNL are very much independent, and Y-12 There also used to be two other Manhattan project sites near Oak Ridge: S-50, which was demolished shortly after WW2; and K-25, which closed in 1987. The K-25 site is currently being cleaned up and converted into a tech park.
(Disclaimer: I work at, though not for, ORNL.)
Re: Troy Baer (#65),
I visited Oak Ridge in the 70s several times. There was a cute science shop where you could buy a block of graphite from the original rector (correction “reactor”) there, sold I guess to show that certain material from inside a reactor goes harmless in a short time. The other beaut purchase was the Great Scientific American Paper Aeroplane book, more for serious engineers than one might think. Ah for those days when Scientific American was dedicated to good science articles and Oak Ridge dedicated to fission production and enrichment. How sorry it is that they were sidelined.
Grant details (ER62601):
http://www.osti.gov/oberabstracts/detail.jsp?projectSerial=2002&query_id=5&searchpage=search.easy.jsp
Not bad for $180k a year…
New grant details:
http://www.osti.gov/oberabstracts/detail.jsp?projectSerial=5859&query_id=6&searchpage=null
Re: GF (#65),
Ok, thanks GF.
I see that the indexing is based on the ER62601 part.
Or, I didn’t do any of my searches correctly.
This is the highest probability. This is purely an empirical exercise, and you can’t add value through theoretical framework. So the value added is in data QC. Take that away, and there’s no value added. When people seek rent without adding value, they’re on thin ice, and know it.
I don’t think we’re going to find the evidence of Michael Jackson’s foul play in the data, either. But we may find something as simple as who shot Mr. Burns.
I suppose a letter to each of the countries in question informing them of the potential data breach and requesting clarification on the existence of the contracts might be in order.
Kevein Post #64
Wrong analogy and wrong conclusion.
Engineers create and make something. Scientists are supposed to analyse data and come to meaningful conclusions where engineers can make and create something. If an engineer spends years putting something together and if it works he submits a patent which is now on public record. If someone else wants to make that product then payment goes to the patent holder.
If supposedly a scientist has spent 30 years investigating something and then he writes a paper and comes to a conclusion. If he has confidence in his work he will let others verify his work which means his data and methods must be made public. The scientist will get credit for his work from others if he is correct and criticized if he is not correct. A scientist should not publish conclusions if he has not finished his research.
I think one of the reasons would be that scientists don’t want to face the possibility that they have wasted 30 years if they come to the wrong conclusions.
Re: Etienne (#80),
One of the things engineers get to do is science. In my business we worked for years to solve an optics problem. Our goal was to make a better product but we needed to solve a problem that science was working on. The solution had to work of course since we live in reality and us lowly engineers know the god of physics is a harsh mistress.
When we solved the problem, we verified for months before spending the many thousands for tooling. You can’t be wrong about these things when you’re not rich.
There is no way we’re going to reveal the method. We only patent the products. What we achieved is not non-scientific because our results are an improvement over science, yet all we did was engineering.
Could we publish? … Yup. Will we?…Nope!
It’s an odd argument about the patent system which applies to the data in HadCrut. In patents you reveal your results and must teach the method for replication, in exchange for that you get the right to some very expensive ($500K minimum) government protection to sue others who replicate what you just taught them. In the case of Jones, he’s claiming he can publish the result yet cannot reveal the method or data for compilation due to confidentiality. Has anyone FOI’d the method?
In the case of government funded science which apparently cannot be shared, shouldn’t there be some form of the same thing? You can publish if you don’t profit or in the case of confidentiality agreements you can’t use the data if you profit?
Whatever the answers, the guy should release the data. It’s insane to require us to pay taxes for an invisible (and small) set of numbers.
Yes. That was irreverent. The irreparable damage done by not citing that indicates irreflectiveness. Some people are just irreducibly irredeemable. That’s so irritating.
Steve Mc : Could you link back to the workshop or symposium where some of the team members emphasized the importance of improving the public availability of climate data. I don’t remember the correct key words to find it in your archive.
Steve: Maybe this one: http://www.climateaudit.org/?p=3259. Also check the left frame Category: Archiving.
Jeff Id #74
I agree with you and also in my humble opinion engineers are more useful then scientists anyway. The results of what you built worked and isn’t that supposedly the end result?
Maybe this will help:
http://www.whitehouse.gov/omb/rewrite/circulars/a110/a110.html
Re: Corey S. (#79)
If the EPA at all relies on Santer et al 2004, DOE Contract DE-FG02-98ER62601 in their CO2 Endangerment Finding and subsequent ruling, then that clause would come into force and the data be obtained via FOIA. Similarly if the Waxman and Markey H.R.2454 Cap and Trade (Tax) bill becomes law.
Searching EPA Under Climate Change for “Santer” brings up: [1] DRAFT 7/2/0704-17-2009, April 17, 2009 Technical Support Document, U.S. Environmental Protection Agency . . .
Wigley, T.M.L., V. Ramaswamy, J.R. Christy, J.R. Lanzante, C.A. Mears, B.D. Santer, C.K. Folland, 2006: Executive Summary in Temperature Trends in the Lower Atmosphere: Steps for Understanding and Reconciling Differences. T. R. Karl, S. J. Hassol, C. D. Miller, and W. L. Murray, editors. A Report by the Climate Change Science Program and the Subcommittee on Global Change Research, Washington, DC.
So any Santer data relied on, included, or cited in Wigley 2006 may be applicable.
——————————-
PS Re: Etienne (#83),
If you have the investment and keep the technology secret, then your method may be used. Otherwise, if the inventor “knows” that the invention will work, he may well file a provisional patent first, and then use that provisional patent to raise the commercial funds to build the prototype. See Patent It Yourself David Pressman etc. Detailed modeling can provide the confidence that it will work.
It would seem that a FOIA (U.S. law) request to the DOE for data prepared under the contract would be productive. Research funded and published under the grant has been used in the IPCC reports, which in turn were relied upon by the EPA in the action of an endangerment finding. This would make the data created and published by Jones subject to the provisions of 36(d)1 of the A110 Circular.
Etienne “If an engineer spends years putting something together and if it works he submits a patent which is now on public record. If someone else wants to make that product then payment goes to the patent holder.”
Just the reverse. The patent is submitted before any work is done to put anything together (often by a research department populated by PhDs of infintesimal put-something-together skills). If someone else wants to make that product then payment goes to the patent holder’s corporate thought harvester. Sometimes we get a few hundred bucks and a plaque.
I still believe the analogy and conclusion are sound. It is a human nature argument, not a technology argument.
He spent a lot of time gathering data, and he’s not going to give it away without a fight, especially not to someone who would use it to attack his past work.
Calvin Ball aptly applied the label “Rent Seeker”.
Uh-oh. I think I stumbled upon the big secret. Check out the first name that he trades for a number (at about 0:50):
The plot thickens…
Kevin #81
Uhm no. If an inventor invents something he will design and build a prototype and if it works he will patent it.
Jeff Id,
Agree that research paid for with tax money should be available to the public absent national security concerns. But that is a side issue in the big public policy debate. The key issue is whether public policy should ever be based on research which cannot be audited or replicated. The answer should be a resounding no. Doesn’t matter who funded the research. If the public cannot scrutinize the research for problems, it isn’t fit to inform public decisions.
If scientist Smith wants to claim that he has discovered fact X but is unwilling to subject his work to scrutiny, fact X should never be accepted as true by governmment policy makers. Fact X remains a mere hypothesis, an untested theory, which has not been proven — regardless of who paid for scientist Smith’s work.
I was interested in how much money had been expended on grant DE-FG02-98ER62601 . No one seems to have made any progress on this. OFten grant amounts are open to the public. Maybe someone could write DOE and inquire how much money’s been spent under this grant and obtain the terms of the grant.
Re: Steve McIntyre (#88),
It looks like GF did with the URLs in this comment.
About 180k USD per year for 10 years.
I know that IP is a frowned-upon subject, and don’t want to go there, but Re: Etienne (#83), it’s not anywhere near that simple. Nuff said.
Steve,
Please see my comment at #85 and the comment at #66. They provide a link to all info on this grant including dollar amount, recipients, papers published, etc. #85 also goes into DOE/BER data sharing guidelines.
Steve
I couldn’t find the grant yet, but it seems it would have been a response this request:
http://www.er.doe.gov/production/grants/fr98_15.html.
It’s closed at the DOE grants page, but the request does provide the requirements for responses to the grant request.
Kayle
2 Trackbacks
[…] couple of other posts at the time of the Mole incident were here and Dr Phil, Confidential Agent , in which I observed that Jones, as a temperature accountant, was sort of the climate equivalent of […]
[…] of Energy (along the lines of a scenario that I contemplated in a CA post of Aug 4, 2009 entitled Dr Phil, Confidential Agent): In regards the information provided to the US Department of Energy, my investigation has revealed […]