The FOIA/Mole incident of July 2009 attracted much public interest and somehow seems connected to the subsequent Climategate events, though precisely how (and even whether) remains unclear. The incident was discussed in both Mosher and Fuller’s CRUTape Letters and Fred Pearce’s Climate Files, though not in Andrew Montford’s Hockey Stick Illusion (which was mostly complete prior to Climategate.)
Climategate 1.0 emails provided relatively little context on the CRU/East Anglia side of the incident. Climategate 2.0 changes this situation dramatically, as it contains a rich (though still, at times, frustratingly incomplete) documentation of the university’s side of events. (The Climate Audit side is, of course, exhaustively documented in contemporary posts and comments, starting here and continuing in threads for the next month or so.)
The FOIA/Mole incident has been central in self-serving rationalizations of Climategate events by institutional climate science. They’ve represented the affair as an attempt to “harass” climate scientists, claiming that the events vindicated many prior years of data obstruction. However, institutional climate science and their house organs (e.g. Nature) have failed to report the origins of the incident. Its proximate cause was an institutional mendacity incident. In July 2009, CRU had refused a FOI request for station data with the flagrantly untrue claim that language in their alleged confidentiality agreements prevented them from sending station data to “non-academics” (while not preventing them from sending the same data to sympathetic academics:
Regulation 12(5)(f) applies because the information requested was received by the University on terms that prevent further transmission to non-academics
This assertion was completely fabricated: there were no agreements that contained the claimed language. Climate Audit readers quickly recognized that the university’s assertion was fabricated and submitted FOI requests to the university for the supposed confidentiality agreements containing the alleged language. The university was unable to produce a single agreement with such language and quickly abandoned this line of refusal in favor of other lines of refusal – these new grounds of refusal still, however, relying on faith in agreements unseen.
The new Climategate 2.0 emails shed fresh light on the process leading to East Anglia’s fabricated claim about language that specifically prohibited the transmission of station data to “non-academics”. The emails show that the untrue claim was not made casually or by accident, but was carefully crafted through a process that included not just Phil Jones and FOI officer David Palmer, but also Michael Mcgarvie, UEA’s Director of Faculty Administration. It also seems unlikely that Mcgarvie would not have reported to his superiors in the Registrar’s office and beyond, but these reports (if they exist) are not in the Climategate dossier.
Although the university’s mendacity in this incident was one of the flashpoints that seems to have precipitated Climategate, it was not investigated by the Muir Russell “inquiry”. To do so would have required examination of Climategate 2.0 (and other) emails. Muir Russell made no effort to examine such emails until late in the review process. These late and feeble efforts were then resisted by Acton and the university administration and nothing came of them. Muir Russell nonetheless proclaimed in his press conference that, while CRU was “unhelpful” at times, the “rigour and honesty” of the scientists in question was “not in doubt”.
In preparing today’s review, I’ve made my own “edition” of the emails in which I’ve arranged threaded emails in chronological order. See here – link to follow.
I’ll start today’s chronology of events somewhat in the middle.
Contrary to a common view, prior to mid-2009, I’d never made an FOI request for the CRUtem data set. As I’ve said on many occasions, my interest in temperature data is very secondary to my interest in proxy reconstructions and I’ve been well aware that time spent on it would pre-empt topics that interested me more and which seemed more scientifically contentious.
However, I firmly supported the principle that data being relied upon by IPCC should be transparent and I had written about efforts of Warwick Hughes and Willis Eschenbach with considerable interest and had done what I could to support their efforts. In late 2006, Willis had initiated the first use of FOI legislation to overcome a non-transparency that the wider scientific community should itself have objected to. After much obfuscation by CRU, Willis managed to extract a list of stations, but not the station data as used by CRU. The progress was reported from time to time at CA.
In May 2009, I noticed that the UK Met Office held a version of the CRU station data, which was cited on one of their webpages. As someone with some familiarity with confidentiality agreements and (by then) FOI regulations, it occurred to me that FOI provisions applying to the Met Office (a government agency) might differ in some respect from the FOI provisions applying to the University of East Anglia. For example, CRU might have sent station data to the Met Office without a confidentiality agreement between CRU and the Met Office. If so, the Met Office might not be able to rely on exemptions available to CRU. In fact, this surmise proved correct though the Met Office refused anyway. Climategate 2.0 emails (not ones reviewed today) show that the Met Office understood this point and looked long, hard and unsuccessfully for such an agreement, which appears not to have existed.
In June 2009, on a blog post discussing my Met Office request, Peter Webster of Georgia Tech commented that he had readily obtained station data from CRU earlier that year (Webster’s comment was quoted by Jones in 4531):
We have asked Phil Jones for data so that we could compare the synthesized surface temperature with actual station data. Jones has provided everything that we have asked for. This is for our study of the 1930/40 climate bump that is ongoing. Alas, these things take time. But my experience has been quite different to yours.
As you know, I have often complained that the right wing and the left wing (the absolutists of AGHW and those who do not have a bar of it) have forced us into corners in which we are not comfortable. If there is to be reasonable resolution of the climate GWH issues and the fidelity of data (both critical and reasonable questions?) I think that the questions and opinions can’t be shouted from one corner or the other…
I immediately (June 25) sent a FOI request to East Anglia for the precisely the same data (4531).
CRU Plans the Refusal
The following day (June 26), Palmer forwarded my request (4531) to Jones, Mcgarvie and Osborn as follows:
A request from Mr. McIntyre under EIR that arrived today. Response due by 24 July. I have acknowledged the request and confirmed that we will be handling this under EIR. Any concerns with this request? Any need for clarification?
Cheers, Dave ______________________________________________________________________________
From: Steve McIntyre
Sent: Friday, June 26, 2009 4:45 AM
To: Palmer Dave Mr (LIB)
Subject: Environmental Information Regulations
Dear Mr Palmer,
Pursuant to the Environmental Information Regulations, I hereby request a copy of any digital version of the CRUTEM station data set that has been sent from CRU to Peter Webster and/or any other person at Georgia Tech between January 1, 2007 and Jun 25, 2009. Thank you for your attention,
Jones immediately explained (June 26 – 4531) to Palmer (cc McGarvie, Osborn) that he sent the data to Georgia Tech in a “personal email”, asserting that “McIntyre has no right to request the data in a personal email”:
I sent some of the station data to a Jun Jian at Georgia Tech on 15 Jan 2009. I see now that Peter Webster was a recipient on the email. I also see from looking at Climate Audit that this request results from Peter saying on CA that he’s not had any difficulty getting data from CRU (see what he said below on June 24).
I regard this as a personal email between me and this group at Georgia Tech. So, McIntyre has no right to request the data in a personal email. I only sent a small part of the dataset anyway. They asked for a specific set and said what they were going to do with the data.
That an IPCC Coordinating Lead Author could regard distribution of CRU station data as “personal” favor rather than an obligation is an issue that the climate community has closed its eyes to.
Palmer rather gingerly explained (June 26 – 1320) to Jones (cc Mcgarvie, Osborn) that data did not become “personal” merely because Jones said so and that they needed a “valid” exemption to avoid disclosure. Palmer observed that Jones’ already sending the data to a third party presented problems; Palmer said that he was “having difficulty” making a case for not sending the data to me that had already been sent to Georgia Tech, a misgiving that he and others should have paid more attention to:
Phil [et al],
The fact that information is within an email that you consider ‘personal’ does not render the information itself personal. In order to not disclose information under EIR, we need to have a valid exception, and then also pass a public interest test that shows that the public interest is better served by non-disclosure than disclosure.
I will have a think about what exceptions are available to us, but, at this moment I am having difficulty making a case for any that would apply here. The other issue is passing the public interest test – we would, I presume be relying on some sort of public interest in preserving the confidentiality of communications between academic colleagues but there is no guarantee that the ICO would uphold this.
I will get back to you next week on this one….
Later that day (June 26 – 2663), Mcgarvie worried that providing data would set a bad precedent for other cases, particularly David Holland’s pending appeal of their refusal of IPCC correspondence not in the IPCC archive. Mcgarvie recommended that they invoke one of the exemptions that they had discussed – but only after stretching the time period as long as possible:
Dave et al,
As we are testing EIR with the other climate audit org request relating to communications with other academic colleagues, I think that we would weaken that case if we supplied the information in this case. So I would suggest that we decline this one (at the very end of the time period), with one of the valid reasons that you, Jonathan [Colam-French] and I disucssed, and let him go through appeal.
Happy to discuss further (but not for a couple of weeks since my diary is pretty full next week and the week after).
Director of Faculty Administration
Jonathan Colam-French, who was responsible for hearing appeals, was involved from time to time in the initial strategy on how to avoid compliance.
A few days later (June 30 – 2663), Palmer reminded Mcgarvie and Jones that refusing station data presented other difficulties relative to refusing emails. Palmer noted that there was no general academic principle opposing the exchange of data – indeed, Palmer observed that archiving of data was even regarded as meritorious as some parts of the academic community – a further point to overcome in constructing a refusal:
I can understand your reluctance to deal with Mr. McIntyre’s request but we do need to have justifiable grounds for claiming an exception under the EIR in order to do so…
To address your point Michael, I think that there might be a difference seen between personal correspondence between academics and actual data which has a life/role outside that correspondence. In regards the public interest test that we have to address, once again, I would think that whilst there is a good argument for protecting the ability of academics to communicate freely and openly, the underlying data that may comprise part of that communication might well fall into another category. One only has to look at the JISC funded projects on national scientific data repositories and exchange to see that there appears to be a perception in the academic community that the exchange & re-use of data is a good thing.
We also have to remember that, much like FOIA, the exception regarding ‘confidentiality’ is in relation to a person providing the information to the organisation – it does not touch upon correspondence from the organisation. That is covered either by ‘internal communications’ exception, or as in the other case with the IPCC, an ‘adverse effect’ on international relations (which I believe to be entirely justifiable)
As you are both quite busy over the next couple of weeks, I would be happy to discuss this further w/c 13 July with you, Michael and verify our approach the following week prior to the deadline of 24 July.
Jones then searched EIR legislation for exemption ideas. Later that day (June 30-1473), he suggested three ways of refusing the data, one of which was exemption 12(5)(f) – confidentiality. Jones told Palmer and Mcgarvie that “some of the data was supplied to CRU on the grounds that we didn’t pass it on”. (Here Jones was conspicuously silent on how CRU circumvented these supposed conditions when it recently passed data on to Georgia Tech and, for that matter, when it passed data on to the US Department of Energy which then created a public archive of station data as it stood in the early 1990s.)
I’ve done something I thought I would never do – I’ve printed off the EIR for 2004!
Here’s a few thoughts.
1. I don’t have the exact data that I sent in January 2009. I’d have to recreate it. The data are part of a larger database. What I’d recreate would be different from what I sent in Jan 2009 (12.4a).
2. The requester has no idea what I sent on January 2009 (12.4c).
3. If I do have to recreate it, then it will contain data where 12.5fi-iii apply. Some of the data was supplied to CRU on the grounds that we didn’t pass it on. These conditions were put on it by some of the National Met Services around the World (including the UK).
On a related matter and back to Michael’s point. The next IPCC process will start in 2010. It is possible that UEA people will be involved in the author writing teams. The members of these teams will be available through IPCC. What is to stop people asking for emails I might write to some or all of these authors. This, in effect, is the purpose of the appeal in the other issue with Keith and Tim and IPCC correspondence.
Palmer wrote back to Jones and Mcgarvie (June 30 – 1473), rejecting Jones’ first two excuses but encouraging the third, with Palmer welcoming Jones to the “‘dark side’ of FOI”. However, Palmer pointed out that confidentiality agreements in themselves would not suffice – they would still have to “overcome the obvious fact that some data was passed to a fellow academic so therefore would need to draw a distinction between that type of disclosure and that requested by the applicant”. This, of course, was the nub of the problem.
Ah, now we are getting somewhere…. (and I will turn you to the ‘dark side’ of FOIA/EIR yet! lol)
I don’t’ think Reg. 12(4)(c) will hold water as I think we know exactly what he is asking for but it’s our ability to provide it that is at issue. However, even if we think it is too general, Regulation 9 mandates us to provide advice and assistance to the requester and indeed, subsection (2) specifically notes that if we do feel that 12(4)(a) applies, we must ask the applicant to provide more particulars & to assist the applicant in providing those particulars. I would think it likely that the applicant in this case would simply ask for the entire base file….?
As to point 1 below, if we don’t have the original email nor any record of what was sent, then there may be a case for the application of 12(4)(a). However, being contrary (and that’s part of the job description), Regulation 9 would also raise it’s hoary head here and we would need to tell the applicant out problem in ‘reassembling’ the data. I suspect the outcome would be exactly the same as above; namely a request for the entire base data.
However, point 3 has definite promise. We would have to demonstrate an adverse effect on the interests of the party providing the information/data, and then pass the public interest test, overcoming the presumption of public interest in disclosure. Clearly, if the data was given to us on terms that forbade its further disclosure to persons/instructions that would exclude the applicant (and we have evidence of that), then we can also assume some presumption of adverse effect (although once again, thinking ahead, evidence of this would be useful).
We would have to overcome the obvious fact that some data was passed to a fellow academic so therefore would need to draw a distinction between that type of disclosure and that requested by the applicant.
I do not disagree with your final point which is why I was drawing the potential distinction between data and private correspondence. Our case before the ICO is all about the confidentiality of information coming to us and the adverse effect its disclosure would have on the persons providing it, and the international relations we have with bodies such as the IPCC. (and I think we might have a case under EIR ‘manifestly unreasonable’ grounds as that definition is wider than that for ‘vexatious’ requests under FOIA).
Climategate 2.0 emails are quiet on developments for the next two weeks. (Presumably further information is in the locked tranche.)
The curtain rises again on Thursday, July 16 (and this passed unnoticed at CA at the time) when Andrew Montford sent an FOI request (2786) presciently requesting copies of the confidentiality agreements governing the station data. Montford’s request did not refer to my pending FOI request, though that was obviously in the air, but referred back to claims that Jones had been in correspondence with Keenan in 2007. Palmer notified (2786) Mcgarvie and Jones of this new development and proposed adding this to the agenda of a meeting on FOI that they had scheduled for Monday July 20.
At the July 20 meeting, Jones, Mcgarvie and Palmer appear to have agreed on the refusal strategy employed in the July 24 refusal letter a few days later: to reject the FOI request on the (untrue) basis that the supposed confidentiality agreements contained language that prohibited the sending of data to “non-academics”. On June 30, Palmer had been concerned about how to justify not sending data to me when the same data had been sent to third parties at Georgia Tech. It was shall-we-say little short of miraculous that Jones (presumably) reported to Palmer and Mcgarvie that his confidentiality agreements entered into so many years ago contained language dealing with the situation, language that prohibited CRU from sending data to “non-academics”.
Palmer appears not to have challenged Jones’ miraculous claim or carried out the mundane due diligence of examining the supposed agreements (or else events would have unfolded differently). On July 21 (3334), Palmer distributed a draft refusal to Jones and Mcgarvie as they had discussed at the meeting (presumably along the lines of the July 24 refusal). In the covering email (3334), Palmer made a last plea to Jones and Mcgarvie to consider the (logical) alternative of simply providing me the data under conditions that prevented my re-distribution (rather than refusing):
A draft response along the lines discussed yesterday. I would expect an almost immediate appeal of this decision by Mr. McIntyre.
Phil, as your concern is the publication of the requested information, I wonder if a possible alternative is to release it but place conditions on it’s use. This will ONLY work if UEA has some rights in the data itself or in the database. ‘Copyright’ in the contents of a database would require some personal creative input by ourselves to the data or database that would render it different from preceding external versions and ‘original’.
However, even if the contents aren’t ‘original’, there is a ‘database right’ where the contents of the database are assembled as the result of substantial investment in obtaining, verifying, or presenting it’s contents. It is the framework, not the contents, that attracts the rights. These rights exist for 15 years from the completion of the database BUT any substantial change to contents will ‘renew’ the database rights for another 15 years. The owner of database rights has the right to prevent the extraction or reuse of all or a substantial portion of the database.
There is ‘fair dealing’ in database rights to the extent that anyone has a right to extract & reuse an insubstantial portion of the database (not really defined in law but it’s very small) for any purpose, or where the portion is substantial, extract and use data for non-commercial research or private study. What can’t be done is re-issuing this information to the public under a different guise.
The upshot of all of this is that, if we have a ‘database right’ in this information, then we can release it BUT insist on our exclusive right to re-use the information – BUT the issue is actually ‘enforcing’ those rights…… more difficult in practice than in law or theory….
Just thought I would proffer this as an option in place of the refusal and the inevitable appeal.
Palmer’s sensible suggestion was rejected by Jones (780, 3712). On July 24, Palmer sent out the refusal (untruthfully) claiming that language in their confidentiality agreements prevented the sending of data to “non-academics”:
Regulation 12(5)(f) applies because the information requested was received by the University on terms that prevent further transmission to non-academics
As noted above, CRU’s claim about contract language pertaining to “non-academics” was fabricated. As matters would show, there were no agreements. Presumably their strategy was simply to force me into a lengthy appeal process which I might not pursue (a tactic that had been successful with Eschenbach a couple of years earlier.)
However, it’s one thing to be “unhelpful” and quite a different thing to make fabricated claims about their so-called confidentiality agreements. As someone with actual experience with real confidentiality agreements (in the mineral exploration business), I didn’t believe East Anglia’s excuse for a minute. It was inconceivable to me that CRU’s “confidentiality agreements”, even if they existed, contained the claimed specific language about “non-academics”. I immediately (July 24) responded at CA here as follows about this surprising development:
This is the first time that we’ve heard that their supposed confidentiality agreements merely restrict “further transmission to non-academics”. A couple of observations on this. I’m sure that CRU will soon receive a similar request from someone to whom this excuse does not apply.
However, aside from that, there are other troubling aspects to this refusal. If there actually are confidentiality agreements, I would expect the relevant language to be framed in terms of “academic use” as opposed to guild membership i.e. I’d be surprised if the language were framed in terms of institutional affiliation as opposed to use. I’ve published relevant articles in peer reviewed literature, acted as an IPCC reviewer, been cited in IPCC AR4, been invited to present to a NAS panel – my use of data is “academic” by any legal standard.
Secondly, over at the Met Office, they say “it cannot be determined which countries or stations data were given in confidence as records were not kept.” But over at CRU, they purport to “know” nuanced details of the contractual language of the confidentiality agreements – clauses that have the effect of justifying the refusal of the data.
I and other Climate Audit readers were offended not simply by CRU’s refusal to provide data, but by the arrogance of an institution that had no compunction about fabricating excuses for non-compliance. Requesting the alleged confidentiality agreements containing language about “non-academics” was an obvious next step. Recognizing that a request by me as an individual was likely to be marginalized, I invited Climate Audit readers to show the university that others were concerned about CRU obstruction by submitting their own FOI requests for confidentiality agreements. Quickly Climate Audit readers submitted nearly 60 FOI requests for the so-called confidentiality agreements, each reader specifying five different countries. In addition, a number of what East Anglia called “legit academics, rather than the usual suspects” submitted FOI requests for the The data set that had been refused to me.
In effect, Climate Audit readers had staged a protest against institutional mendacity.
Considerable interest was added to the protest by the “Mole Incident” (see here) and CRU’s subsequent deletion of numerous files from their FTP site, including two different versions of the supposedly top secret station data that had been online since 1996 and 2003 respectively. (Yet a third version was online at the US Department of Energy since 1991, CRU having sent the data to the US Department of Energy who had funded them.)
However, those are stories for another day. Today, I”m going to follow the gradual realization by East Anglia administrators that there were no confidentiality agreements containing language specifically restricting the transmission of data to non-academics and their seamless and unapologetic change of argument to one that was less transparently untrue.
On July 28 (490), Palmer outlined a strategy for the various FOI requests. Palmer recommended that they place “any/all [confidentiality] agreements” on the CRU website, while continuing to oppose requests for data. At this time, Palmer was presumably unaware that Jones’ “confidentiality agreements” would not be producible or that they would contain no language restricting access to “non-academics”
Just to summarise our approach to the various requests we have received to date that we have agreed:
A. ‘Country’ requests
1. Respond to Montford request as normally – cite s.21, information available (see point 2)
2. Place any/all agreements (or links thereto – Met Office) on the CRU website
3. Acknowledge & respond to all 44? Country requests by citing s.21 and pointing them to the CRU website
B. Data requests
1. Acknowledge requests
2. Deal with as per normal, cite Reg. 12(5)(f) re agreements and Reg. 12(4)(b) ‘manifestly unreasonable’ on the grounds that the data is already available publicly via the ClimateAudit.org website, and note that a format of the data (gridded) already is publicly available
3. Note that raw data is available from the Met Office and other national weather services (also goes to ‘manifestly unreasonable’)
.C. McIntyre appeal
1. Maintain position regarding Reg 12(5)(f) re confidentiality agreements and point him to published versions on website
2. Add ‘manifestly unreasonable’ on basis that he already has the requested information in his possession & is also available elsewhere
3. Handle as per published protocols with initial ‘informal’ approach, followed by review by JCF
D. ‘Other’ requests
1. Acknowledge requests
2. Deal with as usual, citing whatever section is appropriate above to the requested information
E. General points
1. Interaction with any media to be handled by Press Office
2. Approval of transfer to Georgia Tech would be good to find
3. We are NOT citing s.14 for the ‘country’ requests
4. Estimated time to locate ALL agreements regarding data transfer is within the 18 hour appropriate limit
5. Any correspondence to go out will be circulated prior to transmission
I hope I have captured what was agreed – please comment if your understanding is different than mine
The next day (July 29 09:49), Jones agreed (1131) to Palmer’s proposal that the “agreements” be placed online:
Here’s what I propose to do over the next week or two. I will get the agreements scanned and write some text about them and the others that we have had.
A scan of four agreements (none of which pertained to station data) was completed by 11:47 on July 29 (see the document properties of the document placed online on August 11). Later in the day (July 29 17:19 1248902393.txt), in a Climategate 1.0 email, Jones told Peterson of NCDC in the U.S. said that he planned to deal with the FOI requests with a “small [webpage] document”:
Anyway enough of my problems – I have a question for you. I’m going to write a small document for our web site to satisfy (probably the wrong word) the 50 or so FOI/EIR requests we’ve had over the weekend. I will put up the various agreements we have with Met Services.
Jones also consulted (July 30- 1916) with Peter Thorne of the Met Office, who strongly discouraged any attempts by Jones and/or East Anglia to be “opaque” in their responses, a recommendation that East Anglia unwisely disregarded:
Generally the more opaque this is the more work eventually will accrue for you (and us, but mainly you). If we’re forced to wash our laundry we may as well use Daz brilliant white … so be inanely pedantic in the details so that you minimise the angles for attack. It will take longer right now but I suspect in the long run be less work which means you can do science if you can still remember what that is (I vaguely do myself …).
Thorne also recommended that Jones try to locate past evidence of his “championing open access” – not an easy task given Jones’ actual views and prior conduct:
Is there any documentation in GCOS reports we can point to, especially AOPC where we can show that you are championing open access? That would shut them up slightly so is VERY DEFINITELY worth looking for and linking to as it shows you in the correct light. I think some text showing taht CRU is fighting for openness in relevant forums would be a gold dust addition.
By the first week of August, the Climate Audit protest, in combination with the Mole Incident and CRU’s deletion of website documents, had attracted wider attention. Between August 7 and 10, Olive Heffernan, a reporter with a journal (Nature), that sympathized with the trade interests of the climate community, carried out an email interview with Phil Jones, which I’ll discuss on another occasion.
The webpage contemplated in late July was published on August 11 here, prompting an immediate response at Climate Audit here. CRU’s webpage did not include ANY confidentiality agreements concerning station data. Instead, they provided two request letters from 1993-94 for 1961-90 climate normals for 9 variables (to the UK Met Office and Spain) and two response letters regarding 1961-90 climate normals (Bahrain, Norway) – only Bahrain within the tropical zone of the Georgia Tech data. CRU stated that they were unable to locate any actual agreements because they “moved offices”:
Below we list the agreements that we still hold. We know that there were others, but cannot locate them, possibly as we’ve moved offices several times during the 1980s. Some date back at least 20 years. Additional agreements are unwritten and relate to partnerships we’ve made with scientists around the world and visitors to the CRU over this period.
CRU claimed that the language in these letters was representative of the supposed “agreements” that they were unable to find:
Since the early 1980s, some NMSs, other organizations and individual scientists have given or sold us (see Hulme, 1994, for a summary of European data collection efforts) additional data for inclusion in the gridded datasets, often on the understanding that the data are only used for academic purposes with the full permission of the NMSs, organizations and scientists and the original station data are not passed onto third parties. …In some of the examples given, it can be clearly seen that our requests for data from NMSs have always stated that we would not make the data available to third parties. We included such statements as standard from the 1980s, as that is what many NMSs requested.
Even if the documents were to be construed as “actionable” confidentiality agreements (which they weren’t), they contained no language about “non-academics”. To the extent that there were any restrictions, they applied not just to “non academics”, but to all third parties, including not just Georgia Tech, but the US Department of Energy, Mann and Rutherford and perhaps even the Met Office.
CRU also volunteered the embarrassing information that they no longer held the original “raw” data, only “value added data”, (laughably) attributed to limited storage capacity in the 1980s. (This admission provoked much satire and independent criticism):
Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e. quality controlled and homogenized) data.
CRU’s embarrassing document was immediately criticized at CA here. Willis Eschenbach, whose 2007 FOI request for data had been stonewalled, wrote in angrily (Aug 11 – 752) asking that the 2007 refusal be reconsidered, pointing to CRU’s failure to produce a single confidentiality agreement restricting the station data. Jones defended the webpage to Palmer (752), claiming that there were many other faxes that contained restrictive language that he hadn’t included on the webpage because they said the same thing:
He has missed the point. I could have put in loads of the faxes similar to the British Territories one, as all the requests that Mike Hulme and I sent in the mid-1990s included the statement :
The data will not be used unauthorised for any other project and will not be passed onto any third party.
I didn’t include all of these as they just say the same thing. I only included those that reiterated this point when they sent us the data.
This, of course, was inconsistent with statements in the webpage that they had just published. It stated unambiguously that they had listed “the agreements that we still hold.” If Jones had other agreements saying “the same thing”, then he should have said so in such a sensitive document.
UEA is not promoting this dataset as a suitable basis for making billion-dollar decisions on what we should do on regarding the ‘global warming’ supposedly shown by your dataset. This is simply NOT TRUE.
I presume that Palmer must have felt sandbagged when he saw that none of the extant “agreements” contained the language to which Palmer had affixed his name in the refusal letter. Most “non-academics” would have sent Jones a strongly worded WTF letter asking for an explanation. No such letter from Palmer is in the Climategate dossier thus far. Perhaps there’s one in the unopened dossier, perhaps Palmer never sent one.
In any event, East Anglia never again mentioned their fabricated claim about restrictions against “non-academics”.
Instead, East Anglia adopted a new line of refusal. They gave up trying to walk the tightrope between refusing data to potential critics, while giving data to fellow travellers. They now argued that Jones’ still unseen “confidentiality agreements” prevented access not simply to non-academics, but to third parties in general. They now said that the data had been given “in error” to Georgia Tech and that the “same error” should not be repeated. This line was used to refuse requests for data from Ross McKitrick, Jonathan Jones and Don Keiller (and my appeal), using language along the lines of the following:
We do concede that information was provided to Georgia Tech without securing consent of the institutions that provided it, and, upon reflection, this is an action we would not choose to take again. However, having made one error does not, in our eyes, justify making the same error again.
In fact, if there ever were such confidentiality agreements, then CRU had breached them right from the start – by sending the 1991 version of the data to the US Department of Energy which published the station data online; by placing the 1996 version online at CRU as part of the ADVANCE/10K program; and by sending station data out on request (not just to Georgia Tech, but to others, including Mann and Rutherford in 2005 and even to me in 2002 before I was identified as a potential critic).
The inconsistencies in CRU’s position were clearly set out as part of my appeal (Sept 2 – 2929):
I am in possession of three earlier versions of the CRU station data. The 1990 version has been posted at a US Department of Energy website for many years. In September 2002, I requested a copy of this data from Dr Jones. He sent me a 1996 version (cruwlda2) – a version that was also posted at the CRU website until recently – and indicated that the then revised version would posted up when Jones and Moberg (2003) was published, which, according to the date-stamps at your FTP site was done in Feb 2003, as Dr Jones had undertaken to do (the data set newcrustnsall recently removed from your public directory).
Notwithstanding this, when Warwick Hughes and Willis Eschenbach requested station data, for some reason, CRU failed to provide then with this information. As you noted in your letter, following your recent refusal to provide station data to me, I examined the CRU FTP site and determined that the newcrustnsall was the version of the station data for Jones and Moberg 2003 that Dr Jones had previously undertaken to post up on the Internet.
While these data sets are of interest, my request was for the current version of the data set and I do NOT wish to withdraw my appeal of your ruling. Could you please advise me of Mr Colam-French’s email so that I may submit further particulars of the basis of my appeal. In the mean-time, I would appreciate it if you reflected further about the apparent contradictions in your present refusal, given Dr Jones’ previous provision of an earlier version of the dataset to me, the posting of two versions of the data set on your website, one from 1996 to the end of July 2009 and the other from 2003 to the end of July 2009 and the provision of a version to the US Department of Energy, where it has been posted on the internet since 1990.
Jones’ reply (2929) to Palmer was incoherent. Jones now said that he didn’t “recall all the facts from that long ago”. Jones admitted that there was a version at the DOE website and even that this was a “contractual requirement at the time” – not explaining how a contractual requirement to make the data available was consistent with the confidential agreements that he had supposedly entered into with the NMSs:
I don’t recall all the facts from that long ago. There is or was a version on a US Dept of Energy website from about 1990. This was a contract requirement at the time. Much extra data has been added since then, and this is what the restrictions refer to from the mid-to-late 1990s.
Jones’ “explanation” of cruwlda2 being on the CRU website since 1996 was no more coherent. Jones said that it was on their website because this was “easier” than sending disks to the parties – again inconsistent with supposed confidentiality. Jones (falsely) stated that the dataset wasn’t complete(it was):
The 1996 version (cruwlda2) wasn’t a complete version and was something we developed for a number of people in EU projects to use. We made these available to people on these projects via our ftp site, as it was easier to do this than sending disks at that time (email attachments were smaller then).
Jones (falsely) added that he didn’t “have a copy of that file” – this is only a few weeks after he’d ordered the removal of the file from the FTP site during the early August carnage.
Jones’ attitude to the supposed confidentiality agreements is further evidenced in an email to Palmer and Mcgarvie (Sep 23 – 2840) about Jonathan Jones’ FOI request in which he said that they were “quite adept” in the past at getting around any conditions set by NMSs in the alleged confidentiality agreements:
The other issue is that Met Services putting conditions for the use of the data was common in the mid-1980s and 1990s. We were just quite adept at getting around the conditions.
None of Jones’ admissions caused the university to reconsider their refusals. It fought appeals by Jonathan Jones and Don Keiller at every step. Last summer, nearly two years after the original request, the ICO rejected the original UEA refusal. And, despite CRU’s serial violation of the supposed confidentiality agreements now said to be so important, the university closed their eyes to these prior violations rather than reprimand CRU.
Update Dec 31, 2011: In the university’s rejection of my appeal, they conceded that the reference to “non-academics” was an “error”:
In response to your first point in your email of 24 July regarding the non transmission of data to non-academics, I have concluded that the reference to non-academics was in error and that the status of yourself, or any other requester, is irrelevant to the factors to consider regarding disclosure of the requested information.
In response to comments about previous availability of the data set, they stated:
Turning to the points you raised in your email of 2 September, you note that other earlier versions of this data are available on the US Department of Energy website and that Dr. Jones had sent an earlier version of the data to you and had mounted it on FTP server.
In regards the information provided to the US Department of Energy, my investigation has revealed that this was done in the early 1990s prior to the imposition of the restrictions now pertaining to the data pursuant to a contractual obligation at the time. Therefore, the analogy you are drawing does not apply to the data that is the subject of this request.