CRU Erases Data

Jul 29, 2009 – 5:52 AM

As readers know, both myself and various Climate Audit readers have requested CRU station data from both the Met Office and CRU. While my initial request to CRU was refused, I asked for a reconsideration and the matter is still outstanding, as are all the other CRU requests. Under U.K. Freedom of Information Act, once FOI requests have been made for information, public authorities are not permitted to “alter, deface, block, erase, destroy or conceal any record held by the public authority, with the intention of preventing the disclosure by that authority of all, or any part, of the information to the communication of which the applicant would have been entitled.” As fordprefect observed the other day, the Computer Misuse Act also prescribes various offences, one of which is the unauthorized modification of the contents of a computer with the intent “to prevent or hinder access to any program or data held in any computer”.

I was therefore more than a little surprised that on July 27, at about 2:42 p.m. (UK), CRU deleted three files from their data directory ftp://ftp.cru.uea.ac.uk/data/ entitled newcrustnsall.dat.Z, newcruextusall.dat.Z and newcrustnsall.hdr. I have before and after screenshots of the contents of this directory showing the deletion, screenshots that I will show below. I also have a screenshot showing the most recent change of the directory and that no other files were changed. [As readers observed, it is possible and perhaps even likely that the files were moved to a concealed location, rather than totally erased, which would present similar sorts of issues for FOI compliance.]

As noted by CA reader Jean S, on July 28 at about 2:01 p.m. (UK), one or more files was deleted from the directory ftp://ftp.cru.uea.ac.uk/people/philjones/. In this case, we do not have a before screenshot, only evidence of the deletion of one or more files from the directory. [Again, it is possible that the file was moved to a concealed location, rather than totally erased.]

First, here is a screenshot of ftp://ftp.cru.uea.ac.uk/data/ taken on Saturday, July 25. You;ll notice the three files named above, two of which are dated 2/24/2003.

Second, here is a screenshot taken late yesterday showing that the three files named above have been deleted.

Third, here is a screenshot of the host directory showing that ftp://ftp.cru.uea.ac.uk/data was changed on July 27 at 2:42 pm:

Fourth, here is another printscreen of the contents of ftp://ftp.cru.uea.ac.uk/data in time order, showing that nothing was added on July 27 2:42 pm. The July 27, 2009 change was the deletion of the three files named above from the public directory.

As Jean S observed, the FTP directory also shows a change in Phil Jones’ directory on July 28, 2009, a screenshot showing the time stamp is below:

Here is a screenshot of the Phil Jones’ directory showing that nothing was added to it on July 28, leaving only one conclusion: that something was deleted from the public directory. In this case, I do not have a before- screenshot.

One obvious course of action right now will be to make a FOI request to CRU, seeking information about the contents of the various files that were deleted from public directories and whether they were in any way relevant to any of the outstanding inquiries, and, if they were, whether the deletion of these files from the public directories required authorization by the university’s FOI officer and whether such authorization was obtained.

By Stephen McIntyre | Posted in General | Tagged | Comments (186)

Met Office/CRU Finds the Mole

Jul 28, 2009 – 9:29 AM

More news on the Met Office/CRU molehunt.

Late yesterday (Eastern time), I learned that the Met Office/CRU had identified the mole. They are now aware that there has in fact been a breach of security. They have confirmed that I am in fact in possession of CRU temperature data, data so sensitive that, according to the UK Met Office, my being in possession of this data would, “damage the trust that scientists have in those scientists who happen to be employed in the public sector”, interfere with the “effective conduct of international relations”, “hamper the ability to protect and promote United Kingdom interests through international relations” and “seriously affect the relationship between the United Kingdom and other Countries and Institutions.”

Although they have confirmed the breach of security, neither the Met Office nor CRU have issued a statement warning the public of the newCRU_tar leak. Nor, it seems, have they notified the various parties to the alleged confidentiality agreements that there has been a breach in those confidentiality agreements, so that the opposite parties can take appropriate counter-measures to cope with the breach of security by UK institutions. Thus far, the only actions by either the Met Office or CRU appear to have been a concerted and prompt effort to cover up the breach of security by attempting to eradicate all traces of the mole’s activities. My guess is that they will not make the slightest effort to discipline the mole.

Nor have either the Met Office or CRU have contacted me asking me not to further disseminate the sensitive data or to destroy the data that I have in my possession.

By not doing so, they are surely opening themselves up to further charges of negligence for the following reasons. Their stated position is that, as a “non-academic”, my possession of the data would be wrongful (a position with which I do not agree, by the way). Now that they are aware that I am in possession of the data (and they are aware, don’t kid yourselves), any prudent lawyer would advise them to immediately to notify me that I am not entitled to be in possession of the data and to ask/instruct me to destroy the data that I have in my possession and not to further disseminate the sensitive data. You send out that sort of letter even if you think that the letter is going to fall on deaf ears.

Since I am always eager to help climate scientists with these conundrums, I’ll help them out a little here. If, prior to midnight Eastern time on Thursday July 30, 2009, a senior executive of the Met Office or the University of East Anglia notifies me that I am in wrongful possession of the data and directly requests me to destroy my copies of the CRU station data in question and thereby do my part in the avoidance of newCRU_tar proliferation, I will do so.

I will, of course, continue my FOI requests since I do not believe, for a minute, that their excuses have any validity nor am I convinced that the alleged confidentiality agreements actually exist nor, if they exist, am I convinced that they prohibit the provision of the data to me.

By Stephen McIntyre | Posted in CRU, FOIA | Tagged | Comments (204)

A Mole

Jul 25, 2009 – 9:35 PM

OK, folks, guess what. I’m now in possession of a CRU version giving data for every station in their station list .

In their refusal letter, the Met Office described adverse consequences of disclosing CRU station data, an event that apparently would let loose the Four Horsemen of the Apocalypse.

The Met Office stated:

Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released. …

If any of this information were released, scientists could be reluctant to share information and participate in scientific projects with the public sector organisations based in the UK in future. It would also damage the trust that scientists have in those scientists who happen to be employed in the public sector and could show the Met Office ignored the confidentiality in which the data information was provided.

They continued:

the effective conduct of international relations depends upon maintaining trust and confidence between states and international organisations. This relationship of trust allows for the free and frank exchange of information on the understanding that it will be treated in confidence. If the United Kingdom does not respect such confidences, its ability to protect and promote United Kingdom interests through international relations may be hampered.

Competitors/ Collaborators could be damaged by the release of information which was given to us in confidence and this will detrimentally affect the ability of the Met Office (UK) to co-operate with meteorological organisations and governments of other countries. This could also provoke a negative reaction from scientist globally if their information which they have requested remains private is disclosed.

And that’s not all. There’s more:

to release it without authority would seriously affect the relationship between the United Kingdom and other Countries and Institutions.

CRU was a less dramatic but still very clear about the consequences:

we feel that there is a strong public interest in upholding contract terms governing the use of received information. To not do so would be to potentially risk the loss of access to such data in future.

Just to prove that I have actual CRU station data, here is the 60th series (Lund Sweden), covering the period 1753-1773: sensitive information indeed.

1753 -1.8 -1.3 3.7 7.6 11.5 14.6 16.7 15.9 13.4 9.9 3.2 -3.0
1754 -1.0 -1.4 -1.2 5.6 12.9 15.2 15.1 15.5 11.9 10.1 4.7 1.9
1755 -3.8 -5.3 0.8 7.9 12.0 17.8 18.2 15.4 12.1 8.4 3.7 2.0
1756 1.9 2.3 2.5 4.2 9.9 17.6 19.4 15.6 14.1 9.2 1.8 -0.1
1757 -2.8 0.7 1.4 8.2 10.7 18.2 21.4 17.6 13.6 5.2 6.0 1.3
1758 -3.9 -2.0 0.7 3.3 13.9 16.7 16.0 16.8 11.8 6.7 4.5 1.0
1759 2.4 2.3 3.4 6.2 10.2 17.4 20.1 18.1 13.1 9.1 2.1 -2.0
1760 -4.0 -1.0 0.7 6.1 11.8 19.2 18.2 17.1 15.3 8.5 4.0 2.5
1761 0.6 1.2 5.0 6.8 12.9 18.0 17.3 18.3 15.2 6.3 5.1 -0.6
1762 1.1 -0.7 -1.8 8.1 11.5 17.0 17.4 14.2 12.4 4.8 4.1 0.5
1763 -3.9 0.5 0.5 4.6 11.2 14.9 17.8 16.9 11.5 7.7 2.8 3.0
1764 -0.1 3.0 1.4 5.6 12.5 13.6 20.5 16.3 11.8 7.6 2.4 0.1
1765 -0.3 -2.3 2.9 6.9 10.2 15.3 15.9 16.9 11.9 9.0 4.5 -0.2
1766 -1.8 -2.7 2.1 8.1 11.9 17.3 18.8 17.2 13.8 8.7 5.8 -0.9
1767 -6.1 -0.6 2.1 2.7 9.8 13.9 16.4 17.3 15.0 8.9 6.4 0.4
1768 -5.5 -3.1 -2.4 5.1 10.7 16.3 17.9 17.1 12.5 8.2 4.9 2.1
1769 0.6 -0.5 2.3 5.7 11.3 15.6 17.6 15.9 13.6 5.2 2.6 3.2
1770 -2.3 0.0 -2.9 4.5 11.5 15.1 18.1 18.1 15.4 10.5 2.5 1.5
1771 -3.8 -3.8 -3.9 2.2 12.4 18.0 17.2 15.1 12.5 10.1 2.8 2.5
1772 -1.6 -2.1 -1.1 4.7 10.0 16.2 17.8 17.1 13.6 11.0 7.2 2.9
1773 1.0 -0.9 1.6 7.3 14.1 15.9 18.1 18.0 14.5 11.2 5.0 2.6

It’s hard to imagine that my being in possession of CRU station data would “damage the trust that scientists have in those scientists who happen to be employed in the public sector”, interfere with the “effective conduct of international relations”, “hamper the ability to protect and promote United Kingdom interests through international relations” and “seriously affect the relationship between the United Kingdom and other Countries and Institutions.”

But that’s what the Met Office says.

Given such dramatic adverse consequences, I wonder what they’ll do. Will they investigate? I wonder what the form of investigation will be. Will they do it Jack Nicholson-style?

I’ll keep my eye out for secret agents from MI-5. The data’s in a safe place, but I doubt that I would bear up well under waterboarding.

And by the way, just because I’ve got a version of the data doesn’t mean that I’m going to give up trying to get the data through FOI. Quite the opposite.

By Stephen McIntyre | Posted in FOIA, Surface Record | Tagged | Comments (282)

CRU – Then and Now

Jul 25, 2009 – 12:16 PM

In respect to the alleged CRU confidentiality agreements (which look increasingly fictitious), Jean S made the interesting observation that CRU archived station in the 1980s and early 1990s at CDIAC (ndp020) and that the alleged CRU confidentiality agreements, for some reason, did not interfere with that data being archived.

Interestingly, I happen to have a station list and station data of 1994 vintage – a little later than NDP020 – and compared the present station list to the 1994 list, showing both deletions and additions. The number of stations increased from 3524 in the 1994 vintage to 4138 in the current vintage, which gives the impression of more activity and expanded coverage. But see the remarkable maps below. Most of the additions are in the US and a couple of other countries, while deletions are widespread.

I’ve spent a fair bit of time comparing the lists. There’s a lot of sloppiness in the present CRU list – country identifications have not been updated for example. The USSR lives on at CRU. Not infrequently the same station is picked up from two different sources, each duplicate having a separate CRU identification.

But I had another interest in the comparison. In refusing a prior FOI request, Jones said that the data was already publicly available at GHCN. This is not true for the entire data set, but many CRU stations can be conclusively identified with GHCN stations. If Jones is deriving his CRU version for (say) Bahrain from GHCN, he cannot also rely on a confidentiality agreement with Bahrain as an excuse for not providing data.

It should be possible to go through the countries with data additions since 1994 one-by-one, mark off the ones whose data additions come from GHCN and see what’s left. This sort of thing takes a bit of time, but the results are always interesting. Here’s a preliminary report on the provenance of the additions – the ones to which the alleged confidentiality might apply.

In the list below, there are 59 countries in which a station has been added since the 1994 “public” version.

The vast majority of the increase came from the US through the addition of much of the USHCN network (frequently duplicating prior sources). Some US stations could not be tied so far to USHCN or GHCN versions.

New non-GHCN versions were added for Canada and Australia, frequently as duplicates.

A number of European countries added non-GHCN versions: Austria, Denmark, Sweden, Norway, Finland, Netherlands, one UK station (Wick), one Ireland station (Phoenix Park), one Italy station (Milano/Brera).

In the rest of the world, the additions were nearly all GHCN stations. Exceptions were Syria and Taiwan. Perhaps China, though I suspect that more identifications with GHCN versions are possible than I’ve accomplished so far. Given that the additional data for Spain comes from GHCN stations, the question of a confidentiality agreement with Spain becomes moot. Similarly for say South Africa or PNG. There are a few odds and ends that I couldn’t match right away – a few stations in Iran, one in Brazil, a couple in Bolivia. Argentina has a number of additions (nearly all airports).

It’s hard for me to imagine that CRU has entered into confidentiality agreements during the IPCC period with the countries adding non-GHCN versions to the version of the data to which no restrictions applied. Countries like the US, Canada, Australia, Denmark, Netherlands, Sweden, Finland, Norway – a list that includes some self-righteous countries.

SUMMARY:
country count Comment
ALGERIA-2: Both GHCN stations
ANTARCTICA-1. GHCN stations (Bernardo O’Higgins renumbered)
ARGENTINA 21. New stations are mostly airports
AUSTRALIA 147 Mostly GHCN. Need to crosscheck others.
AUSTRIA 11 Mostly new.
BAHRAIN 1 GHCN station.
BANGLADESH 2 GHCN stations
BOLIVIA 2 Two new: Potosi, Central La Paz
BRAZIL 2 One GHCN (Brasilia airport); one new- Caceres
BURKINA FASO 1 GHCN station.
CANADA 95 Many new; many duplicates
CHILE 4 Two GHCN; two “Unknown” 858341, 858921
CHINA 34 Many GHCN; many unmatched, but perhaps can be collated.
COTE D’IVOIRE 1 GHCN station
CROATIA 4 Renumbered GHCN
DENMARK 3 Three new.
DOMINICAN REPU 1 GHCN station
ECUADOR 1 GHCN station
EGYPT 1 GHCN station
FINLAND 3 Three new
FRANCE 1 GHCN station
GERMANY 13 GHCN stations
GREECE 3 Two GHCN; check ATHENS/HELLENIKON
GREENLAND 2 GHCN – check renumbering of Kap Tobin
HONDURAS 1 GHCN station
HUNGARY 2 GHCN stations
ICELAND 1 GHCN station renunbered
INDIA 2 GHCN stations
IRAN 35: 32 GHCN, Oroomieh, Bandar Azali, Chahbahar
IRELAND 3 Two GHCN- check Phoenix Park
ISRAEL 5 All new
ITALY 1 Check Milano/Brera
JAPAN 4 GHCN stations
JORDAN 1 GHCN stations
KOREA 9 GHCN stations
KUWAIT 1 GHCN stations
MADEIRA 1 GHCN stations
MALAWI 1 GHCN stations
MALAYSIA 1 GHCN stations
MEXICO 14 GHCN stations
MONGOLIA 2 GHCN stations
NETHERLANDS 4. One GHCN – check De Kooy, Eelde, Vlissingen
NEW ZEALAND 7 GHCN stations
NORWAY 12 Six GHCN, some new
PAPUA NEW GUIN 5 GHCN stations
PHILIPPINES 1 GHCN stations
PORTUGAL 3 GHCN
PUERTO RICO 1 GHCN stations
SLOVENIA 1 Renumbered GHCN
SOUTH AFRICA 34 GHCN
SPAIN 4 GHCN stations
SWEDEN 20 Mostly new
SYRIA 6 Mostly new
TAIWAN 14 Mostly new
TURKEY 1 GHCN station
UK 2 One GHCN – check Wick
URUGUAY 1 GHCN stations
USA 1051 additions – mostly USHCN
RUSSIA (“USSR”) 1 GHCN station

By Stephen McIntyre | Posted in CRU, Surface Record | Tagged , | Comments (28)

CRU Refuses Data Once Again

Jul 24, 2009 – 8:38 AM

Let me review the request situation for readers. There are two institutions involved in the present round of FOI/EIR requests: CRU and the Met Office. Phil Jones of CRU collects station data and sends his “value added” version to the Met Office, who publish the HadCRU combined land-and-ocean index and also distribute the CRUTEM series online.

I requested a copy of the “value added” version from the Met Office (marion.archer at metoffice.uk.gov) which has been refused for excuses provided in my last post. On June 25, 2009, learning that Phil Jones had sent a copy of the station data to Peter Webster of Georgia Tech, I sent a new FOI request to CRU ( david.palmer at uea.ac.uk) requesting the data in the form sent to Peter Webster. This too was refused today.

We now have a new excuse to add to our collection of excuses – each excuse seemingly more ridiculous than the previous one.

My most recent request was as follows:

Dear Mr Palmer,

Pursuant to the Environmental Information Regulations, I hereby request a copy of any digital version of the CRUTEM station data set that has been sent from CRU to Peter Webster and/or any other person at Georgia Tech between January 1, 2007 and Jun 25, 2009.

Thank you for your attention,

Stephen McIntyre

The full response was as follows. (I’ve included full address particulars for readers that may wish to follow up):

Dear Mr McIntyre

ENVIRONMENTAL INFORMATION REGULATIONS 2004 – INFORMATION REQUEST (FOI_09-44; EIR_09-03)

Your request for information received on 26 June 2009 has now been considered and it is, unfortunately, not possible to meet all of your request.

In accordance with Regulation 14 of the Environmental Information Regulations 2004 this letter acts as a partial Refusal Notice, and I am not obliged to supply this information and the reasons for exemption are as stated below:

 Exception  Reason
 Reg. 12(5)(f) – Adverse effect on the person providing information  Information is covered by a confidentiality agreement

Regulation 12(5)(f) applies because the information requested was received by the University on terms that prevent further transmission to non-academics

Regulation 12(1)(b) mandates that we consider the public interest in any decision to release or refuse information under Regulation 12(4). In this case, we feel that there is a strong public interest in upholding contract terms governing the use of received information. To not do so would be to potentially risk the loss of access to such data in future.

I apologise that not all of your request will be met but if you have any further information needs in the future then please contact me.

If you have any queries or concerns, or, if you are dissatisfied with the handling of your request please contact me at:
University of East Anglia
Norwich
NR4 7TJ
Telephone: 0160 393 523
E-mail: foi AT uea.ac.uk

You also have the right of appeal against the decision. If you wish to appeal please set out in writing your grounds of appeal and send to me at the same address as noted above.

Subsequent to our determination of your appeal, you also have a further right of appeal to the Information Commissioner at:
Information Commissioner’s Office
Wycliffe House
Water Lane
Wilmslow
Cheshire
SK9 5AF
Telephone: 01625 545 700
http://www.ico.gov.uk

Yours sincerely
David Palmer
Information Policy and Compliance Manager
University of East Anglia

This is the first time that we’ve heard that their supposed confidentiality agreements merely restrict “further transmission to non-academics”. A couple of observations on this. I’m sure that CRU will soon receive a similar request from someone to whom this excuse does not apply.

However, aside from that, there are other troubling aspects to this refusal. If there actually are confidentiality agreements, I would expect the relevant language to be framed in terms of “academic use” as opposed to guild membership i.e. I’d be surprised if the language were framed in terms of institutional affiliation as opposed to use. I’ve published relevant articles in peer reviewed literature, acted as an IPCC reviewer, been cited in IPCC AR4, been invited to present to a NAS panel – my use of data is “academic” by any legal standard.

Secondly, over at the Met Office, they say “it cannot be determined which countries or stations data were given in confidence as records were not kept.” But over at CRU, they purport to “know” nuanced details of the contractual language of the confidentiality agreements – clauses that have the effect of justifying the refusal of the data.

By Stephen McIntyre | Posted in CRU, FOIA | Tagged , , , | Comments (188)

UK Met Office Refuses to Disclose Station Data Once Again

Jul 23, 2009 – 8:49 AM

It must be humiliating for the UK Met Office to have to protect Phil Jones and CRU. Even a seasoned bureaucrat must have winced in order to write the following:

Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released and it cannot be determined which countries or stations data were given in confidence as records were not kept.

Here is the complete text of the UK Met Office’s most recent refusal of their station data.

Our Ref: 22-06-2009-131902-003 23 July 2009
Dear Mr McIntyre

Request for Information – Information not Held and Refusal to Disclose Information
Your correspondence dated 9 June 2009 has been considered to be a request for information in accordance with the Environmental Information Regulations 2004. The Ministry of Defence is permitted to withhold information where exceptions are considered justifiable.

You asked “You stated that CRUTEM3 data that you held was the value added data. Pursuant to the Environmental Information Regulations Act 2004, please provide me with this data in the digital form, together with any documents that you hold describing the procedures under which the data has been quality controlled and where deemed appropriate, adjusted to account for apparent non-climatic influences”.

Your request has been assessed and this letter is to inform you that the Met Office does hold some information covered by the request. We do not hold documents describing the procedures under which the data has been quality controlled or adjusted to account for apparent non-climatic influences.

The information held by the Met Office is withheld in accordance with the following exceptions pursuant to the Environmental Information Regulations Act 2004:
• Section 12 (5) (a) Information likely to prejudice relations between the United Kingdom and any International organisation;
• Section 12 (5) (e) Confidentiality of commercial or industrial information where such confidentiality is provided by law to protect a legitimate economic interest.
• Section 12 (5) (f) (i) (iii) The supplier was not under legal obligation to supply the information and has not consented to its disclosure.

As the above exceptions are qualified exceptions, a public interest test was undertaken by the Met Office to consider whether there are overriding reasons why disclosure of this information would not be in the public interest. The Met Office has duly considered these reasons in conjunction with the public interest in disclosing the requested information, in particular the benefits of assisting the public having information on environmental information, whereby they would hope to influence decisions from a position of knowledge rather than speculation.
Access to environmental information is particularly important as environmental issues affect
the whole population.

Consideration of Exception Regulation 12 (5) (a)
Much of the requested data comes from individual Scientists and Institutions from several countries. The Met Office received the data information from Professor Jones at the University of East Anglia on the strict understanding by the data providers that this station data must not be publicly released. If any of this information were released, scientists could be reluctant to share information and participate in scientific projects with the public sector organisations based in the UK in future. It would also damage the trust that scientists have in those scientists who happen to be employed in the public sector and could show the Met Office ignored the confidentiality in which the data information was provided.

We considered that if the public have information on environmental matters, they could hope to influence decisions from a position of knowledge rather than speculation. However, the effective conduct of international relations depends upon maintaining trust and confidence between states and international organisations. This relationship of trust allows for the free and frank exchange of information on the understanding that it will be treated in confidence. If the United Kingdom does not respect such confidences, its ability to protect and promote United Kingdom interests through international relations may be hampered. Competitors/ Collaborators could be damaged by the release of information which was given to us in confidence and this will detrimentally affect the ability of the Met Office (UK) to co-operate with meteorological organisations and governments of other countries. This could also provoke a negative reaction from scientist globally if their information which they have requested remains private is disclosed.

Consideration of Exception Regulation 12 (5) (e)
The information is also withheld in accordance with the exception under regulation 12 (5) (e) because the information comprises of Station Data which are commercially sensitive for many of the data sources (particularly European and African Meteorological services) release of any data could adversely affect relationships with other Institutions and individuals, who may plan to use their data for their own commercial interests. Some of this is documented in Hulme, 1996 but this is not a globally comprehensive summary.

The Met Office are not party to information which would allow us to determine which countries and stations data can or cannot be released as records were not kept, or given to the Met Office, therefore we cannot release data where we have no authority to do so. Competitors or collaborators could be damaged by the release of information which was given to us in confidence and could affect their ability to trade.

The Met Office uses the data solely and expressly to create a gridded product that we distribute without condition.

Consideration of Exception Regulation 12 (5) (f) (i) and (iii)
The information is also withheld in accordance with the exception under regulation 12 (5) (f) (i) (iii) as Professor Jones was not legally bound to release the data to the Met Office and has not consented to the disclosure to any other party. As stated above in 12 (5) (a) Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released and it cannot be determined which countries or stations data were given in confidence as records were not kept. The Met Office received the data from Professor Jones on the proviso that it would not be released to any other source and to release it without authority would seriously affect the relationship between the United Kingdom and other Countries and Institutions.

I hope this answers your enquiry.

If you are not satisfied with this response or you wish to complain about any aspect of the handling of your request, then you should contact me in the first instance. If informal resolution is not possible and you are still dissatisfied then you may apply for an independent internal review by contacting the Head of Corporate Information, 6th Floor, MOD Main Building, Whitehall, SW1A 2HB (e-mail CIO-XD@mod.uk). Please note that any request for an internal review must be made within 40 working days of the date on which the attempt to reach informal resolution has come to an end.

If you remain dissatisfied following an internal review, you may take your complaint to the Information Commissioner under the provisions of Section 50 of the Freedom of Information Act. Please note that the Information Commissioner will not investigate your case until the MOD internal review process has been completed. Further details of the role and powers of the Information Commissioner can be found on the Commissioner’s website, http://www.ico.gov.uk.

Yours sincerely,
Marion Archer
FOI Manager

Update Aug 4. Following reply sent:

Dear Ms Archer.

This is to acknowledge receipt of your letter of 23 July 2009. I request that you reconsider your decision for the following reasons.

You say that there is a strict understanding between CRU and data providers that station data not be publicly released. CRU’s actions show otherwise. CRU and Dr Jones have routinely posted station data online since 1990, with 1990, 1996 and 2003 versions being online at the time of my request to you. The 1990 version is online at CDIAC in the U.S. ( http://cdiac.ornl.gov/ftp/ndp020 ); from 1996 to until July 31, 2009, the 1996 version was online at CRU ( formerly accessible through http://www.cru.uea.ac.uk/advance10k/climdata.htm); from early 2003 to July 27, 2009, the 2003 version was online at CRU (formerly at ftp://ftp.cru.uea.ac.uk/data/). These latter two files were removed from public directories only after FOI/EIR requests for station data were submitted to the Met Office and CRU. Obviously any of the “lost” confidentiality agreements did not present a problem for placing data online up to 2003. Would you please consider your refusal in light of this additional information regarding online availability of earlier versions of these data sets between 1990 and July 31, 2009.

In addition, you stated that the Met Office had entered into an agreement with Dr Jones and/or CRU which included a proviso that the data “not be released to any other source”. Under the EIR, would you please provide me with the date of this agreement, the parties to the agreement, a copy of the clause containing the precise language of this proviso and a copy of the entire agreement or any other document evidencing that the Met Office had entered into such an agreement.

Yours truly,
Stephen McIntyre

By Stephen McIntyre | Posted in CRU, FOIA | Tagged , , , | Comments (126)

HadCRU Sept 1850

Jul 22, 2009 – 8:18 AM

For some reason, the main HadCRU global data set http://hadobs.metoffice.com/crutem3/diagnostics/global/nh+sh/monthly is missing September 1850. I noticed this when the plot came up one month short. I wonder why this month went AWOL.

By Stephen McIntyre | Posted in CRU | Comments (21)

More Tropical Troposphere: UAH versus NOAA

Jul 21, 2009 – 9:45 PM

I recently showed a couple of breakpoint comparisons for satellite data: RSS versus NOAA and RSS versus UAH. Today, I’ll show a similar comparison for UAH versus NOAA, again stratifying by Land, Ocean and All. (Unfortunately, I was unable to extract a satellite comparison for other major food group: the CRU_TAR (airport tarmac).

Again, the breakpoints are calculated using strucchange and line segments are fitted between breakpoints.


Figure 1. UAH minus NOAA, by firma.

At this point, I’m simply plotting results using a relevant algorithm as an exploratory analysis and am not asserting that any of the breakpoints are “Significant”. Having said that, the breakpoints (which have CIs attached to them) plausibly relate to the (first) termination of NOAA-6 (1983-11), NOAA-10 (1988-10), NOAA-12 (1998-12) and the introduction of AMSU units (2004-12). Here is one more interesting graphic from Christy et al 2000 showing the differences between instrument temperature for different satellites up to 2000 – something that has to be adjusted for. Could errors of 0.2 deg C arise in this standardization? Seems possible to me. Could three errors all be of the same sign? Sure, they could.

The breakpoint structure point to an interesting statistical issue in respect to trend estimation, which may have important implications for the ongoing issue of satellites versus observation. Take a look at the bottom panel (Land). The intra-segment trends are all flat or slightly positive, while the trend without breakpoints is negative – this negative trend is the T2LT warming less than surface (NOAA) warming inconsistent with model expectations.

However, suppose that the breakpoints analysis has identified real breakpoints at important satellite transitions. Is it possible that three inter-satellite adjustments were each off by 0.2 deg C or so (and all in the same direction)? I don’t know enough about the inter-satellite adjustment procedures to comment on this, but my quick perusal of the literature hardly leads me to exclude that possibility.

Let’s also suppose that there were three breakpoints of the sort shown and that otherwise the fluctuations around the trend are white-to-low order red noise. If the breakpoints are not recognized, this will ARIMA-model out as quite high order red noise.

Also if there were breakpoints of the form shown, note that the “true” trend is the composite of the intrasegment trends, which is positive i.e. troposphere warming slightly more than the surface in line with models. So quite a lot turns on breakpoint analysis – an issue that I do not recall being addressed in the CCSP or IPCC reports on this topic. (FWIW, I’ve reflected a bit on this matter, about which much has been made in “skeptical” literature and will try to re-visit it some time. After some reflection, the idea that the tropical troposphere would warm more than the surface doesn’t seem to me to be a particularly unreasonable model property. On the other hand, surface histories of tropical tarmac may have flaws of their own. As also histories of SST with complicated transitions from buckets to engine inlets to buoys.)

By Stephen McIntyre | Posted in Satellite and gridcell | Comments (19)

Christy et al 2009: Surface Temperature Variations in East Africa

Jul 19, 2009 – 3:55 PM

Christy et al (J Clim 2009), Surface Temperature Variations in East Africa and Possible Causes, is a really excellent article that will interest many readers interested in surface temperature data sets. It’s interesting on a number of counts, not all of which I have time to summarize today.

It is a detailed study of station records from Kenya and Tanzania in East Africa an area which more or less covers nine 5-degree gridcells from 10S to 5N and from 30E to 45E. They collected and digitized original British East Africa and German colonial station data, as well as GHCN, GISS and their sources (but not CRU data), resulting in a substantial expansion of available data. Christy obviously has an excellent record of placing data online and I hope that this extends to the newly collected station data (which is not online at the moment.)

Although surface data is the backbone of temperature history, detailed analyses of station data are rare and detailed analyses of non-US non-European data are even rarer. The absence of such analyses is an indictment of the authors of the major temperature indices (CRU, GISS, NOAA). They are funded to publish temperature indices and this sort of technical study should be part and parcel of their obligations.

Christy et al approach the calculation of gridcell temperatures a little differently than GISS (CRU still not providing an operational description of methodology). First, they try to identify different versions of the same station and to obtain a station history from these versions. This is along the lines of GISS’ calculation: GISS collecting various versions in dset0 and combining them in dset1. The approach of Christy et al looks more sensible than the GISS approach, though I doubt that the difference “matters” a lot to the final answer, other than being more logical.

Next Christy et al apply a breakpoints algorithm to distinguish unreported step changes in the station history citing the radiosonde method of Haimberger et al 2007. Breakpoint detection, as I read the article, is done through internal properties of the series, rather than through neighbor comparison (a la USHCN), neighbors being a lot sparser than in the US. They set the sensitivity of the changepoint parameter at three different settings and report on its impact on the trend (it’s non-trivial.) Breakpoint detection is a complicated statistical procedure and not one that I’ve studied enough to have an independent opinion on the merits of the various approaches (both the Christy et al code and USHCN code are unarchived in any event). However, Christy et al describe their test statistic (p 3345) and their methodology seems clearly preferable to the weird GISS two-legged coercion (CRU methodology needless to say is unknown).

Then they average (“merge”) the station anomaly series. I can’t tell in a first reading when they converted the data into anomalies – their Figure 1 shows anomaly series.

Then they compare their trends with trends calculated from HadCRUT3v, CRUTEM3v and GISS, reporting that they were unable to replicate the high trend from the major indices. It seems that the CRU series is dominated by the Nairobi airport:

The recent trends of TMean calculated from global datasets do not agree with our results for this cell. As shown in Table 2, the 1979–2004 TMean trend of the central cell as produced by HadCRUT3v, CRUTEM3v, and GISS (0.31, 0.47, and 0.35 deg C/decade, respectively) are markedly inconsistent with all of the time series for that cell constructed in this study. Evidently, the main signal used by HadCRUT3v for this cell since 1979 is derived from the single Nairobi, Kenya, station at Jomo Kenyatta Airport (P. Jones 2004, personal communication). Our unadjusted time series for this site does indeed show significant warming since 1979 (0.25 deg C/decade), but the higher trend is not corroborated by the many nearby stations used in our analysis. Such differences were also found in central California (Christy et al. 2006) and northern Alabama (Christy 2002), where our more comprehensive reconstructions were on average about 0.1 deg C/decade more negative in the cells covering those areas versus values for the cell from global databases.

Christy et al continue with an interesting discussion of Tmin versus Tmax, arguing that Tmax samples a bigger volume of air than Tmin and is a more reliable index of large-scale changes (citing Pielke et al 2007).

The idea that the Jones CRUTEM series is dominated by Nairobi airport will come as little surprise to CA readers – recently we saw that the CRUTEM Hawaii series is little more than an alter ego for Honolulu airport.

I think that it is time to recognize that calling CRUTEM an index of “Land” temperature is a misnomer as, in addition to “Ocean” and “Land”, there is a third important surface in major temperature indices: airport tarmac. In appreciation for Phil Jones’ efforts at measuring airport temperatures around the world, perhaps it would be appropriate that his series be rebranded CRUTAR.

By Stephen McIntyre | Posted in CRU, Surface Record | Tagged | Comments (38)

RSS versus UAH: Battles over Tropical Land and Ocean

Jul 18, 2009 – 1:48 PM

Until I recently examined the underlying technical literature on the construction of the UAH and RSS satellite data sets, I had little appreciation of the complicated adjustment and estimation procedure involved in the satellite temperature indices.

The UAH and RSS satellite temperature indices are constructed from many different satellites (TIROS-N, NOAA-6,7,8,9,10,11,12,14,…), each of which has distinctive instrumental and orbital properties, which need to be allowed for in the estimation of the “true” tropospheric temperature. The estimation of these parameters is done differently by Christy and his associates on the one hand and Mears and his associates on the other. Controversies over the merits of these adjustment procedures have been ongoing for over a decade.

From the perspective of CA readers, there is a substantial statistical component to the estimation of the various adjustment and bias correction parameters, as can be seen from examining Christy et al 2000 url and Mears et al 2003 url , both of which describe statistical procedures, though not always expressing things in ordinary statistical terms.

In looking at tropical tropospheric results recently, I happened to do a crosscut of UAH versus RSS, which seemed to me to provide an interesting perspective on this debate. I am not nearly familiar enough with the overall issues to venture an opinion on who is “right” and who is “wrong” in any of these disputes. I am merely presenting a graph that intrigued me.

Before doing so, I want to present a few graphics that illustrate the history of a few relevant parameters for the satellites considered in the two articles references above (which do not include the most recent satellites), but which illustrate the scale of variation which can occur.

First, here is a graphic from Christy et al 2000 showing their adjustment by satellite for orbital decay. Aside from particulars of orbital decay, this graphic also provides information on satellites used for construction of the index and major transitions. For example, in 1986, there was a switchover from N-8 to N-10, with N-6 being briefly called out of retirement because of the short lifespan of N-8. In 1992, there was another transition from N-10 to N-12 with overlap being provided by N-11.


Figure 1. From Christy et al 2000. Impact of orbital decay.

Second, here is another graphic showing changes in local equatorial crossing time LECT), another effect adjusted for by both parties. This is presented here primarily to help readers get their eye into when the satellite switches took place.


Figure 2. From Christy et al 2000. Changes in local equatorial crossing time.

Next is a crosscut comparing UAH and RSS presented by Tamino last year in a surprisingly sober assessment of the differences between UAH and RSS.

Tamino observed:

There are two differences which are apparent to the eye. First, there’s a “step change” at 1992, with RSS being higher than UAH after that but lower before that. Second, in the most recent time period (from about 2003 on) there’s an annual cycle, with RSS being relatively higher during northern hemisphere summer and UAH relatively higher in northern hemisphere winter.

1992 and Other Steps

The existence of a 1992 step appears to be common ground among the parties. It was specifically mentioned by John Christy in a recent email to me following my recent post on satellites (in which he kindly also provided a variety of references on the interesting adjustment issues.)

The 1992 step has been attributed by the various parties to differences in handling the NOAA-10 to NOAA-12 satellite switch.

Recently, I experimented with strucchange on RSS versus UAH in the tropics, stratifying land and ocean series separately, yielding a pretty interesting crosscut relative to the above debate. I think that there are a variety of interesting points in this graphic, which I’ll discuss below.


Figure 3. Tropical T2LT UAH versus RSS, stratified by Ocean and Land, showing breakpoints from strucchange.

First, there is a remarkable lack of similarity between the Ocean and Land differentials. Over tropical oceans, RSS increases relative to UAH quite consistently, whereas the patterns over tropical land are quite erratic. Since 2004, UAH tropical land has a dramatic increase relative to RSS overlying the strong annual cycle in the difference series.

In the land series, strucchange picks out several breakpoints, each of which can be plausibly identified with a satellite changeover. The 1992 breakpoint at the transition from NOAA-10 to NOAA-12 is picked out. However, the 1992 breakpoint is not as unique as presumed in previous discussion nor even primus inter pares. There is a substantial breakpoint in 1986 at the transition to NOAA-10 (from a patchwork of satellites in the immediately prior period. There is a noticeable breakpoint in 1998 at the end of NOAA-12.

There is also a noticeable breakpoint around 2004-2005. Mears et al 2009 observe:

In order to continue the atmospheric temperature record past 2004, when measurements from the last MSU instrument degraded in quality, AMSU and MSU measurements must be intercalibrated and combined to extend the atmospheric temperature data records.

While the earlier history of the respective RSS and UAH adjustments has been much debated, it looks like another chapter needs to be written on the MSU/AMSU changeover, where, far from differences diminishing from prior analyses, the differences are increasing, with RSS/UAH difference trends being opposite over tropical land and over tropical ocean – a point not apparent in the combined difference series, where the trends offset.

Both Christy et al (J Clim 2009) and Mears et al ( Journal of Atmospheric and Oceanic Technology, 2009a,b) see url have recent technical discussions, but I didn’t notice any discussion of this point in a quick perusal.

By Stephen McIntyre | Posted in Satellite and gridcell | Comments (42)