Willis Eschenbach’s FOI Request

Willis Eschenbach’s account of his FOI request has been published on other blogs (e.g. here ) but I’m re-publishing it because Willis actually sent it to me first and the events all played out and were documented in real time at Climate Audit (see here for posts on FOI).  After pursuing matters until April 2007, Willis gave up.  The next part of the story started again in June 2009 when I decided to try again for the station data, this time initially requesting station data received by the Hadley Center (on the basis that they might not be able to assert exemptions claimed by Hadley Center. ) This, of course, led to an interesting sequence of events last summer, resulting in the present situation.

In the Climategate Letters, early on, Phil Jones expressed his worry that McKitrick and I might discover this legislation. In fact, it was Willis who discovered that the applicability of the legislation. Thus, his story below.  This is one perspective and an interesting one.

 

 

People seem to be missing the real issue in the CRU emails. Gavin over at realclimate keeps distracting people by saying the issue is the scientists being nasty to each other, and what Trenberth said, and the Nature “trick”, and the like. Those are side trails. To me, the main issue is the frontal attack on the heart of science, which is transparency.

Science works by one person making a claim, and backing it up with the data and methods that they used to make the claim. Other scientists attack the work by (among other things) trying to replicate the first scientist’s work. If they can’t replicate it, it doesn’t stand. So blocking the FOIA allowed Phil Jones to claim that his temperature record (HadCRUT3) was valid science.

This is not just trivial gamesmanship, this is central to the very idea of scientific inquiry. This is an attack on the heart of science, by keeping people who disagree with you from ever checking your work and seeing if your math is correct.

As far as I know, I am the person who made the original Freedom Of Information Act to CRU that started getting all this stirred up. I was trying to get access to the taxpayer funded raw data that they built the global temperature record out of. I was not representing anybody, or trying to prove a point. I am not funded by Mobil, I’m an amateur scientist with a lifelong interest in the weather. I’m not “directed” by anyone, I’m not a member of a right-wing conspiracy. I’m just a guy trying to move science forwards.

The recent release of the hacked emails from CRU has provided me with an amazing insight into the attempt by Steve McIntyre, myself, and others from CA and elsewhere to obtain the raw station data from Phil Jones at the CRU. We wanted the data that was used to make the global temperature record that is used to claim “unprecedented” global warming. I want to give a chronological account of the interactions. I will reference the email numbers so that people can see the entire emails if they wish. While we don’t know if all of these emails are valid, the researchers involved such as Gavin Schmidt and Michael Mann that clearly indicate that they think they are authentic.

The story actually starts with Warwick Hughes, a climate researcher who had previously been in cordial contact with Phil Jones, the lead researcher of the CRU. I find only one email in the archive (0969308954) where Phil emails Warwick, from 2000. This is in response to some inconsistencies that Warwick had found in Phil’s work:

Warwick Hughes to Phil Jones, September ‘04:

Dear Phillip and Chris Folland (with your IPCC hat on),
Some days ago Chris I emailed to Tom Karl and you replied re the grid cells in north Siberia with no stations, yet carrying red circle grid point anomalies in the TAR Fig 2.9 global maps. I even sent a gif file map showing the grid cells barren of stations greyed out. You said this was due to interpolation and referred me to Phillip and procedures described in a submitted paper. In the last couple of days I have put up a page detailing shortcomings in your TAR Fig 2.9 maps in the north Siberian region, everything is specified there with diagrams and numbered grid points.

[1] One issue is that two of the interpolated grid cells have larger anomalies than the parent cells !!!!?????
This must be explained.

[2] Another serious issue is that obvious non-homogenous warming in Olenek and Verhojansk is being interpolated through to adjoining grid cells with no stations, like cancer.

[3] The third serious issue is that the urbanization affected trend from the Irkutsk grid cell neare Lake Baikal, looks to be interpolated into its western neighbour.

I am sure there are many other cases of this, 2 and 3 happening.
Best regards,
Warwick Hughes (I have sent this to CKF)

Phil to Warwick, same email:

Warwick,
I did not think I would get a chance today to look at the web page. I see what boxes you are referring to. The interpolation procedure cannot produce larger anomalies than neighbours (larger values in a single month). If you have found any of these I will investigate. If you are talking about larger trends then that is a different matter. Trends say in Fig 2.9 for the 1976-99 period require 16 years to have data and at least 10 months in each year. It is conceivable that at there are 24 years in this period that missing values in some boxes influence trend calculation. I would expect this to be random across the globe.

Warwick,
Been away. Just checked my program and the interpolation shouldn’t produce larger anomalies than the neighbouring cells. So can you send me the cells, months and year of the two cells you’ve found ? If I have this I can check to see what has happened and answer (1). As for (2) and (3) we compared all stations with neighbours and these two stations did not have problems when the work was done (around 1985/6). I am not around much for the next 3 weeks but will be here most of this week and will try to answer (1) if I get more details. If you have the names of stations that you’ve compared Olenek and Verhojansk with I would appreciate that.

Cheers
Phil

OK, so far we have a couple of scientists discussing issues in a scientific work, no problem. But as he found more inconsistencies, in order to understand what was going on, in 2005 Warwick asked Phil for the dataset that was used to create the CRU temperature record. Phil Jones famously replied:

Subject: Re: WMO non respondo
… Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. …
Cheers Phil

Hmmm … not good. Science can only progress if there is a free exchange of scientific data. The scientific model works like this:

1. A scientist makes claims, and reveals the data and methods he used to come to his conclusions.
2. Other scientists who don’t agree attack the claim by (inter alia) seeing if they can replicate the result, using the first scientist’s data and methods.
3. If the claims cannot be replicated, the claim is adjudged to be false.

Obviously, if the data or the methods are kept secret, the claims cannot be verified. Attacking other scientist’s claims is what what scientists do. This adversarial system is the heart of science.

When I found out about this, I couldn’t believe it. I thought, a scientist can’t do that, can they? So I wrote to CRU on September 8, 2006, saying:

I would like to obtain a list of the meteorological stations used in the preparation of the HadCRUT3 global temperature average, and the raw data for those stations. I cannot find it anywhere on the web. The lead author for the temperature average is Dr. Phil Jones of the Climate Research Unit.

Many thanks, Willis Eschenbach

I got no response from Phil Jones or anyone at CRU. So I filed a Freedom of Information act request

Now at this point, let me diverge to what was happening at CRU during this time. The first reference to Freedom of Information in their emails is from 2005, before they had received a single request. Immediately, they start to plan how to evade requests should some come in:

Tom Wigley, Former Director CRU, to Phil Jones, 21/01/2005

Phil,

I got a brochure on the FOI Act from UEA. Does this mean that, if someone asks for a computer program we have to give it out?? Can you check this for me (and Sarah). [snip]
Thanks,
Tom.

Phil replies to Tom:

Tom,

On the FOI Act there is a little leaflet we have all been sent. It doesn’t really clarify what we might have to do re programs or data. Like all things in Britain we will only find out when the first person or organization asks. I wouldn’t tell anybody about the FOI Act in Britain. I don’t think UEA really knows what’s involved.

As you’re no longer an employee I would use this argument if anything comes along. I think it is supposed to mainly apply to issues of personal information – references for jobs etc.

[snip]
Cheers
Phil

So the coverup starts immediately, even before the first request. “I wouldn’t tell anyone about the FOI act in Britain”.

Tom to Phil

Phil,

Thanks for the quick reply. The leaflet appeared so general, but it was prepared by UEA so they may have simplified things. From their wording, computer code would be covered by the FOIA. My concern was if Sarah is/was still employed by UEA. I guess she could claim that she had only written one tenth of the code and release every tenth line.

Tom

You can see how they plan to observe the spirit of the FOI Act.

Phil to Tom

Tom,

As for FOIA Sarah isn’t technically employed by UEA and she will likely be paid by Manchester Metropolitan University. I wouldn’t worry about the code. If FOIA does ever get used by anyone, there is also IPR to consider as well. Data is covered by all the agreements we sign with people, so I will be hiding behind them. I’ll be passing any requests onto the person at UEA who has been given a post to deal with them.
Cheers
Phil

Phil Jones has just gotten the news, and immediately he starts to plan how he is going to hide from an FOI request.

The next email (1109021312) is later in 2005:

At 09:41 AM 2/2/2005, Phil Jones wrote to Michael Mann :

Mike,

Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites – you never know who is trawling them. The two MMs have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone. Does your similar act in the US force you to respond to enquiries within 20 days? – our does ! The UK works on precedents, so the first request will test it.

We also have a data protection act, which I will hide behind. Tom Wigley has sent me a worried email when he heard about it – thought people could ask him for his model code. He has retired officially from UEA so he can hide behind that. IPR should be relevant here, but I can see me getting into an argument with someone at UEA who’ll say we must adhere to it !

….

Phil

So now we have two more ways for Phil to hide from the FOI Act … along with a threat to delete the data rather than release it.

Mann replies to Jones:

Thanks Phil,

Yes, we’ve learned out lesson about FTP. We’re going to be very careful in the future what gets put there. Scott really screwed up big time when he established that directory so that Tim could access the data.

Yeah, there is a freedom of information act in the U.S., and the contrarians are going to try to use it for all its worth. But there are also intellectual property rights issues, so it isn’t clear how these sorts of things will play out ultimately in the U.S….
mike

Next, from February 05. Jones to Mann, cc to Hughes and Bradley (co-authors of the “hockeystick” study)

From: Phil Jones:

To: mann
Subject: Fwd: CCNet: PRESSURE GROWING ON CONTROVERSIAL RESEARCHER TO DISCLOSE SECRET DATA
Date: Mon Feb 21 16:28:32 2005
Cc: “raymond s. bradley”, “Malcolm Hughes”

Mike, Ray and Malcolm,

Leave it to you to delete as appropriate !
Cheers
Phil
PS I’m getting hassled by a couple of people to release the CRU station temperature data. Don’t any of you three tell anybody that the UK has a Freedom of Information Act !

The first rule of the Freedom of Information act … nobody talks about the Freedom of Information Act.

With that as a prologue, let me return to my FOI request. On February 10, 2007, I received my reply from Mr. Dave Palmer of CRU:

Dear Mr. Eschenbach

FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)

Your request for information received on 28 September now been considered and I can report that the information requested is available on non-UEA websites as detailed below.

The Global Historical Climatology Network (GHCN-Monthly) page within US National Climate Data Centre website provides one of the two US versions of the global dataset and includes raw station data. This site is at: http://www.ncdc. noaa.gov/ oa/climate/ ghcn-monthly/ index.php

This page is where you can get one of the two US versions of the global dataset, and it appears that the raw station data can be obtained from this site.

Datasets named ds564.0 and ds570.0 can be found at The Climate & Global Dynamics Division (CGD) page of the Earth and Sun Systems Laboratory (ESSL) at the National Center for Atmospheric Research (NCAR) site at: http://www.cgd. ucar.edu/ cas/tn404/

Between them, these two datasets have the data which the UEA Climate Research Unit (CRU) uses to derive the HadCRUT3 analysis. The latter, NCAR site holds the raw station data (including temperature, but other variables as well). The GHCN would give their set of station data (with adjustments for all the numerous problems).

They both have a lot more data than the CRU have (in simple station number counts), but the extra are almost entirely within the USA. We have sent all our data to GHCN, so they do, in fact, possess all our data.

In accordance with S. 17 of the Freedom of Information Act 2000 this letter acts as a Refusal Notice, and the reasons for exemption are as stated below

Exemption Reason
s. 21, Information accessible to applicant via other means Some information is publicly available on external websites

I was outraged. So the next day, I made a second request:

Dear Mr. Palmer:

Thank you for your reply (attached below). However, I fear that it is totally unresponsive. I had asked for a list of the sites actually used. While it may (or may not) be true that “it appears that the raw station data can be obtained from [GHCN]“, this is meaningless without an actual list of the sites that Dr. Jones and his team used.

The debate about changes in the climate is quite important. Dr. Jones’ work is one of the most frequently cited statistics in the field. Dr. Jones has refused to provide a list of the sites used for his work, and as such, it cannot be replicated. Replication is central to science. I find Dr. Jones attitude quite difficult to understand, and I find your refusal to provide the data requested quite baffling.

You are making the rather curious claim that because the data “appears” to be out on the web somewhere, there is no need for Dr. Jones to reveal which stations were actually used. The claim is even more baffling since you say that the original data used by CRU is available at the GHCN web site, and then follow that with the statement that some of the GHCN data originally came from CRU. Which is the case? Did CRU get the data from GHCN, or did GHCN get the data from CRU?

Rather than immediately appealing this ruling (with the consequent negative publicity that would inevitably accrue to CRU from such an action), I am again requesting that you provide:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

I find it somewhat disquieting that an FOI request is necessary to force a scientist to reveal the data used in his publicly funded research … is this truly the standard that the CRU is promulgating?

Thank you for your cooperation in this matter.

Willis Eschenbach

On April 12, 2007, I got my second reply:

In regards the “gridded network” stations, I have been informed that the Climate Research Unit’s (CRU) monthly mean surface temperature dataset has been constructed principally from data available on the two websites identified in my letter of 12 March 2007. Our estimate is that more than 98% of the CRU data are on these sites.

The remaining 2% of data that is not in the websites consists of data CRU has collected from National Met Services (NMSs) in many countries of the world. In gaining access to these NMS data, we have signed agreements with many NMSs not to pass on the raw station data, but the NMSs concerned are happy for us to use the data in our gridding, and these station data are included in our gridded products, which are available from the CRU web site. These NMS-supplied data may only form a very small percentage of the database, but we have to respect their wishes and therefore this information would be exempt from disclosure under FOIA pursuant to s.41. The World Meteorological Organization has a list of all NMSs.

That didn’t help one bit. Without knowing which data was used, it was meaningless. They’ve tried s.21, they’ve tried s.41, neither exemption applies. So the next day, I replied:

While it is good to know that the data is available at those two web sites, that information is useless without a list of stations used by Jones et al. to prepare the HadCRUT3 dataset. As I said in my request, I am asking for:

1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and

2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.

Without knowing the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service), it is not possible to access the information. Thus, Exemption 21 does not apply – I still cannot access the data.

I don’t understand why this is so hard. All I am asking for is a simple list of the sites and where each site’s data is located. Pointing at two huge piles of data and saying, in effect, “The data is in there somewhere” does not help at all.

To clarify what I am requesting, I am only asking for a list of the stations used in HadCRUT3, a list that would look like this:

WMO# Name Source
58457 HangZhou NCAR
58659 WenZhou NCAR
59316 ShanTou GHCN
57516 ChongQing NMS

etc. for all of the stations used to prepare the HadCRUT3 temperature data.

That is the information requested, and it is not available “on non-UEA websites”, or anywhere else that I have been able to find.

I appreciate all of your assistance in this matter, and I trust we can get it resolved satisfactorily.

Best regards,

I received another letter, saying that they could not identify the locations of the requested information. I wrote back again, saying:

Dear Mr. Palmer:

It appears we have gone full circle here, and ended up back where we started.

I had originally asked for the raw station data used to produce the HadCRUT3 dataset to be posted up on the UEA website, or made available in some other form.

You refused, saying that the information was available elsewhere on non-UEA websites, which is a valid reason for FOI refusals.

I can report that the information requested is not available on non-UEA websites as detailed below.

Your most recent letter (Further _information_ letter_final_ 070418_rev01. doc), however, says that you are unable to identify the locations of the requested information. Thus, the original reason for refusing to provide station data for HadCRUT3 was invalid.

Therefore, since the information requested is not available on non-UEA websites, I wish to re-instate my original request, that the information itself be made available on your website or in some other form. I understand that a small amount of this data (about 2%, according to your letter) is not available due to privacy requests from the countries involved. In that case, a listing of which stations this applies to will suffice.

The HadCRUT3 dataset is one of the fundamental datasets in the current climate discussion. As such, it is vitally important that it can be peer reviewed and examined to verify its accuracy. The only way this can be done is for the data to be made available to other researchers in the field.

Once again, thank you for your assistance in all of this. It is truly not a difficult request, and is fully in line with both standard scientific practice and your “CODE OF PRACTICE FOR RESPONDING TO REQUESTS FOR INFORMATION UNDER THE FREEDOM OF INFORMATION ACT 2000″. I am sure that we can bring this to a satisfactory resolution without involving appeals or unfavorable publicity.

My best regards to you,

w.

Here is the response from 27 April:

Dear Mr. Eschenbach FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)

Further to your email of 14 April 2007 in which you re-stated your request to see

“a list of stations used by Jones et al. to prepare the HadCRUT3 dataset” I am asking for: 1) A list of the actual sites used by Dr. Jones in the preparation of the HadCRUT3 dataset, and 2) A clear indication of where the data for each site is available. This is quite important, as there are significant differences between the versions of each site’s data at e.g. GHCN and NCAR.”

In your note you also requested “the name and WMO number of each site and the location of the source data (NCAR, GHCN, or National Met Service)”,

I have contacted Dr. Jones and can update you on our efforts to resolve this matter.

We cannot produce a simple list with this format and with the information you described in your note of 14 April. Firstly, we do not have a list consisting solely of the sites we currently use. Our list is larger, as it includes data not used due to incomplete reference periods, for example. Additionally, even if we were able to create such a list we would not be able to link the sites with sources of data. The station database has evolved over time and the Climate Research Unit was not able to keep multiple versions of it as stations were added, amended and deleted. This was a consequence of a lack of data storage in the 1980s and early 1990s compared to what we have at our disposal currently. It is also likely that quite a few stations consist of a mixture of sources.

I have also been informed that, as the GHCN and NCAR are merely databases, the ultimate source of all data is the respective NMS in the country where the station is located. Even GHCN and NCAR can’t say with precision where they got their data from as the data comes not only from each NMS, but also comes from scientists in each reporting country.

In short, we simply don’t have what you are requesting. The only true source would be the NMS for each reporting country. We can, however, send a list of all stations used, but without sources. This would include locations, names and lengths of record, although the latter are no guide as to the completeness of the series.

This is, in effect, our final attempt to resolve this matter informally. If this response is not to your satisfaction, I will initiate the second stage of our internal complaint process and
will advise you of progress and outcome as appropriate. For your information, the complaint process is within our Code of Practice and can be found at: http://www1. uea.ac.uk/ polopoly_ fs/1.2750! uea_manual_ draft_04b. pdf

Yours sincerely David Palmer Information Policy Officer University of East Anglia

I loved the story line in this one “we do not have a list consisting solely of the sites we currently use”. Say what? How do they produce updates that change the temperature all the way back to 1870? But I digress …

So I advised him that I was appealing. His letter was passed to a Ms. Kitty Inglis, who replied

May 21, 2007, Decision of Information Commissoners’ Office
FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)

Dear Mr Eschenbach

Following David Palmer’s letter of 27th April 2007 to you regarding your dissatisfaction with our response to your FOI request of 25th January 2007, I have undertaken a thorough review of the contents of our file and have spoken with both Mr. Palmer and Professor Jones.

As a result of this investigation, I am satisfied that we have done all we can to fulfil [sic] your request and to provide you with the information you require where it is possible for us to do so.

I confirm that we are able to make available on the Climatic Research Unit website a list of stations, including name, latitude, longitude, elevation and WMO number (where available).

We are unable to provide a simple list of sources for these stations as we do not hold this information. Nor do we hold the raw (i.e. unadjusted) station data, as you describe it, at UEA. As stated in prior letters to you, raw station data are available on the NCAR and GHCN websites and gridded data are available on the Climatic Research Unit website. If these data are insufficient for your requirements, you will need to contact the NMS for the country in which the station is located to obtain the information you require.

I hope you are able to accept this response. We have contacted the Information Commissioner’s Office in relation to this matter and their advice is that if you are still dissatisfied with this response, you can, at this time, exercise your right of appeal to the
Information Commissioner by contacting them at:
Information Commissioner’ s Office
Wycliffe House

At that point, I let it go. I had a small victory, we got a list of the stations. It took me a couple more letters to actually get them to post the list. But nothing else.

Meanwhile, behind the scenes at CRU, I now find out that they were circling the wagons … what follows are their internal discussions about a series of FOI requests from myself, Steve McIntyre, Doug Keenan and others to CRU for various data. Phil Jones to Tom Keenan and Wei-Chyung Wang, 6/19/2007:

Wei-Chyung and Tom,

1. Think I’ve managed to persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.
2. Had an email from David Jones of BMRC, Melbourne. [EMAIL NOT FOUND IN CRU EMAILS – Willis] He said they are ignoring anybody who has dealings with CA, as there are threads on it about Australian sites.
3. CA is in dispute with IPCC (Susan Solomon and Martin Manning) about the availability of the responses to reviewer’s at the various stages of the AR4 drafts. They are most interested here re Ch 6 on paleo.
Cheers
Phil

Well, that explains a few things … they’ve managed to “persuade UEA to ignore all further FOIA requests if the people have anything to do with Climate Audit.” I hadn’t noticed that exemption in the FOI documentation I’d seen. Call me crazy, but I don’t think that’s legal, and it definitely isn’t ethical.

I note that they are circling the wagons in Australia as well …

Phil Jones to Thomas Peterson of NOAA, 6/20/2007 AM (1182342470) :

Tom P.

Just for interest. Don’t pass on.

Might be a precedent for your paper to J. Climate when it comes out. There are a few interesting comments on the CA web site. One says it is up to me to prove the paper from 1990 was correct, not for Keenan to prove we’re wrong. Interesting logic.
Cheers
Phil

Wei-Chyung, Tom,
I won’t be replying to either of the emails below [FROM STEVE MCINTYRE AND DOUG KEENAN], nor to any
of the accusations on the Climate Audit website. I’ve sent them on to someone here at UEA to see if we
should be discussing anything with our legal staff. The second letter seems an attempt to be nice to me,
and somehow split up the original author team. I do now wish I’d never sent them the data after their FOIA
request!

Cheers
Phil

He obviously views sending data in response to an FOIA request as optional.

Thomas Peterson to Jones, same email:

Fascinating. Thanks for keeping me in the loop, Phil. I won’t pass it on but I will keep it in the back of my mind when/if Russ asks about appropriate responses to CA requests. Russ’ view is that you can never satisfy them so why bother to try?

Again, responding to an FOIA request is viewed as optional.

Phil Jones :

PS to Gavin – been following (sporadically) the CA stuff about the GISS data and release of the code etc by Jim. May take some of the pressure of you soon, by releasing a list of the stations we use – just a list, no code and no data. Have agreed to under the FOIA here in the UK.

Oh Happy days!

So … that’s why I only got the station list and not the data, just to take the pressure off. Thanks, Phil.

Jones to Bradley and Amman, 5/9/08 (1210341221):

Mike, Ray, Caspar,

A couple of things – don’t pass on either.

2. You can delete this attachment if you want. Keep this quiet also, but this is the person [DAVID HOLLAND – Willis] who is putting in FOI requests for all emails Keith and Tim have written and received re Ch 6 of AR4. We think we’ve found a way around this.

Finding ways around FOI requests seems to be a popular sport. This is in reference to Steve trying to get the review comments to Chapter 6 of the UN IPCC Fourth Assessment Report.

Next, here’s the brilliant way that they found around the FOIA, a bombshell of an idea, Jones to Michael Mann, 29 May 2008 (1212063122):

Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He’s not in at the moment – minor family crisis.

Can you also email Gene and get him to do the same? I don’t have his new email address.

We will be getting Caspar to do likewise.

Cheers
Phil

Again, call me crazy, but deleting evidence in the face of an FOI request must be illegal. Gene is Eugene Wahl. Of course, what these guys don’t realize is that there are multiple copies of most emails floating around. In some ways, I hope they deleted them, so that it can be proven.

Tim Osborne to Jones, Briffa, and Mann, 23 Jun 2008 (1214229243) :

Subject: Re: CA

Hi Phil, Keith and “Confidential Agent Ammann”,
At 17:00 21/06/2008, P.Jones wrote:

This is a confidential email

So is this.

Have a look at Climate Audit. Holland has put all the responses and letters up. There are three threads – two beginning with Fortress and a third later one. Worth saving the comments on a Jim Edwards – can you do this Tim?

I’ve saved all three threads as they now stand. No time to read all the comments, but I did note in “Fortress Met Office” that someone has provided a link to a website that helps you to submit FOI requests to UK public institutions, and subsequently someone has made a further FOI request to Met Office and someone else made one to DEFRA. If it turns into an organised campaign designed more to inconvenience us than to obtain useful information, then we may be able to decline all related requests without spending ages on considering
them. Worth looking out for evidence of such an organised campaign.

Tim

Another thing to hide behind, a false claim of an “organised campaign” …

Phil Jones:

To: Gavin Schmidt
Subject: Re: Revised version the Wengen paper
Date: Wed Aug 20 09:32:52 2008
Cc: Michael Mann

Gavin,

Keith/Tim still getting FOI requests as well as MOHC and Reading. All our FOI officers have been in discussions and are now using the same exceptions not to respond – advice they got from the Information Commissioner. As an aside and just between us, it seems that Brian Hoskins has withdrawn himself from the WG1 Lead nominations. It seems he doesn’t want to have to deal with this hassle.

The FOI line we’re all using is this. IPCC is exempt from any countries FOI – the skeptics have been told this. Even though we (MOHC, CRU/UEA) possibly hold relevant info the IPCC is not part our remit (mission statement, aims etc) therefore we don’t have an obligation to pass it on.

Cheers
Phil

So now the Information Commissioner is in on the deal, s/he’s advising them to use the same exceptions not to respond.

Next, Ben Santer chimes in:

to Thomas Karl, Karen Owen, Sharon Leduc , “Thorne, Peter”, Leopold Haimberger , Karl Taylor, Tom Wigley, John Lanzante, Susan Solomon, Melissa Free, peter gleckler , “‘Philip D. Jones’”, Thomas R Karl, Steve Klein, carl mears, Doug Nychka, Gavin Schmidt, Steven Sherwood, Frank Wentz, “David C. Bader”, Professor Glenn McGregor, “Bamzai, Anjuli”

Dear Tom,

My personal opinion is that both FOI requests (1) and (2) are intrusive and unreasonable. Steven McIntyre provides absolutely no scientific justification or explanation for such requests. I believe that McIntyre is pursuing a calculated strategy to divert my attention and focus away from research. As the recent experiences of Mike Mann and Phil Jones have shown, this request is the thin edge of wedge. It will be followed by further requests for computer programs, additional material and explanations, etc., etc.

Quite frankly, Tom, having spent nearly 10 months of my life addressing the serious scientific flaws in the Douglass et al. IJoC paper, I am unwilling to waste more of my time fulfilling the intrusive and frivolous requests of Steven McIntyre. The supreme irony is that Mr. McIntyre has focused his attention on our IJoC paper rather than the Douglass et al. IJoC paper which we criticized. As you know, Douglass et
al. relied on a seriously flawed statistical test, and reached incorrect conclusions on the basis of that flawed test.

I believe that our community should no longer tolerate the behavior of Mr. McIntyre and his cronies. McIntyre has no interest in improving our scientific understanding of the nature and causes of climate change. He
has no interest in rational scientific discourse. He deals in the currency of threats and intimidation. We should be able to conduct our scientific research without constant fear of an “audit” by Steven McIntyre; without having to weigh every word we write in every email we send to our scientific colleagues.

In my opinion, Steven McIntyre is the self-appointed Joe McCarthy of climate science. I am unwilling to submit to this McCarthy-style investigation of my scientific research. As you know, I have refused to
send McIntyre the “derived” model data he requests, since all of the primary model data necessary to replicate our results are freely available to him. I will continue to refuse such data requests in the
future. Nor will I provide McIntyre with computer programs, email correspondence, etc. I feel very strongly about these issues. We should not be coerced by the scientific equivalent of a playground bully.

I will be consulting LLNL’s Legal Affairs Office in order to determine how the DOE and LLNL should respond to any FOI requests that we receive from McIntyre. I assume that such requests will be forthcoming.

I am copying this email to all co-authors of our 2008 IJoC paper, to my immediate superior at PCMDI (Dave Bader), to Anjuli Bamzai at DOE headquarters, and to Professor Glenn McGregor (the editor who was in
charge of our paper at IJoC).

I’d be very happy to discuss these issues with you tomorrow. I’m sorry that the tone of this letter is so formal, Tom. Unfortunately, after today’s events, I must assume that any email I write to you may be
subject to FOI requests, and could ultimately appear on McIntyre’s “ClimateAudit” website.

With best personal wishes,

Ben

Well, he got the last paragraph right, at least. He also thinks that an FOIA request must serve some “scientific justification”, as determined by … well … by the person receiving the request.

Ben Santer to Tom Wigly, 12 Dec 07 (1228330629):

At 01:17 03/12/2008, Ben Santer wrote:

Dear Tom,

One of the problems is that I’m caught in a real Catch-22 situation. At present, I’m damned and publicly vilified because I refused to provide McIntyre with the data he requested. But had I acceded to McIntyre’s initial request for climate model data, I’m convinced (based on the past experiences of Mike Mann, Phil, and Gavin) that I would have spent years of my scientific career dealing with demands for further explanations,
additional data, Fortran code, etc. (Phil has been complying with FOIA requests from McIntyre and his cronies for over two years). And if I ever denied a single request for further information, McIntyre would have rubbed his hands gleefully and written: “You see – he’s guilty as charged!” on his website.

You and I have spent over a decade of our scientific careers on the MSU issue, Tom. During much of that time, we’ve had to do science in “reactive mode”, responding to the latest outrageous claims and inept science by John Christy, David Douglass, or S. Fred Singer. For the remainder of my scientific career, I’d like to dictate my own research agenda. I don’t want that agenda driven by the constant need to respond to Christy, Douglass, and Singer. And I certainly don’t want to spend years of my life interacting
with the likes of Steven McIntyre.

I hope LLNL management will provide me with their full support. If they do not, I’m fully prepared to seek employment elsewhere.

With best regards,
Ben

Dr. Santer, here’s a novel idea. Put enough information out when you publish the work so that your work can be replicated. Put on the web whatever is necessary in the way of code, data, and methods to allow your work to be checked by someone else. If you do that, not only will you not be bothered, but you will be following the scientific method. None of us at CA are doing this to harass anyone, as you claim. We’re doing this because we cannot replicate your work, and thus your work is purely anecdotal rather than scientific.

Phil responds (same email):

Cc: mann , Gavin Schmidt, Karl Taylor, peter gleckler

Ben,
When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions – one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school – the head of school and a few others) became very supportive. I’ve got to know the FOI person quite well and the Chief Librarian – who deals with appeals. The VC is also
aware of what is going on – at least for one of the requests, but probably doesn’t know the number we’re dealing with. We are in double figures.

One issue is that these requests aren’t that widely known within the School. So I don’t know who else at UEA may be getting them. CRU is moving up the ladder of requests at UEA though – we’re way behind computing though. We’re away of requests going to others in the UK – MOHC, Reading, DEFRA and Imperial College.
So spelling out all the detail to the LLNL management should be the first thing you do. I hope that Dave is being supportive at PCMDI. The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian, saying that the email maligned his scientific credibility with his peers!

If he pays 10 pounds (which he hasn’t yet) I am supposed to go through my emails and he can get anything I’ve written about him. About 2 months ago I deleted loads of emails, so have very little – if anything at all. This legislation is different from the FOI – it is supposed to be used to find put why you might have a poor credit rating !

In response to FOI and EIR requests, we’ve put up some data – mainly paleo data. Each request generally leads to more – to explain what we’ve put up. Every time, so far, that hasn’t led to anything being added – instead just statements saying read what is in the papers and what is on the web site! Tim Osborn sent one such response (via the FOI person) earlier this week. We’ve never sent programs, any codes and manuals.

In the UK, the Research Assessment Exercise results will be out in 2 weeks time. These are expensive to produce and take too much time, so from next year we’ll be moving onto a metric based system. The metrics will be # and amounts of grants, papers and citations etc. I did flippantly suggest that the # of FOI requests you get should be another.

When you look at CA, they only look papers from a handful of people. They will start on another coming out in The Holocene early next year. Gavin and Mike are on this with loads of others. I’ve told both exactly what will appear on CA once they get access to it!

Cheers
Phil

Well, that explains why David Palmer and Ms. Kitty Inglis, the Chief Librarian, were so unsupportive. Took a couple of half hour sessions, but at the end of that, rather than being a representative of the FOI process, they were functioning as the personal representatives of Phil Jones. We have a new reason I hadn’t noticed in the FOI law for refusing a request, because the requester posts at CA.

Jones to Ben Santer again, 10 Dec 2008:

Ben,

Haven’t got a reply from the FOI person here at UEA. So I’m not entirely confident the numbers are correct. One way of checking would be to look on CA, but I’m not doing that. I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable! McIntyre hasn’t paid his £10, so nothing looks likely to happen re his Data Protection Act email.
Anyway requests have been of three types – observational data, paleo data and who made IPCC changes and why. Keith has got all the latter – and there have been at least 4. We made Susan aware of these – all came from David Holland. According to the FOI Commissioner’ s Office, IPCC is an international organization, so is above any national FOI. Even if UEA holds anything about IPCC, we are not obliged to pass it on, unless it has anything to do with our core business – and it doesn’t! I’m sounding like Sir Humphrey here! McIntyre often gets others to do the requesting, but requests and responses all get posted up on CA regardless of who sends them.

On observational data, there have been at least 5 including a couple from McIntyre. Others here came from Eschenbach and also Douglas Keenan. The latter relate to Wei-Chyung Wang, and despite his being exonerated by SUNY, Keenan has not changed his web site since being told the result by SUNY! [1]http://www.informat h.org/

The paleo data requests have all been to Keith, and here Tim and Keith reply. The recent couple have come from McIntyre but there have been at least two others from Holland. So since Feb 2007, CRU is in double figures. We never get any thanks for putting things up – only abuse and threats. The latest lot is up in the last 3-4 threads on CA.

I got this email over the weekend – see end of this email. This relates to what Tim sent back late last week. There was another one as well – a chatty one saying why didn’t I respond to keep these people on CA quiet. I’ve ignored both. Finally, I know that DEFRA receive Parliamentary Questions from MPs to answer. One of these 2 months ago was from a Tory MP asking how much money DEFRA has given to CRU over the last 5 years. DEFRA replied that they don’t give money – they award grants based on open competition. DEFRA’s system also told them there were no awards to CRU, as when we do get something it is down as UEA!

I’ve occasionally checked DEFRA responses to FOI requests – all from Holland.

Cheers
Phil

Since he and Mann and the others have already deleted their emails, looks like David Palmer was a bit too late with his excellent advice … however, I did get a “Mentioned In Dispatches”, at least …

Phil Jones to Raymond Pierrehumbert, 16 Jan 09 (1200493432):

Cc: Michael Mann , Gavin Schmidt

Ray,

I have had a couple of exchanges with Courtillot. This is the last of them from March 26, 2007. I sent him a number of papers to read. He seems incapable of grasping the concept of spatial degrees of freedom, and how this number can change according to timescale. I also told him where he can get station data at NCDC and GISS (as I took a decision ages ago not to release our station data, mainly because of McIntyre). I told him all this as well when we met at a meeting of the French Academy in early March.

Cheers, Phil

This is a very clear statement of what he has done. He has refused to release the data, not because there is any logical reason to do so, but “because of McIntyre”. This is shameful, and the fact that the Dave Peters and Kitty Inglis went along with this is dereliction of duty.

Finally, Steven Schneider chimes in to write Ben from Stanford University, 6 Jan 09 (1231257056):

Cc: “David C. Bader”, Bill Goldstein, Pat Berge, Cherry Murray, George Miller, Anjuli Bamzai, Tomas Diaz De La Rubia, Doug Rotman, Peter Thorne, Leopold Haimberger, Karl Taylor, Tom Wigley John Lanzante, Susan Solomon, Melissa Free, peter gleckler, “Philip D. Jones”, Thomas R Karl, Steve Klein, carl mears, Doug Nychka, Gavin Schmidt, Steven Sherwood, Frank Wentz

“Thanks” Ben for this, hi all and happy new year. I had a similar experience– but not FOIA since we at Climatic Change are a private institution- -with Stephen McIntyre demanding that I have the Mann et al cohort publish all their computer codes for papers published in Climatic Change. I put the question to the editorial board who debated it for weeks. The vast majority opinion was that scientists should give enough information on their data sources and methods so others who are scientifically capable can do their own brand of replication work, but that this does not extend to personal computer codes with all their undocumented sub routines etc. It would be odious requirement to have scientists document every line of code so outsiders could then just apply them instantly. Not only is this an intellectual property issue, but it would dramatically reduce our productivity since we are not in the business of producing software products for general consumption and have no resources to do so. The NSF, which funded the studies I published, concurred–so that ended that issue with Climatic Change at the time a few years ago.

This continuing pattern of harassment, as Ben rightly puts it in my opinion, in the name of due diligence is in my view an attempt to create a fishing expedition to find minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work–and then assert that the entire result is thus suspect. Our best way to deal with this issue of replication is to have multiple independent author teams, with their own codes and data sets, publishing independent work on the same topics–like has been done on the “hockey stick”. That is how credible scientific replication should proceed.

Let the lawyers figure this out, but be sure that, like Ben is doing now, you disclose the maximum reasonable amount of information so competent scientists can do replication work, but short of publishing undocumented personalized codes etc. The end of the email Ben attached shows their intent–to discredit papers so they have no “evidentiary value in public policy”–what you resort to when you can’t win the intellectual battle scientifically at IPCC or NAS.

Good luck with this, and expect more of it as we get closer to international climate policy actions, We are witnessing the “contrarian battle of the bulge” now, and expect that all weapons will be used.

Cheers, Steve

PS Please do not copy or forward this email.

Now, why would Dr. Schneider not want his email copied or forwarded … perhaps because he is saying don’t follow the spirit of the law, do as little as you possibly can? He also foolishly thinks that studies can be “replicated” by using different data and different codes … but that says absolutely nothing about the original study and whether it contains any mistakes.

The researchers complain in various places that they do not want to reveal their “primary data” because it is available on the web. While this is often true, as I saw in my FOIA requests to CRU, just saying “I got the information from X” is often totally inadequate. Santer made this charge, that anyone could go the CMIP website and get the data themselves … but unless he says exactly which data from which run of which model, that information is meaningless.

The main impression that I get from the emails is that the various scientists think that I and others are simply doing this to harass them. Nothing could be further from the truth. I have no desire to put any scientist to any extra effort beyond providing what science requires – a full accounting of the data, the methods, and in some cases the computer code used to do the research. Anything more is harassment … but anything less is scientific obstruction.

As I said, the issue is not Trenberth or scientists talking smack. It is the illegal evasion of legitmate scientific requests for data needed to replicate a scientific study. Without replication, science cannot move forwards. And when you only give data to friends of yours, and not to people who actually might take a critical look at it, you know what you end up with? A “consensus” …

REFERENCES:

A Collation of CRU Correspondence, Stephen McIntyre, May 30, 2008,

129 Comments

  1. Paul Penrose
    Posted Nov 26, 2009 at 12:21 AM | Permalink

    I was amused at how they thought a few FOI requests were a deluge, or words to that effect. In fact, if they had just responded honestly to them they would have received a lot fewer of them in the end. But of course we now know why they didn’t. The databases and code are a mess and even they don’t know how it all works. They were probably afraid of not only the criticism they would receive, but maybe even more, the questions about it that they would not be able to answer.

    • bender
      Posted Nov 26, 2009 at 10:56 AM | Permalink

      I think they *feared* a “deluge”.

      There is this warmista meme out there that these FOIAs are pure obstructionism – an organized strategy by the denialists to simply grind the alarmist research to a halt. As though McIntyre and Essenbach were never going to do anything with the data they hoped to receive. But history shows they really did want the data. They really did want to understand how, for example, New York City Central Park temperature record (slim shady) had been turned into a hockey stick by cooling the 1970s (through the use of a “UHI correction”) [Here I'm obviously talking GISS, not CRU. But you get the point.].

  2. bender
    Posted Nov 26, 2009 at 12:25 AM | Permalink

    Willis, you’ve surely read the Harry file by now. Do you think they were in a position to answer any of your questions? Frankly, I’m not sure they could if they tried. It should be easy to provide a list of stations used in a calculation. Should. Perhaps that they couldn’t was a major source of embarrassment and frustration? Perhaps foolish pride prevented Jones from coming clean with a confession that their multi-million research program was a shambles. Just a thought.

    • Tony Hansen
      Posted Nov 26, 2009 at 4:39 AM | Permalink

      What are the most basic questions that they apparently can not answer, but should be able to answer?
      What questions can they answer – with supporting evidence?

  3. Faustino
    Posted Nov 26, 2009 at 12:32 AM | Permalink

    This is a horrifying tale. I have referred this page to The Australian newspaper and urged them to pursue it vigorously, including taking up with the Prime Minister how Australia can proceed with its “Carbon Pollution Reduction Scheme” (sic) prior to all Hadley-CRU data and workings being released and independently verified. Thanks so much, Steve and Willis, for your work in this field.

  4. jfischoff
    Posted Nov 26, 2009 at 12:39 AM | Permalink

    We need more of this. Most still think that these emails are just a random collection. I will eventually get around to organizing them in threads if no one else does.

    Has anyone talked to “Sarah.” It would be interesting to hear her thoughts.

    • Splice
      Posted Nov 26, 2009 at 4:21 AM | Permalink

      Could Sarah be Sarah Raper?

  5. theduke
    Posted Nov 26, 2009 at 12:43 AM | Permalink

    There is an error just short of half-way down. The paragraph starting with “That didn’t help one bit” should not be in block quotes.

  6. theduke
    Posted Nov 26, 2009 at 12:54 AM | Permalink

    This paragraph a little more than half-way down should also not be in blockquotes:

    “Finding ways around FOI requests seems to be a popular sport. This is in reference to Steve trying to get the review comments to Chapter 6 of the UN IPCC Fourth Assessment Report.”

  7. malcolm
    Posted Nov 26, 2009 at 12:58 AM | Permalink

    There is a typo in the line “Ben Santer to Tom Wigly, 12 Dec 07 (1228330629):” – it should be 08 not 07.

  8. Tom Ganley
    Posted Nov 26, 2009 at 12:58 AM | Permalink

    From Steven Schneider’s email;

    “This continuing pattern of harassment, as Ben rightly puts it in my opinion, in the name of due diligence is in my view an attempt to create a fishing expedition to find minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work–and then assert that the entire result is thus suspect.”

    To me, this type of statement is the smoking gun of the emails. It says to me that even THEY don’t have enough confidence in their result to believe it will survive even cursory examination of their work.

    If someone can dispute your result by citing a minor glitch or unexplained piece of code you never had a result in the first place.

  9. dearieme
    Posted Nov 26, 2009 at 1:19 AM | Permalink

    Is there an error in this, Steve?
    “requesting station data received by the Hadley Center (on the basis that they might not be able to assert exemptions claimed by Hadley Center. ) “

  10. theduke
    Posted Nov 26, 2009 at 1:33 AM | Permalink

    Thank you, Willis. A fascinating account achieved by blending what we knew then with what we know now.

    It appears that CRU data was always a mess and any release of it would not only have proved a major embarrassment, but would have weakened the momentum toward regulation of emissions world-wide.

    This was science that was always conducted with one goal in mind: forceful government regulation of industry and economies on an international scale.

    My sense is that they sincerely believed it was necessary to do this. I don’t, however, believe that they even proved to themselves it was necessary to do this.

  11. K11
    Posted Nov 26, 2009 at 2:15 AM | Permalink

    Wow. Great summation. Should be a “must read”.

  12. jef
    Posted Nov 26, 2009 at 2:16 AM | Permalink

    It is rich that the very same people who argue that climate (or many other things) is too complex for the little people, therefore we need experts and techocrats to run the world, are the same who say that complex programs are bound to have some errors.

    It is way too rich when the truth of the “complexity” is found to be a undocumented mess.

    Yes, yes, mistakes were made.

  13. Cecil
    Posted Nov 26, 2009 at 2:27 AM | Permalink

    This post is the missing context. The context we were told we needed to properly understand the emails.

  14. Posted Nov 26, 2009 at 2:29 AM | Permalink

    I strongly support the statement about the transparency of information behind published scientific papers. Reproducibility of results is indispensable for science to be true science.

    But in these e-mails one can trace the idea that information should be made accessible to competent scientists, not just to general public. This looks like an excuse — we experts know how to use our data and reproduce our results, if you cannot (being so unprofessional) — you are free to call us “bad scientists” or whatever, we do not care. There is some reason in this idea, to save time for research instead of answering incompetent requests. Like, e.g., anthropologists and linguists involved in deciphering the language of a perished civilization may not care very much about whether the laymen (knowing practically nothing about the issue) consider them professional or not. For them, it would be important that their sponsor counts them professional, nothing else. Provided it is the case, they can be enjoying the full happiness of a pure academic research.

    However, this logic is absolutely unacceptable in the present case of climate science. In my opinion, climate scientists should just forget about it, for ever, or change their profession. Until very recently, science has been producing technologies that were consumed by the society without bothering too much about the credibility of science behind the innovations. Bad jet planes simply did not fly. It has been a natural selection of incompetitive ideas.

    Now science claims to be producing global forecasts and prescribtions to handle global problems to affect everybody’s life. There are untested predictions — every person has only one life of his/her own to test them, and there is only one planet. So the issue of scientific credibility has grown in importance in a manner unprecedented in the entire human history. Now, yes, scientists involved in global change research must allocate time to make all the information understandable and accessible to every educated Earth’s citizen. Only in this case the taxpayers’ money currently given to climate research could be justified.

    I think that the published e-mails indicate that such an understanding has yet to penetrate the culture of climate science. Global change is a recent problem, there has not been sufficient time for the generation of scientists to change and face the new realities.

    • Basil Copeland
      Posted Nov 26, 2009 at 10:32 AM | Permalink

      The defense that scientists do not want to be bothered with FOI’s from people not competent to understand what they are doing, but might not object to sharing the information with “qualified” people in their field fails on this point: THIS AIN’T ROCKET SCIENCE, PEOPLE! We’re not talking here about a GCM that embodies untold number of assumptions about complex physical processes that only someone with a Ph.D. in thermodynamics could understand. I’ve seen that confusion in the popular press reporting of this incident. We’re not talking about a “computer model” here at all. We’re talking about how individual station climate data — the main interest is temperature, but CRU TS has some other climate data as well — is taken from stations all over the world, for varying lengths of time, and combined into a global temperature data set. Now, admittedly, that is not a simple process. How do you “average” all this together? How do you handle data “inhomogeneities?” And so forth. But there is little to be done here, if anything, that requires much more than an M.S. in anything, let alone a Ph.D. Basically we’re looking here at something that is a basic combination of programming, statistics, and meteorology. (I’m writing here primarily of CRU TS. But it would mostly apply to the tree ring stuff, too. That ain’t rocket science, either.)

      So what does it say that these blokes didn’t want to reveal the details behind their temperature data? Well, Harry_Read_Me.txt shows reveals a lot about the state of their programming. The emails and code snippets show a lot evidence of questionable statistics practices (“hide the decline”). But I think the bottom line here is that precisely because this isn’t “rocket science” they knew that if they laid it all out for others to see, there would be enough “judgment calls” in how they handled the data that it would lead to questions about whether the temperature data show what they claim it shows.

  15. Scott Gibson
    Posted Nov 26, 2009 at 2:55 AM | Permalink

    @Anastassia-

    I would concur with your idea that they only wanted to provide information to “competent scientists,” if they hadn’t sought to limit peer review to only their own yes-men. It was the lack of true peer review that spawned Steve’s inquiries.

    I agree with the rest of your comment.

  16. Posted Nov 26, 2009 at 3:01 AM | Permalink

    Thanksgiving Day Invitation to the Climate Change Skeptics

    I’m a working engineer and an amateur scientist with a lifelong interest in oil depletion (and am obviously not funded by Mobil).

    All the data and what models exist seem to indicate that we have reached a peak in oil production and a long steady decline is ahead of us.

    This chronic oil depletion crisis happens to be occurring in parallel with climate change awareness, although peak oil has gotten much less publicity. This is serious stuff, yet the government barely acknowledges its significance because it has serious ramifications on the economy.

    So, unfortunately, much of the public has no awareness of our petroleum predicament, and further, a significant number of people actively question the phenomena of peak oil.

    Like it or not, a sizable cross-section climate change skeptics also happen to be oil depletion skeptics. There is no arguing this, as you hear it on radio talk shows constantly and you read about it in opinion pieces such as by George Will and a host of others.

    So the challenge to the climate science skeptics who claim to be hot-shot statistics wizards — why don’t you “audit” those of us who post on http://theoildrum.com and other oil depletion blogs?

    We are waiting, all of our data is open, we only discuss matters in open daylight, and we will kick your asses with our skillz. Seriously, we have to change our energy policy and soon. Whether it is due to a crisis in climate change, or due to an inexorable decline in easily accessible petroleum, things will change, like it or not. Here is your chance once more to prove someone wrong. Good luck.

    And BTW, whatever data we uncover does not belong to us; oil reserve estimates are usually locked up in some corporate or foreign nation-state vault totally immune to FOIA requests. Double good luck in getting that. You will get to see how the people with the REAL money play hardball.

    It looks like you have finished mopping the floor with the climate science crowd, and you should be ready for your next conquest. Yes, we are waiting for you, and we knew this day would come.

    • Raven
      Posted Nov 26, 2009 at 3:43 AM | Permalink

      Climate and Peak Oil are two very different issues calling for very different policy responses. For example, if Peak Oil is our primary concern then would want ensure we have a cheap and reliable electricity supply by using Coal and Gas and we would want to aggressively expand unconventional sources like the Oil Sands and Oil Shales. We would also want to explore the development of cheap nuclear powered engines for cargo ships.

      I suspect few people here would object any of those policies and no one in the political area is proposing them. That means that climate, for now, is going attract more interest from amateur auditors.

      • Craig B
        Posted Nov 26, 2009 at 5:47 AM | Permalink

        snip – we have many new posters. Blog editorial policies discourage/prohibit discussions of policy which would otherwise swamp the blog.

    • Dave Dardinger
      Posted Nov 26, 2009 at 9:06 AM | Permalink

      snip – non-climate

    • Posted Nov 26, 2009 at 9:43 AM | Permalink

      Twit, I share your attitudes to a large extent.

      I’ve hung in with Climate Science for a while now as I felt it was essential to get ClimateGate exposed for the huge, global, official-science-supported shambolic group think corruption that it is. It broke my heart but it also inspired me to do the good science needed. Click my name; I’ve also had two pages at WUWT. I felt that we had to clear root corruption in science plus total arrogant unawareness of such – what Willis is doing – first.

      snip – OT

      Feel free to contact me.

    • Tinker
      Posted Nov 26, 2009 at 9:44 AM | Permalink

      Robinson,

      “natural warming deniers” – OMG – that is so sweet!!! Hadn’t heard that one before. Gonna use it, only capitalized (NWD’s). Puts the AGW’s on the defensive.

      Thanks!

    • Marie Elks
      Posted Nov 26, 2009 at 1:52 PM | Permalink

      If it happens, and I’m not convinced it will, someone will come up with the solution independent of the UN, US Gov’t, etc. That’s how real science and development occurs–organic, rather than regulated.

      The folks here are quite busy with new information. Perhaps you could use your science skills and try to debunk your own work before asking someone else to do it for you.

  17. Alan Wilkinson
    Posted Nov 26, 2009 at 3:07 AM | Permalink

    Twit, I think you should be directing your challenge to the AGW alarmists.

    After all, if you are right they have nothing to fear because there will be no oil to make CO2 from.

  18. David Harrington
    Posted Nov 26, 2009 at 3:22 AM | Permalink

    I have created a petition on the UK Number 10 web site to gather signatures for an inquiry into this affair. If you are a UK citizen or resident please add your name to this petition.

    http://petitions.number10.gov.uk/HADLEY-LEAK/

    Thanks

    • Splice
      Posted Nov 26, 2009 at 4:28 AM | Permalink

      This is not helpful as there are now 2 petitions on the number10 website and could prove confusing.

    • ncmoon
      Posted Nov 26, 2009 at 5:00 AM | Permalink

      Better if people sign this one:

      http://petitions.number10.gov.uk/UEACRU/

      It already has ten times as many ‘signatures’.

    • jcspe
      Posted Nov 26, 2009 at 5:11 AM | Permalink

      David Harrington,

      I am a Yank, so I wouldn’t be signing your petition anyway, but one thing puzzles me.

      Why are you asking for a investigation in the leaking rather than an investigation into the information presented or generated by the items that were leaked?

      If my reading comprehension is up to task, some panel could fulfill your request by investigating how, when and where the information was leaked without ever giving one minute’s consideration as to what was actually within the leaked material.

      Others might disagree. Food for thought.

    • Posted Nov 26, 2009 at 5:21 AM | Permalink

      Isn’t you petition asking the wrong question? Surely it should focus on the obfuscation that was revealed by the leak.

    • P Gosselin
      Posted Nov 26, 2009 at 9:16 AM | Permalink

      Oh yes of course. Thanks for reminding us.
      The real crime here is that someone has exposed the truth.
      Such temerity!

    • Chris Schoneveld
      Posted Nov 26, 2009 at 9:20 AM | Permalink

      Hadley?

  19. Joe
    Posted Nov 26, 2009 at 3:24 AM | Permalink

    Anastassia,
    There is a subtle point in your comment that is important to note :

    “Until very recently, science has been producing technologies that were consumed by the society without bothering too much about the credibility of science behind the innovations. Bad jet planes simply did not fly….”

    The consensus on climate change is always claimed among “scientists”. But, engineers build the technology that we use, those jet planes that do fly. Scientists attempt to model the real world into simple equations while engineers go the other way, adapting those equations to work in Mother Nature’s chaotic cauldron. Which profession is by necessity more in touch with reality? Ask any engineer what he or she thinks about this subject.

  20. debreuil
    Posted Nov 26, 2009 at 3:50 AM | Permalink

    snip – no policy discussion

  21. MattA
    Posted Nov 26, 2009 at 4:00 AM | Permalink

    Will David Palmer and Ms. Kitty Inglis, the Chief Librarian, be disciplined for not fullfiling their duties in what appears to be a less than fair and balanced manner?

    Who has oversite of this process?

    This episode clearly shows that internal investigations within the university stucture are suspect and in order to avoid the appearance of bias an outside body should conduct an investigaion of the above.

  22. Jonathan
    Posted Nov 26, 2009 at 4:52 AM | Permalink

    Bender, you are probably right that UEA/CRU couldn’t provide much of what Willis was asking for. One reason why I made such a specific FOI request (for the data as sent to Webster) was to identify something that they certainly could provide if they wished, thus providing some distinction between can’t and won’t.

    MattA: Dave Palmer is probably safe, but I wouldn’t like to be Kitty Inglis or Jonathan Colam-French right now.

    • bender
      Posted Nov 26, 2009 at 11:02 AM | Permalink

      Very good point, Jonathon. I think Willis thought his request was simpler, less onerous. Because it was just a station list, no data. I think your request would have been simpler to respond to. Remind us: what was the outcome of your process?

      • Jonathan
        Posted Nov 26, 2009 at 1:42 PM | Permalink

        My request was refused. I appealed. The appeal was refused. I am appealing to the Commissioner. Full details on the CRU Gong Show thread.

  23. Raven
    Posted Nov 26, 2009 at 5:05 AM | Permalink

    I see RC has ‘moved on’ and claiming that all of the code and database problems are fixed in HadCRUTv3. A neat trick considering the fact that they ‘lost’ the raw data.

    • Posted Nov 26, 2009 at 5:56 AM | Permalink

      I think they are attempting a rather clumsy managed retreat and are several days behind where this is going.

      Now we’ve digested the emails, and picked apart the code/data manipulation – we’re onto the actual data hunt. The NZ stuff looks odd and has pointed us in a new direction.

      There are far too many of us compared to them – thank God Al Gore invented the internet :D

      • Peter S
        Posted Nov 26, 2009 at 8:32 AM | Permalink

        But if it proves to be inconvenient – can Al Gore uninvent the internet?

  24. Posted Nov 26, 2009 at 5:27 AM | Permalink

    I think we can safely exclude the option, that the FOIA2009.zip file was prepared by CRU as to be submitted to skeptics in case, had they could not avoid the FOIA request.

    CRU would rather delete all these e-mails (as they did before) than openly publishing their criminal behavior. The person which prepared the zip file knew exactly what to put in.

    Excellent summary by author.

    • Calvin Ball
      Posted Nov 26, 2009 at 4:42 PM | Permalink

      Agreed. I’m not sure where that’s coming from (though I have my suspicions), but it doesn’t make any sense, that 1) they would prepare it, and then sit on it, 2) that someone would go through the trouble and risk of leaking it if it was going to be released anyway, and 3) that it would contains all of those emails that aren’t germane to much of anything.

      It’s completely illogical, and may be media speculation, but is more than likely the product of the ‘disinformation machine’ that I hear tell about.

  25. debreuil
    Posted Nov 26, 2009 at 6:48 AM | Permalink

    I think slowly, one by one, people are actually opening and reading some of that stuff for themselves. That is all it takes. Nothing like primary source material.

  26. Posted Nov 26, 2009 at 6:48 AM | Permalink

    Check out Update 3 to this article – http://briefingroom.typepad.com/the_briefing_room/2009/11/breaking-nzs-niwa-accused-of-cru-style-temperature-faking.html

  27. DABbio
    Posted Nov 26, 2009 at 6:51 AM | Permalink

    Utterly convincing. What an incredible abrogation of scientific integrity that Willis evokes.

    Is it well understood among the (healthily) skeptical crowd that someone has to actually take the lead, get organized, get a bunch of money–if you do the first two things, I will be in line to donate– and use that to go out and hire the best legal team that can be had in order for this to move forward? Someone has to actually sue someone. Notwithstanding that, a reasonable prelude would be some sort of government-sanctioned investigation. How does that happen in Merrie Old, I wonder?

  28. Stacey
    Posted Nov 26, 2009 at 7:30 AM | Permalink

    This is an excellant post, although I would add that not only do the actions strike at the heart of science they also strike at the heart of democracy.

    The will of our uk paliament not only subverted by the scientists in question but also in collusion with FOI officers and the Commissioner’s office.

    • Posted Nov 26, 2009 at 8:58 AM | Permalink

      The whole ICO and FOI collusion are really bad.

      All the FOI decision makers I’ve worked with have been total pain in the arse honesty sticlkers – they understand the big picture, but frankly don’t care if they drop you right in it.

      I can only assume that either Jones browbeat his/too big to fail or they are in the wrong job.

      Richard Thomas [UK Info Commissioner] has a track record for dropping even HMG right in it over the Cabinet Minutes on going to war with Iraq.

  29. James RS
    Posted Nov 26, 2009 at 7:35 AM | Permalink

    A lot of published responses to this affair has called this the nail-in-the-coffin for global warming. Is that really true?

    Is there any other line of scientific work, independent of the stuff the Team promoted that the warmers can point to that “takes up the mantle” so to speak. Is there anything left to serve as a solid foundation for their claims? Is there any other work that uses a valid set of temperature data that supports the AGW premise? Is there even a valid set of data that is accepted as reliable?

    I know this circus has just started and perhaps this question comes too soon. I guess I am trying to jump ahead to the end of the book. Can’t help it. I am just wondering how big a hole this will (could, might) leave in AGW “science” if this plays out the way like it looks like it may.

    • Raven
      Posted Nov 26, 2009 at 7:46 AM | Permalink

      Stories of the death of climate alarmism have been greatly exagerrated.

      The most significant lesson from this scandal is the peer review process has been corrupted. i.e. while the alarmists were busy criticizing sceptics for not publishing peer reviewed papers they were working in the background to blackmail journals that accepted skeptical papers.

      What this means is the peer reviewed literature is now biased towards the alarmist view because sceptical papers were rejected and, more importantly, they likely never got written in the first place as scientists figured out the ‘fix’ was in.

      So the next time an alarmist tells you that the science is settled because the overwhelming majority of peer reviewed papers support alarmism you can tell him that peer review in climate science has been shown to be corrupt so we cannot really know what the true state of science.

      • k
        Posted Nov 26, 2009 at 1:10 PM | Permalink

        The motivation for Phil, Gavin and other AGWers to continue to claim that AGW is still valid because it’s been peer reviewed is transparent. At this moment in time, the true dereliction of duty resides with the main stream media. They continue to parrot that line even though they have now been presented with overwhelming evidence that the peer review process has been subverted.

      • Eyesapoppin'
        Posted Nov 26, 2009 at 2:18 PM | Permalink

        What I love is how, when the peer-reviewed journal the antis were writing in didn’t follow the AGW agenda, they de-peered it!

    • Dave Dardinger
      Posted Nov 26, 2009 at 9:32 AM | Permalink

      If you look at the lists of “proofs” of AGW provided, the only ones which aren’t very iffy are the ones which rely on melting ice: Glaciers, ice caps and sea ice. There, at least theoretically, there are objective measurements available. Note that this doesn’t apply to using Ice cores as temperature proxies as there we run into warmers like Lonnie Thompson who won’t release his raw data and where Steve M has shown some major problems with the dating.

      The biggest problem with melting ice bodies is that for glaciers there has not been an exhaustive list of them and their ice balance. Most of the ones studied are adjacent to major human habitation and subject to various land use changes which can affect glaciers (Kilimanjaro for example). The same would be true in the alps and many of the SA glaciers. Add to this the fact that we’ve been coming out of the LIA and it’s a stretch to make definitive statements. Then there’s arctic sea ice melting, which CA has had a series of threads on, and while it’s certain the general trend is downward, it’s bounced back a bit the past couple of years and it’s not certain the trend will continue. In addition, the thermodynamics of cooling the earth (which Steve M generally doesn’t like discussed here so I’ll make this short), makes it favorable to release excess energy, as from CO2 and its feedbacks, in places where the temperature can be as low as possible to avoid large increases in entropy. This means the arctic sea (since Antarctica is too cold to release much energy). Thus, the less ice there is, the more energy can be released by either ice formation or water evaporation or simple IR release from surface water. Thus the Arctic ocean is the best radiator an overheated earth has, and IMO it’s nowhere close to being saturated in heat release ability.

      • Bernie
        Posted Nov 26, 2009 at 10:23 AM | Permalink

        Matt Briggs has an interesting take on the evidentiary value of glacier melting as proof of AGW. I share his view. To prove the theory of AGW one has to make predictions that are distinctive to the mechanism proposed in this theory. The melt back of glaciers is too general and can be expalined as much by the very vaguely worded theory of a recovery from the LIA. The importance of the hockey stick in part stems from the notion of unprecendented warming – but there still needs to be a specific prediction that differentiates futue warming from other non-CO2 based theories that also predict warming. See here: http://wmbriggs.com/blog/?p=1373

      • bender
        Posted Nov 26, 2009 at 11:07 AM | Permalink

        Has the local effect of black carbon on ice melt been adequately disentangled from any global atmospheric effects? Dirty ice melts fast.

    • Posted Nov 26, 2009 at 9:54 AM | Permalink

      James RS, my studies over 18 months have found serious below-water-line holes in every single warmist claim. This is the proverbial tip of the iceberg that holed the “unsinkable” Titanic. A sad day for official Science yet a good day for all those who have held on for the integrity in Science they once knew and still believe in.

      Click my name if you want a general primer in skeptical Climate Science – and my story. A bit out of date now but still 100% relevant.

  30. RomanM
    Posted Nov 26, 2009 at 8:25 AM | Permalink

    Great narrative, Willis! It is a lot easier to promote an agenda if one can do it behind a curtain.

    One of the defenses of the emails has been that they were being interpreted “out of context”. You have certainly shown that the wrongful actions in avoiding FOI were not one-off abberations, but rather an ongoing mode of behaviour by a group of people arrogant enough to think that they were above the law. It also makes it more obvious what the proper interpretion of the “hiding the decline” and “trick” comments should be as well.

    • Posted Nov 26, 2009 at 10:00 AM | Permalink

      I second that. This is the best narrative I’ve come across and Bishop Hill plans to include it, not in his forthcoming book but one after that about the records, I believe.

  31. Posted Nov 26, 2009 at 9:12 AM | Permalink

    More strange data discovered – see page 11 of comments, first post

    http://blogs.news.com.au/heraldsun/andrewbolt/index.php/heraldsun/comments/tips_for_thursday_november_26/P220/

  32. Posted Nov 26, 2009 at 9:20 AM | Permalink

    Could we read the the freedom of Information act requests by Mr. Eschenbach?

  33. Posted Nov 26, 2009 at 9:56 AM | Permalink

    Can any help me please to understand what i see..
    I download the Monthly series of HadCRUT3 Diagnostics: global average (NH+SH)/2
    they are rere:

    http://hadobs.metoffice.com/hadcrut3/diagnostics/global/nh+sh/

    Their data format is:
    * Column 1 is the date.
    * Column 2 is the best estimate anomaly.
    * Columns 3 and 4 are the upper and lower 95% uncertainty ranges from the station and grid-box sampling uncertainties.
    * Columns 5 and 6 are the upper and lower 95% uncertainty ranges from the coverage uncertainties.
    * Columns 7 and 8 are the upper and lower 95% uncertainty ranges from the bias uncertainties.
    * Columns 9 and 10 are the upper and lower 95% uncertainty ranges from the combined station and grid-box sampling, and coverage uncertainties.
    * Columns 11 and 12 are the upper and lower 95% uncertainty ranges from the combined effects of all the uncertainties.

    while i triing to create a plot, by error i do and find any “strange”

    I create a plot of the COL8 – COL7 operation result.
    the upper minus the lower 95% uncertainty ranges from the bias uncertainties.

    More i see.. more strange is to me..

    So i uploat it to google docs..

    What happen between 1937 – 1960?

    http://spreadsheets.google.com/pub?key=tVV0JAI6BxY8dqPiTWXlWUg&gid=5

    Can any explain it?
    1850-2009 plot:

    http://spreadsheets.google.com/pub?key=tVV0JAI6BxY8dqPiTWXlWUg&gid=4

    What i am seeing?

    Sorry for my errors in english.. not my language.

  34. Charlie
    Posted Nov 26, 2009 at 10:29 AM | Permalink

    In addition to the FOI act, there is also Section 515 of Public Law 106-554, known as the Information Quality Act.

    This sets standards that US Goverment agencies are supposed to comply with as to quality, reliability, and transparency of information that they disseminate.

    Wikipedia has a good description. http://en.wikipedia.org/wiki/Data_Quality_Act

    Organizations such as NOAA and NASA have procedures through which one can request corrections. It has been rarely used. In the annual reports issued by the OMB, most government agencies report no requests at all, and other report just one or two.

    When you see bogus or unsupported conclusion posted or disseminated by US government agencies, consider making a formal Request for Correction.

    All it takes is a well thought out and written e-mail.

  35. Barry R.
    Posted Nov 26, 2009 at 10:39 AM | Permalink

    I had an idea: I think most of us suspect that at best the Phil Jones team screwed up the data retention and programming to the point where nobody on their team actually knows how they’re getting their results. If that’s the case, they are hiding behind the skeptic/warmist debate to keep from getting exposed. The question then becomes, how do you put them in a position where the question at the font of everyone’s mind is the quality of their data and programming?

    I suggest that skeptics push the idea of a non-partisan data processing practices audit conducted by people acceptable to both CA and the Jones. The people involved could sign a non-disclosure agreement that covers any data that Jones can prove he has agreed not to disclose. It might cover any specific Intellectual Property claims, though that’s a slippery slope that it is dangerous to get on.

    That audit should report back on:

    a) Whether the results can be replicated
    b) The algorithms that are being used in the process. (In other words the logic or lack thereof of whatever they are doing from raw data to finished product)
    c) The quality and reliability of the data and programming code.
    d) They would also release any and all data subject to FOI requests that are not specifically covered by their non-disclosure agreements.

    That doesn’t do everything that needs to be done, but it strips away the apparently phony defenses Jones and company have been using. They almost certainly wouldn’t agree to it if the data and programming is as screwed up as it looks like it is, but their refusal to allow something like this would be very difficult to defend.

    Just an idea to bounce around.

    • bender
      Posted Nov 26, 2009 at 11:12 AM | Permalink

      It is unfathomable that with the size of their research budgets and the amount of global travel they were doing that they could not afford a real, full-time set of programmers. I keep thinking: poor Harry. Back at UEA trying to unscramble spaghetti while Phil travels the world, behaving like royalty.

      • Posted Nov 26, 2009 at 11:33 AM | Permalink

        I agree – Jones had £13.7m from the tax payer over 10 yrs – WTF was he spending it on?

      • Coalsoffire
        Posted Nov 26, 2009 at 1:30 PM | Permalink

        The CRU crew were essentially a ponzi scheme. You might just as well ask Bernie Madoff what he was spending his money on. It’s irrelevant to the issue of what he was “reporting” was happening to the money. In this case the paper payback to “investors” was a temperature reconstruction that fit their bias. Like Bernie’s investors they were tickled to death to see that their investment was paying off. The troubling fact that nothing was audited or confirmed was easy to overlook when the results were so favourable to their perceived interests. Had the temperature record not shown what they wanted (or if Bernie’s investors reports not shown wonderful returns) they would have been all screaming for an investigation. The old saying “never look a gift horse in the mouth” applies to this. Imagine if a skeptic based group was regularly publishing temperature results that showed a dramatic cooling, Warmists would be howling to see the data, the software, the whole thing. But as long as the results fit the dogma better to just leave things the way they are. Ponzi scheme victims are the least cooperative victims in all of criminal justice because they think their only hope of recovery is to keep playing the game. They are the ultimate deniers. We will see this in the supporters of AGW as well. They will NOT want anyone sniffing around in the CRU methodology. Especially anyone who is independent. They will behave exactly as ponzi scheme victims behave. Shouting down anyone who actually tries to question the scheme.

  36. Terry
    Posted Nov 26, 2009 at 10:52 AM | Permalink

    This is off-topic but I did not know where to post this. Feel free to delete this or move it to a more appropriate spot.

    There seems to be some breaking news down in New Zealand on revelations regarding their scientists fudging the temperature record.

    http://briefingroom.typepad.com/the_briefing_room/2009/11/breaking-nzs-niwa-accused-of-cru-style-temperature-faking.html

  37. theduke
    Posted Nov 26, 2009 at 12:01 PM | Permalink

    Jonathan wrote: “MattA: Dave Palmer is probably safe, but I wouldn’t like to be Kitty Inglis or Jonathan Colam-French right now.”

    If they start going after accomplices in this affair and not Phil Jones, you will know the fix is in. It’s clear to me from the emails that these people were bullied into this by their local superstar, world-renowned climatologist Phil Jones. The question needs to be asked: did Jones have the juice to get them fired or make their lives miserable if they didn’t comply with what were clearly his demands in this matter?

  38. Posted Nov 26, 2009 at 12:05 PM | Permalink

    I had already read Eschenbach’s account elsewhere but it is good to have it re-posted here.

    I notice that Eschenbach gave up after winning a small victory:

    At that point, I let it go. I had a small victory, we got a list of the stations.

    After reading this account and the emails in general, I am beginning to think that legal actions are going to be needed to get these people to comply with the spirit of laws, and only to the extent that a lawyer can show that the request meets the letter of the laws as well.

    I noticed in the last couple of days that CEI is sueing Goddard and Nasa. Good for CEI, but they will always have a right-of-center smell that will be used against them even if they find something impoortant.

    It would be better if some sympathetic lawyer(s) came forward to do some pro bono work on behalf of CA or other scientists on the relatively skeptical side of the debate. After the emails, it seems perfectly clear that legal force is the only thing certain people will respect.

    I’m afraid I don’t much trust government-sponsored audits or commissions.

  39. Posted Nov 26, 2009 at 12:18 PM | Permalink

    DAMNING http://chiefio.wordpress.com/2009/11/03/ghcn-the-global-analysis/

    Manipulation of temp station selection – like having 3/4 in California on the beach WTF?

    My, this is a whole new way of collecting data – if you can fiddle the numbers, do so – if not change the selection of data points to those most likely to give you the result you want by truncating those you don’t like.

    *WTF*

  40. Gerald Machnee
    Posted Nov 26, 2009 at 12:24 PM | Permalink

    RE: Subject: Re: WMO non respondo
    ***… Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it. …
    Cheers Phil***
    This creates the impression that Phil knew there were problems which could be found quite easily.

  41. Posted Nov 26, 2009 at 12:30 PM | Permalink

    This sums the CRU et al crap up

    “The Pacific Islands and Australia / New Zealand:

    http://chiefio.wordpress.com/2009/10/29/ghcn-pacific-basin-lies-statistics-and-australia/

    http://chiefio.wordpress.com/2009/10/23/gistemp-aussy-fair-go-and-far-gone/

    And one of my favorites where we see how one island can shift the whole region:

    http://chiefio.wordpress.com/2009/11/01/new-zealand-polynesian-polarphobia/

    The end of it all is that the entire Pacific Basin is substantially flat on temperatures. Hard to have “Global Warming” if the Pacific is not participating. Australia and New Zealand show warming, but only due to thermometer change artifacts. For New Zealand, it is one single cold thermometer: And when that one is deleted from the whole record, not just the last few years, New Zealand has no “Global Warming” either.

    Hard to have “Global Warming” when the 1/2 of the planet that is the Pacific Basin is dead flat with only a small “ripple” as the PDO flips state every 30 or so years.”

    http://chiefio.wordpress.com/2009/11/03/ghcn-the-global-analysis/

  42. Chris Schoneveld
    Posted Nov 26, 2009 at 12:54 PM | Permalink

    OT. I hate these big colored logos on the side of each comment.Very distracting and cumbersome. Also, there is too much white space between comments. I have to scroll down too often to continue reading. I hope this mirror site will soon be over.

  43. Fred
    Posted Nov 26, 2009 at 1:24 PM | Permalink

    “It is unfathomable that with the size of their research budgets and the amount of global travel they were doing that they could not afford a real, full-time set of programmers.”

    Even more amazing is they couldn’t afford an Ethics Officer.

    These guys are seriously ethically challenged. They make the old Enron Trading Desk Team look like paragons of ethical virtue.

  44. steven mosher
    Posted Nov 26, 2009 at 2:15 PM | Permalink

    Slightly OT, but when I say free the data and free the code THIS is what I am aiming at.

    http://kc.vanderbilt.edu/quant/Seminar/ReproducibleAnalyses.pdf

  45. sylvainr
    Posted Nov 26, 2009 at 2:23 PM | Permalink

    I just had a great time reading this excellent post ! Thanks for it.

    My opinion : a scientist discovering something important (or even just something) should be happy to give others all he can to prove his point.

    Having said that, hiding data and processes is at least suspect.

  46. Jonathan
    Posted Nov 26, 2009 at 2:29 PM | Permalink

    The Duke: The FOI people are employees of UEA not the CRU, and were not directly answerable to Phil Jones. They had to be persuaded to cooperate; they weren’t coerced – though of course Palmer may have been pressured by his superiors.

  47. geo
    Posted Nov 26, 2009 at 2:36 PM | Permalink

    Ohnoes, they’re in “double digits” of requests at CRU over a two year period! The horror. This is the “bombarded” that Jones complains of?

    Really excellent piece here putting all the FOIA bits together. . . which paints a damning picture that must be actionable. . . if anyone in authority will act on it.

    The attitude of ownership of data comes through so strongly from Jones. It’s *his* data –not just to use, but to own, provide, not provide, or destroy at his own and sole will. Really quite apalling and (I sincerely hope!) entirely wrong in law.

    The most despicable and apalling moment in the whole thing is where he talks about just deleting the station data file rather than see Steve McI get his hands on it. Jones is tempermentally unsuited for the high responsibilities he currently has, and that he could even have that thought –in “the heat of the moment” or not– proves it.

    If this was in the US, this would have RICO written all over it for some district attorney to go after.

  48. Posted Nov 26, 2009 at 2:41 PM | Permalink

    We know now, thanks to courageous fighters for the truth, as Steve McIntyre, what the meaning of “consensus” was: it meant “conspiracy”.
    If Copenhagen goes on it will be a shameful gathering.

  49. anon
    Posted Nov 26, 2009 at 2:58 PM | Permalink

    This is slightly off topic, but it seems to me the constant focus on Phil Jones and his shenanigans — though necessary simply to expose the extent — may ultimately be a problem.

    The constant calls for his resignation seems to me that the establishment cabal will simply use him as the fall guy, along with one or two other light weights, and then it will be back to usual with the damage control spin being “there were a few rogue scientists and they have been released and so now everything is fine — move along nothing to see here.” There might be a whitewash investigation that will come to the same conclusion.

    This is the standard cover-up damage control play in these situations, and has been used very effectively time and time again to appease the public that nothing serious happened and get back to “business as usual” — i.e. nothing really changes: new players/leaders, same game.

    I’ve yet to see truth really prevail in any major scandal involving any large entrenched establishment (military, govt, academic, etc.) despite intense heat from the internt because too many vested interests are involved, too many people have too much too loose and so whatever the reason ( pay-cheques, reputations, etc.) everyone joins the bandwagon to have a fall guy out of self-interest and preservation.

    Not sure what the solution is.

    Furthermore, as we’ve seen in the spin from Phil et al in the few statements that have been issued so far, is that the CRU data-set is one of 4 temprature data sets and all 4 are consistent with each other. Given that we now know CRU’s is essentially doctored rubbish, it is self-evident the others must be too as it seems the all the custodians are a very small clique who are directly or indirectly related to the hockey team.

    Therefore it seems to me that the data from these other organizations needs to be outed also and unless there are similar FOI requests pending (or stalled — which would indicate something to hide again) at the US institutions, we’re going to find the spin “we’re only one of 4 data-sets and we’re all consistent with each other” win over the public. Because it is a reasonable explanation absent evidence (read leaked data) to the contrary from the other organizations.

  50. Posted Nov 26, 2009 at 3:05 PM | Permalink

    Delingpole is picking up this blog

    http://chiefio.wordpress.com/2009/11/03/ghcn-the-global-analysis/

    text here “@platosays – thanks for that fascinating link. Will post on it if I have time. Confirms what Anthony Watts – of Watts Up With That – said in a very interesting talk he gave in Brussels at Roger Helmer’s conference last week. He showed dozens of photos of officially weather stations which had effectively been rendered useless by their repositioning (eg near heating vents, in the middle of car parks, at airports).”

  51. ThinkingScientist
    Posted Nov 26, 2009 at 3:51 PM | Permalink

    Without wanting to draw too much attention to the comments on peak oil by twit but twit said:

    “And BTW, whatever data we uncover does not belong to us; oil reserve estimates are usually locked up in some corporate or foreign nation-state vault totally immune to FOIA requests. Double good luck in getting that. You will get to see how the people with the REAL money play hardball.”

    As someone who works in the oil industry I think that, whilst the statement is true for some nation states, for any stock exchange listed oil company then they go to great efforts to fully disclose there oil reserves as failure to do so would (a) cause their share price to fall and (b) get them into serious trouble as it did Shell a couple of years back because the USA SEC regulations for proper disclosing of reserves are VERY strict.

    All public quoted oil companies provide very detailed annual public reports on their oil reserves in several probability categories referred to as P1, P2 and P3 for proven, probable and possible. As I said above, the definitions required by SEC regulations are very strict and in fact tend to make reserve reporting by oil companies conservative.

  52. Vinny Burgoo
    Posted Nov 26, 2009 at 4:29 PM | Permalink

    Phillip Stott and Tom Crowley were asked their opinions about the stolen e-mails on today’s _Material World_ on BBC Radio 4.

    Stott’s view: See his (recently revived) blog _The Clamour of The Times_.

    Crowley’s view: McIntyre is wasting climate scientists’ time because the data is already easily available and anyway they are too busy doing science to make it available.

    *

    Partial transcript

    Interviewer: [What's with the apparent FOI obstructionism by Jones?]

    Tom Crowley: Well, all of the data – most data are actually public-domain, and it’s – so people like Stephen McIntyre who request data can actually go to many public-domain websites where that data already exists, and all the work that we are involved in, for example, are all available – almost all of it is available on public websites. We don’t collect it ourselves. So, you know, it’s a moot point, I think, a little bit. He’s trying to get the data but he actually has the data from other sites already.

    Interviewer: [What about Phil Jones saying he would rather delete data than release it 'into those hands' under FOI legislation?]

    Tom Crowley: [Jones wouldn't do that. He was expressing exasperation.] You see, the person who has been involved in trying to get most of this data just does not stop with one request. If you send him something he’ll say, well what about this, what about this, why have you done it this way rather than that way. And he’s almost asking questions as if, you know, I have nothing better in the world to do than sit around responding to his e-mails. People are busy. They don’t want to be completely pestered by him.


    Steve:
    Crowley’s allegation is absurd. I know my way around the public data. I might add that, if the requested data were in the public domain, all Crowley or whoever would need to do is direct me to the public website.

  53. Posted Nov 26, 2009 at 4:43 PM | Permalink

    More from Delingpole

    http://blogs.telegraph.co.uk/news/jamesdelingpole/100018056/climategate-this-is-our-berlin-wall-moment/

  54. Calvin Ball
    Posted Nov 26, 2009 at 4:43 PM | Permalink

    What happened to the comments? They got flattened.

  55. checker22
    Posted Nov 26, 2009 at 5:10 PM | Permalink

    I don’t know if the eagle-eyed have already picked this up but here is Keith Briffa(938018124) explicitly stating that recent biological proxies “do not match the recent warming”. If they are not reliable proxies now, why would they ever have been?

    ” There is still a potential problem with non-linear responses in the
    >very recent period of some biological proxies ( or perhaps a fertilisation
    >through high CO2 or nitrate input) . I know there is pressure to present a
    >nice tidy story as regards ‘apparent unprecedented warming in a thousand
    >years or more in the proxy data’ but in reality the situation is not quite
    >so simple. We don’t have a lot of proxies that come right up to date and
    >those that do (at least a significant number of tree proxies ) some
    >unexpected changes in response that do not match the recent warming.”

  56. checker22
    Posted Nov 26, 2009 at 5:14 PM | Permalink

    Note also reference to “pressure to present a nice tidy story”. Another explicit recognition that they have drifted from science to self preservation.

  57. Jeff the thinker
    Posted Nov 26, 2009 at 5:43 PM | Permalink

    For adhering to the highest standards in science and inquiry it’s about time we gave Steve a real Nobel Prize in science.

  58. geo
    Posted Nov 26, 2009 at 6:16 PM | Permalink

    Mentioned this at WUWT as well. . .

    One should not miss that while Dr. Jones was doing his best Captain Queeg impersonation about organized campaigns to keep him from working via FOIA demands because CRU received “double digits” (one suspects he means less than 20) in two years, in the US the EPA (Environmental Protection Agency) received 11,492 FOIA requests *in one year*, FY2008. See here: http://www.justice.gov/oip/foiapost/2009foiapost16.htm

    Keep that context in mind when Phil Queeg starts rolling the balls around in his hand ranting about McIntyre and his minions plotting to bury him in FOIA work.

    My brother is a physicist for the US Army, and it being a family holiday in the US today I found myself talking to him about this issue. . . he offered that in his lab the scientists were all shaking their heads at what these guys were saying to each other in email, particularly re documenting their concerted schemes to avoid FOIA requests.

  59. Richard Halpern
    Posted Nov 26, 2009 at 8:18 PM | Permalink

    I have not had time to read this entire thread, so I may be duplicating someone else’s comment. Be that as it may, it seems to me that the obvious course of action now is to make a new FOI request for all Jones et al’s data. Surely such a request would have to be honoured now.

    If the data for the temperature record no longer exist or are full of the same garbage we’ve seen in the HarryReadMe file, then the whole edifice crumbles.

    In any event, it would be impossible for such a request to be denied after what has happened.

  60. James RS
    Posted Nov 26, 2009 at 9:24 PM | Permalink

    From Lucy Skywalker:

    “Click my name if you want a general primer in skeptical Climate Science – and my story”

    Thanks Lucy. Great site and I will explore it in depth.

  61. Graham
    Posted Nov 26, 2009 at 10:23 PM | Permalink

    quote: “I got no response from Phil Jones or anyone at CRU. So I filed a Freedom of Information act request”

    obviously not, you’re not a scientist working in the field and you had no right to pester him for the data.

    quote: “Now at this point, let me diverge to what was happening at CRU during this time. The first reference to Freedom of Information in their emails is from 2005, before they had received a single request.”

    – all of their discussions of the FOIA in 2005 stems from the fact that it came in force in 2005. Every scientist in the UK was talking about the FOIA and how they could protect their research from it. With that context in mind, most of the emails from that time are either reactions to current events and friendly personal banter. If they had known then that they would get flooded by 60+ requests by ‘amateur scientists’ they might not have treated the matter so lightly.

    Moving onto the rest of the replies to the FOIA request, if you know the history of the data being asked for, and consider the request is being made by basically a nobody, then frankly they are patient, generous and informative to a fault. They could have easily dismissed your entire claim on the basis that it would cost them more than £600 to process the data.

  62. Eyesapoppin'
    Posted Nov 27, 2009 at 12:19 AM | Permalink

    Graham’s obfuscating screed– for example, it makes absolutely no difference who asks for data, nor should any reputable scientist have to make any effort to extract and publish the dataset upon which his crucial experiments and claims reply– is exactly the kind of bullying smokescreen that “amateurs” no longer feel intimidated by. The professionals, who unlike us have had their careers in the balance, no longer need to feel so intimidated either, now that the bullies have been outed. You are going to hear a lot more from them now.

    “Foolkiller[s] coming, better make my getaway!” (with apologies to Mose Allison)

  63. Eyesapoppin'
    Posted Nov 27, 2009 at 12:22 AM | Permalink

    Lucy Skywalker: I think I’m in love.

  64. DABbio
    Posted Nov 27, 2009 at 1:03 AM | Permalink

    I guess I should have said in my reply to Graham that it should not take that much effort for a scientist to compile and publish the data–one time–upon which a crucial experiment rests. Anyway, Willis’s fundamental point is that tussles over the primary and transformed data are what science is partly and vitally about, so there really ought not to be resistance to requests to see the data, especially on the basis of ad hominem arguments.

  65. P Gosselin
    Posted Nov 27, 2009 at 1:14 AM | Permalink

    If there was ever a case of violating the FOIA, this is it.
    I’m particularly concerned by the overall psychology that permeated throughout the scientists surrounding and including Mann and Jones. I see a lot of delusions and paranoia.
    This is not uncommon for people who reach dictatorial stages. History has always shown this – I’ll spare the obvious examples. Clearly we see a science here that indeed became extremely insecure, and later defensive, secretive and dictatorial.

  66. P Gosselin
    Posted Nov 27, 2009 at 1:16 AM | Permalink

    It would be interesting to see what a psychiatrist would have to say about this behaviour.

  67. Willis Eschenbach
    Posted Nov 27, 2009 at 1:42 AM | Permalink

    Graham

    quote: “I got no response from Phil Jones or anyone at CRU. So I filed a Freedom of Information act request”

    obviously not, you’re not a scientist working in the field and you had no right to pester him for the data.

    quote: “Now at this point, let me diverge to what was happening at CRU during this time. The first reference to Freedom of Information in their emails is from 2005, before they had received a single request.”

    – all of their discussions of the FOIA in 2005 stems from the fact that it came in force in 2005. Every scientist in the UK was talking about the FOIA and how they could protect their research from it. With that context in mind, most of the emails from that time are either reactions to current events and friendly personal banter. If they had known then that they would get flooded by 60+ requests by ‘amateur scientists’ they might not have treated the matter so lightly.

    Moving onto the rest of the replies to the FOIA request, if you know the history of the data being asked for, and consider the request is being made by basically a nobody, then frankly they are patient, generous and informative to a fault. They could have easily dismissed your entire claim on the basis that it would cost them more than £600 to process the data.

    In fact, I am a scientist working in the field, a minor scientist to be sure, but that’s not the point. ANYONE can ask for public taxpayer funded data. The people who hold the data don’t get to decide what to give out, that’s the whole point of why it is called the Freedom of Information Act. To set information free that someone wants to hide.

    Next, you say they were “patient, generous, and informative to a fault”. Say what??? I asked for their data, and got nothing. On my planet I call that obstruction, not generosity. I asked for data, and failing that, a station list with information for each station which shows where that particular data is publicly available. I’m still waiting for both.

    Finally, the thing that both you and these “scientists” seem to be missing is that a real scientist not only has to reveal his data and methods so they can be scientifically verified, but also that he should give his data and methods to his worst enemy. Because if his worst enemy can’t punch holes in his theories, then he can bet that they are solid. That’s one reason I publish on the web, so people who disagree with me (and plenty do) can show me where I’m wrong.

    Finally, a scientist should never have to face an FOI as Jones and Mann and the rest have. Why not? Because making the data public is PART OF EVERY SCIENTIST’S JOB DESCRIPTION. I did not want to file an FOI, it’s a hassle for everyone concerned. But I was forced to file because Jones and the other scientists are afraid to make their data public, claiming that someone will want to poke holes in it. Poor babies …

    Graham, science is not some namby pamby game. It is a tough blood sport, where the participants are expected and required to throw their data and their methods and their claims and their reputation into the public arena, to subject them to the merciless glare of public view, and to watch as other ravenous scientists try to rip their ideas to shreds. It’s ugly, I know, because I play the game from both sides. But that’s how science works.

    Science stops working when some wimpy scientist goes “Oh, I don’t like what those bad boys and girls at Climate audit are up to, they harass people, I’m not going to show my work to them, they just want to find holes in it, I’ll keep it secret.”

    Damn right we want to find holes in it, that’s the whole point of science. The only way that science advances is by a scientist poking holes in someone’s favourite theory. Jones and Mann want to poke holes in other people’s work, but shield their own work from public examination. If you can’t stand it when it’s your turn, when it’s your theory that the bad scientist boys and girls want to find holes in, go become a novelist. Science doesn’t progress when scientists hide the codes and data.

  68. Scott Gibson
    Posted Nov 27, 2009 at 2:11 AM | Permalink

    Graham-

    “They could have easily dismissed your entire claim on the basis that it would cost them more than £600 to process the data.”

    The data was released. How much did that cost them?

  69. P Gosselin
    Posted Nov 27, 2009 at 3:44 AM | Permalink

    Yes, no joke, from the BBC!

  70. 3x2
    Posted Nov 27, 2009 at 5:17 AM | Permalink

    Graham,

    let’s try it another way.

    If your Doctor prescribed a drug for you as part of some treatment you might expect that the drug had been thoroughly tested. Independently of the manufacturer and irrespective of their various claims. You trust that there are strict procedures in place to protect you. You would be horrified to find otherwise.

    Yet here we are with trillions of dollars on the table and nobody outside the CRU knows where their numbers come from. They could be picking them out of a hat for all anybody outside knows. Unacceptable.

  71. Graham
    Posted Nov 27, 2009 at 7:28 AM | Permalink

    Quote: “ANYONE can ask for public taxpayer funded data. The people who hold the data don’t get to decide what to give out, that’s the whole point of why it is called the Freedom of Information Act. To set information free that someone wants to hide.”

    – Sure they can, but whether they are permitted to it is an entirely different matter. The FOIA in the UK is not a carte blanche to request anything you want. I can’t, for instance walk into my local council office and demand the personal details of all the residents in their database, nor could I demand confidential information related to the British Army’s strategic deployments. In fact, there are a huge number of limiting factors on the UK FOIA, including, but not limited to;
    -trade secrets
    -unpublished research
    -documents under a confidentiality or non-disclosure agreement
    -anything that would cost more than £600 to process
    Several of those, and more, apply here.

    quote: “Finally, the thing that both you and these “scientists” seem to be missing is that a real scientist not only has to reveal his data and methods so they can be scientifically verified”

    – As you well know, they have released enough data to the public that the result can be reproduced. You have no doubt not even done that.

    quote: “Finally, a scientist should never have to face an FOI as Jones and Mann and the rest have. Why not? Because making the data public is PART OF EVERY SCIENTIST’S JOB DESCRIPTION”

    – You’re repeating yourself and you were wrong the first time. It is not part of their job description to deal with request after request, from people who have no legitimate interest, for data that must be compiled from disparate sources and that has various degrees of confidentiality attached to it. It takes a considerable amount of work to compile the data and get the necessary permissions to send it, and there is not a University in the world that could deal with that workload over and over again.
    That’s not to say they’ve never sent it to anyone, it is available to academics who can justify the amount of work it takes.

    Quote: …

    – The rest of your post is just tired, repeated rhetoric based on faulty assumptions.

  72. Graham
    Posted Nov 27, 2009 at 9:21 AM | Permalink

    Here’s the real kicker though; it doesn’t matter which subset of station data you use, if you did it properly you’d get similar results with any significant subset. They say as much in the emails. The changes are on every continent.

    If you had taken the raw, publicly available data offered to you, (even if they weren’t the exact same subset used in the original because of confidentiality contracts) formatted the data to your own liking and then, after all that, came away with significantly different results to Mann et al, and if this was repeatable with different subsets and different methods, then you might have a case for questioning the original data and methods.

    The fact is though that between the lot of you, you have still not managed to perform even the most basic amount of actual science to validate your position, and until then, the only possible reasons you have for asking for the data amount to nothing but harassment.

  73. Nick Temple
    Posted Nov 27, 2009 at 9:37 AM | Permalink

    Has anybody ever found out which country(ies) the CRU crowd are referring too when they say “signed agreements with national meteorological services” limiting disclosure to third parties?

    I think that would be a great FOI request.

    Steve: they’ve said that they’ve lost the agreements and don’t know who they are with. Look at the FOIA category in the CA left frame

  74. Posted Nov 27, 2009 at 10:33 AM | Permalink

    The reference grid system used by CRU to obtain their surface temperature plot is indeed complex and arcane, but it’s also redundant. If you take the average derivative of temperature for all stations each year and integrate from year to year, you obtain obtain almost exactly the same surface temperature plot as CRU from the raw station data, which is indeed available from GHCN by ftp. I show the result in the Home Analysis section here:

    http://www.hashemifamily.com/Kevan/Climate/

    It takes only ten lines of code and less than a minute to generate the graph, which makes it easy to sift the data and see how sensitive the graph is to urban warming, and so on, which I have attempted to do at the above site.

    So, there’s not much to be gained in obtaining the details of their reference grid calculation, because it’s redundant. I guess we could wonder why, in 25 years, neither NCDC and CRU never figured out that the reference grid is redundant, but then again, perhaps I’m missing a subtle detail.

  75. Larry
    Posted Nov 27, 2009 at 11:57 AM | Permalink

    Willis
    In the document “CRU strategic review agenda 1.doc” from the UEA collection we have:
    “The mission of the Climatic Research Unit is:
    • to undertake pioneering research on the nature, predictability and impact of natural and anthropogenic climate change, maintaining a position as a world authority in this field;
    • to act as a primary source of information, data, analysis tools and training on climate-related issues; and
    • to support sustainable responses to the challenges and opportunities created by climatic variability and change, through the provision of expert assessment and advice to all sectors of society.”
    I’m looking for the subsequent revisions to these ideals.

  76. Willis Eschenbach
    Posted Nov 27, 2009 at 12:55 PM | Permalink

    Graham

    Here’s the real kicker though; it doesn’t matter which subset of station data you use, if you did it properly you’d get similar results with any significant subset. They say as much in the emails. The changes are on every continent.

    Umm … not true. I’ve just prepared a post on this question and sent it to Steve. This story is a long ways from over.

    If you had taken the raw, publicly available data offered to you, (even if they weren’t the exact same subset used in the original because of confidentiality contracts) formatted the data to your own liking and then, after all that, came away with significantly different results to Mann et al, and if this was repeatable with different subsets and different methods, then you might have a case for questioning the original data and methods.

    What part of “they have not yet released the data” do you not understand? I was quite happy to accept the subset. Read my interactions again. I said

    Therefore, since the information requested is not available on non-UEA websites, I wish to re-instate my original request, that the information itself be made available on your website or in some other form. I understand that a small amount of this data (about 2%, according to your letter) is not available due to privacy requests from the countries involved. In that case, a listing of which stations this applies to will suffice.

    So I re-asked, saying I’d gladly accept the slightly truncated data, and got … nothing.

    The fact is though that between the lot of you, you have still not managed to perform even the most basic amount of actual science to validate your position, and until then, the only possible reasons you have for asking for the data amount to nothing but harassment.

    Until we have the data, we can’t do any “actual science” at all … no data, no science. Think this all the way through, please. Your accusations are untrue.

  77. Willis Eschenbach
    Posted Nov 27, 2009 at 1:04 PM | Permalink

    Graham

    Quote: “ANYONE can ask for public taxpayer funded data. The people who hold the data don’t get to decide what to give out, that’s the whole point of why it is called the Freedom of Information Act. To set information free that someone wants to hide.”

    – Sure they can, but whether they are permitted to it is an entirely different matter. The FOIA in the UK is not a carte blanche to request anything you want. I can’t, for instance walk into my local council office and demand the personal details of all the residents in their database, nor could I demand confidential information related to the British Army’s strategic deployments. In fact, there are a huge number of limiting factors on the UK FOIA, including, but not limited to;
    -trade secrets
    -unpublished research
    -documents under a confidentiality or non-disclosure agreement
    -anything that would cost more than £600 to process
    Several of those, and more, apply here.

    I acknowledged that the confidentiality agreements applied to about 2% of the data, so that was a non-issue. They never claimed under the £600 exemption. There are no trade secrets involved. This was not for unpublished research. None of your exemptions apply.

    quote: “Finally, the thing that both you and these “scientists” seem to be missing is that a real scientist not only has to reveal his data and methods so they can be scientifically verified”

    – As you well know, they have released enough data to the public that the result can be reproduced. You have no doubt not even done that.

    No. No. THEY HAVE NOT RELEASED THE DATA. If you want to prove this, give us a URL. We’re looking for the raw data, not their finished products. If you know where it is, bring it on. If not, please stop repeating this false claim

    Also, I haven’t a clue what you mean when you say I have not released my data …?

    quote: “Finally, a scientist should never have to face an FOI as Jones and Mann and the rest have. Why not? Because making the data public is PART OF EVERY SCIENTIST’S JOB DESCRIPTION”

    – You’re repeating yourself and you were wrong the first time. It is not part of their job description to deal with request after request, from people who have no legitimate interest, for data that must be compiled from disparate sources and that has various degrees of confidentiality attached to it. It takes a considerable amount of work to compile the data and get the necessary permissions to send it, and there is not a University in the world that could deal with that workload over and over again.
    That’s not to say they’ve never sent it to anyone, it is available to academics who can justify the amount of work it takes.

    Request after request? What are you smoking? I made the FIRST REQUEST to CRU. There were no other request. And I do have a “legitimate interest” in the data, because fools want to spend billions of dollars of taxpayer money based on the claims CRU made using the data.

    Finally, “work to compile the data”? Dude, it’s just data. I’m not asking them to “compile” anything. Zip up the file and post it on the web, how tough can it be??

  78. history student
    Posted Nov 27, 2009 at 2:02 PM | Permalink

    Re: bender on black soot and Arctic ice:
    As I recall, Pielke Sr. thinks this is not given enough attention. Ask yourself:
    which candidate (warming or soot) would be more likely to reduce ice in summer, while leaving winter ice less affected?

  79. P Gosselin
    Posted Nov 27, 2009 at 2:40 PM | Permalink

    I think it’s simple, either you provide the data you used for the paper, or the paper gets flushed. There’s nothing more to it. Teacher always told us: show your work!

  80. KimW
    Posted Nov 27, 2009 at 10:05 PM | Permalink

    Again, the belief that it is too much trouble to release information. When I handed in my MSc thesis, all my data was included with it. In the same manner, if you publish, all the data should be all lined up ready to go, not spread in little packets across the files. You want a copy of a paper in Science, with the data ?, push a button or two and send it – unless the author is a shoddy organiser.

  81. Micky C
    Posted Nov 29, 2009 at 7:31 AM | Permalink

    Having worked in the space industry for a while I can sympathise with some of the reactions from the scientists. In particular Ben Santer’s frustration at having to constantly answer questions about his research; which, by the tone, he feels is explained well enough anyway. I have had on numerous occasions to answer seemingly petty questions about a test or analysis report asked by someone on the customers team. It can be never ending and it can breed reactionary (RTFM) behaviour quite quickly.
    However a lot of the time, when you realise that your report may seem to be completely open and has all the relevant data, it isn’t so the question is actually valid. Other times its because of an unrelated communication that has somehow become tangled into the question, or this is perceived as such.
    The way to deal with it is the same: take a breath and try and see it from their point of view. If all the methods, assumptions and data are there and the results are clear, you are probably talking to the monkey and you need the organ grinder. Say as much i.e. ‘if you don’t understand it that’s okay, just find me someone in your company who does understand it and have them explain it to you. Otherwise you are unnecessarily delaying the program’.
    If you haven’t shown all your methods or described them in a way that is repeatable then questions are fair. As are the requests to see more detail. Because, for example, in the space industry, after all, its more about doing it right rather than egos. You only get one shot at a launch.

    Steve: In fact, there are real issues with Santer’s paper. He only used data up to 1999! Using data up to 2008 (or 2007 in some cases) reverses key results. Ross and I submitted a comment on this which has been rejected twice. In my opinion, the referee comments have been intended primarily at tying up/keeping this result out of the PeerReviewedLitchurchur than in improving the quality of the paper. It appears that Phil Jones may have been involved in the selection of referees.

  82. Micky C
    Posted Nov 29, 2009 at 3:15 PM | Permalink

    Yes Steve I remember the posts you did on that. I’m not saying I agree with Santer ( I think they use autocorrelation so much it ends up inflating the confidence intervals but yet don’t clearly say that this is a severe limitation on their findings). I’m saying that some of the reactions are familiar. The difference though is that I know that often it is not the fault of the person asking for the request; it tends to be overreaction by the person being asked. I’ve been guilty of that a lot in my career and being calm is the way to go. Especially if the stakes are high.

  83. Posted Nov 29, 2009 at 6:03 PM | Permalink

    Just curious about the letter quoted in the post for May 21st 2007 and headed:

    “Decision of Information Commissoners’ Office
    FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)”

    Given that this appears to be a reply from someone at the UEA as part of their internal appeal sytem, I wonder about the legality of heading it in a way that clearly purports to be from the ICO and, as such, not subject to further appeal.

    Regardless of whether or not they’ve included the right of appeal to the ICO at the bottom, using the ICO’s title (as a recognised public authority) in the heading seems a clear attempt to subvert that right.

  84. Willis Eschenbach
    Posted Dec 1, 2009 at 2:15 AM | Permalink

    Joe Horner

    Just curious about the letter quoted in the post for May 21st 2007 and headed:

    “Decision of Information Commissoners’ Office
    FREEDOM OF INFORMATION ACT 2000 – INFORMATION REQUEST (FOI_07-04)”

    Given that this appears to be a reply from someone at the UEA as part of their internal appeal sytem, I wonder about the legality of heading it in a way that clearly purports to be from the ICO and, as such, not subject to further appeal.

    Regardless of whether or not they’ve included the right of appeal to the ICO at the bottom, using the ICO’s title (as a recognised public authority) in the heading seems a clear attempt to subvert that right.

    Don’t know about the legalities, but when I got the letters, I assumed that they were University employees whose job description included dealing with FOI requests. Their job titles said they worked for the University, not the FOI.

    w.

    • Posted Jan 4, 2010 at 11:28 AM | Permalink

      Finally got a reply from the ICO about this. Unfortunately, I’ve lost my original queriy (maybe I could get a job in research? :P ) but the gist of it was whether heading an internal review reply as if it had come from an external body was allowed.

      The ICO’s position is that:

      A public authority conducting an internal review of a request for information should make it clear that the response has come from itself and not from any other body. Advice on the conduct of internal reviews is contained in a Code of Practice from the Secretary of State for Constitutional Affairs and guidance that we have published on our website.

      So, the question is whether heading a reply as from the ICO but including the reviewer’s job position amounts to “making it clear” that it was conducted internally. My gut feeling is that they’re (probably) complying with the letter of the law but certainly not the spirit by using such a format.

      It’s hard to see any reason for using “Decision of Information Commissioners’ Office” unless they’re hoping to present an impression that the review has been done by the ICO. It’s even harder to see how presenting such an impression could be accidental. Which does suggest an institutional dislike of playing the FOI game by the rules.

  85. Chants
    Posted Dec 3, 2009 at 5:24 AM | Permalink

    Thank-you ever so much for taking the time to synthesize these e-mails. This is a must read primer for anyone looking for a contextual understanding of Climategate.

  86. Kevin Marley
    Posted Dec 3, 2009 at 1:36 PM | Permalink

    A point missing in all this is that if Jones, Mann et al. had engaged in a legitimate peer review process the need for FOI/FOIA requests would have been minimized. I’ve seen no evidence in the emails that anything was genuinely peer reviewed. Jones, Mann et al. counted on no one trying to actually reproduce their results and the only data/process (code) sharing that took place was amongst those on the “Team”… those who agreed with the results, NOT THE SCIENCE.

    It would be one thing if the science was privately funded, corporately developed for corporate profit. But we are talking about research that was publicly funded, and specifically developed for international policy-making that affects the lives of the entire globe. By hiding the data/process, Jones and Mann became the moral equivalent of scientists working for a totalitarian regime.

  87. Posted Dec 5, 2009 at 9:07 AM | Permalink

    Thanks for that, Willis. I’m unsure of the legality as well but am enquiring gently to try and find out.

    After all, the fact that you read it for what it is doesn’t negate the suggestion that they may have tried to subvert the FOI process by giving an (even cursory) impression that the process was complete.

    If that’s the case then it would lend weight to the suggested context of some of the leaked emails by placing them in a wider institutional context of blocking the FOI process.

  88. LMB
    Posted Dec 12, 2009 at 6:02 PM | Permalink

    Can anyone summarize the FOIA laws related to the requests by various parties seeking climate records? Were they in the UK, Penn, and/or somewhere else? Were the PA ones by federal FOIA or state FOIA?

    I’d like to know if the process of getting information was delayed because nobody filed a FOIA lawsuit. Could the leaked Climategate emails have come out 5-10 years ago if a lawsuit had been filed then?

    I’m also interested to know who all is filing lawsuits now that there is a basis to argue FOIA laws were broken based on leaked emails (i.e: statements of intention to destroy records in violation of FOIA law).

    I haven’t read enough in the news yet to be convinced anti-alarmist climate activists are being aggressive enough to get important new records when organizations are hinting they will do damage control and whitewash internal “investigations.”

    Judicial Watch is hyper activist, filing new FOIAs every week, but I see nothing by them on this scandal. Right now they are piddling around with White House visitor logs according to home page of http://www.judicialwatch.org/

    Somebody should contact JW and get them in the loop.

    Otherwise, and correct me if I’m wrong, but I don’t think the plaintiff in a FOIA lawsuit actually needs to be present where the lawsuit is filed, or can get an attorney to be present (e.g. if you’re in Canada and want a FOIA lawsuit filed in the US).

    Non-liberal reporters should now be aggressively hammering at the highly suspicious institutions to get at new records by FOIA request that could blow the case wide(r) open.

    If this was a “Republican” scam, you can be the AP, NYT, WP, and everybody else would be FOIAing every target they could think of.

    Finally, IIRC, I read that under one law (UK?), it’s actually a crime to break the FOIA law. If that is true, who does the criminal investigation, and before that, who files the complaint which initiates the criminal investigation?

  89. LMB
    Posted Dec 12, 2009 at 6:11 PM | Permalink

    P.S. Just spotted the blog: Judicial Watch has done an FOIA on climate issues and are following Climategate:

    http://www.judicialwatch.org/foiablog/2009/dec/climategate-and-geoengineering

    Excerpt:

    “… individuals within NASA find it “challenging to provide a convincing case that spending zillions of dollars in geoengineering will have a specific positive impact.”

    Despite little action on geoengineering, the strategy is circulating around the federal government. NASA’s emails note that “DOE, because of their CCTP [Climate Change Technology Program] focus, have been thinking grand thoughts in this area” even though DOE claims to have no records on the subject.

    “Geoengineering is in its preliminary stage, with the government currently assessing the strategy from a financial and managerial view. The financial dimension is certainly significant, but the larger concerns should be the potentially irreversible ramifications as well as the actual necessity. Just as the United States intended to hit the reset button with Russia on diplomatic relations, the climategate scandal necessitates the need to pause and reset our notion of climate change.

    “Dissenting views in climate science need to stop being suppressed so that the data showing cyclical variations over hundreds of years can be evaluated alongside the “crisis” data of just the past 60 years. In addition, agencies need to be responsible for producing accurate data, and geoengineering ideas of space mirrors need to remain science fiction at least for now. Judicial Watch will continue to investigate geoengineering and climategate.”

  90. sHx
    Posted Dec 17, 2009 at 9:57 AM | Permalink

    Willis,

    Just read this post. Mate you are writing history here.

    Believe me if I say this single post provides much more context to the FOI chatter in the CRU emails than three weeks of ‘context’ offered by Gavin Schmidt et al of the Real Climate blog.

    Mate, thanks a million.

  91. SteveGinIL
    Posted Jan 14, 2010 at 4:15 PM | Permalink

    It is difficult to understand any un-suspicious reasons these people would have for not sharing their exact data and their specific methodology. If they did their homework, they should have ZERO fears of more than (as Schneider puts it) anyone finding “minor glitches or unexplained bits of code–which exist in nearly all our kinds of complex work. They should be proud of their work and DARE everyone else to tear down their construct. Their doing the opposite does not invite confidence.

    Everyone here has a sense of what kind of shape the data is probably in. Any scientist worth his salt would have organized it all into clearly defined databases, each one for use with a particular method of adjustment. Any scientist worth his salt would have given over his databases – in their entirety – and every one of their methods at the first request, FOI/FOIA or not. Scientists sure of their methods would welcome replication efforts.

    The only reason ANY of this is an issue is not because Steve M asked for the data and methods, but because they refused to give it all over as any normal scientists would have done – and have done, since forever. That the peer reviewed journals originally gave them a pass and have continued to not demand the data and methodology shines a sinister light on the entire affair.

    Steve: snip – sorry about that. EDitorially, you’ll find that you get a better response with shorter points, where you don’t try to solve all the problems of the world all at once.

  92. Roy_US_Ohio
    Posted Feb 9, 2010 at 7:58 PM | Permalink

    My IT development work is in private industry. I agree that there may be some level of embarrassment at the state of their programming and data. What I find surprising is:

    1) The presumption that they “own” their code. They should check their employment agreements. UEA probably owns their code.
    2) Documentation of their code is an onerous task because of its complexity. It is precisely the more complex code that we are the most diligent about documenting.
    3) Data and methods are some sort of personal secret which must be defended; finding minor flaws is an irritant. Rubbish. Our first priority is “bullet-proofing” our code.
    4) They’ve lost track of their NDAs. If that is true, then they are legally exposed every time they publish.

11 Trackbacks

  1. [...] Ni bara måste läsa hela denna konversationen, McIntyre lägger upp några FOI förfrågningar som han och andra har gjort och sedan svaren dem fick. Och nu med hela climategate affären så kan vi nu läsa lite av diskussionen bakom kulisserna. Ni måste läsa hela inlägget, det är långt långt men värt. Och om dessa personer arbetar helt ärligt enligt alla gällande krav om att kunna replikera resultatet så är jag Rudolf. Även om all data är korrekt och alla deras Nostradamus teorier är sanna så är inte detta beteende ärligt eller etiskt. Läs det nu! Good luck with this, and expect more of it as we get closer to international climate policy actions, We are witnessing the “contrarian battle of the bulge” now, and expect that all weapons will be used. [...]

  2. [...] Publicado por JoaoMiranda em 26 Novembro, 2009 Willis Eschenbach’s FOI Request [...]

  3. [...] Comment; K Rudd and Turn-bull slither into action as appallingly they try by any means possible to bring in the ETS. Yet voter anger has overwhelmed politicians – it seems that the long perpetrated lie that the populace support ETS is now being shown as completely wrong. In what will be an interesting day in Australia the world might just start ending here – that is the dying world of political climate alarmism. Though it seems like in both politics and science what needs to be revived is TRANSPARENCY. [...]

  4. [...] Ambiente, Blogosfera, Educação, Media, Política, Teoria — André Azevedo Alves @ 00:47 Willis Eschenbach’s FOI Request People seem to be missing the real issue in the CRU emails. Gavin over at realclimate keeps [...]

  5. By Climategate 2 « Jim’s Blog on Nov 27, 2009 at 11:31 PM

    [...] The climategate letters, programs, and datafiles show a systematic and repetitious pattern of hiding and falsifying inconvenient data, cherry picking, self deception, and replacing peer review with theological review of the holy synod.  It is worth examining particular incidents from this sorry story in detail. [...]

  6. [...] Eschenbach’s Freedom of Information Act request is well documented at Steve McIntyre’s Climate Audit mirror site.  It is truly an informing [...]

  7. [...] explanation of what he considered obfuscation and thwarting of the request is chronicled in this ClimateAudit post. The hacked emails contain disparaging comments regarding his request, and authorities in Britain [...]

  8. By Climategate | Hoystory on Dec 2, 2009 at 12:58 AM

    [...] the most telling example of scientific misconduct is the curtain that has been pulled back on Willis Eschenbach’s FOI request to the University of East Anglia Climate Research Unit that was repeatedly denied. While Eschenbach [...]

  9. [...] Why did Phil Jones et al hide and delete data critical to reproducing published claims? Will Eschenbach’s FOI Request. [...]

  10. [...] individuals to request information related to government-funded projects. The peer-to-peer network requested the data directly from the University of East Anglia’s Climate Research Unit and McIntyre even approached [...]

  11. [...] circle of committed "climate scientists" to requests for data, take a look at Willis Eschenberg's account at Climate [...]

Follow

Get every new post delivered to your Inbox.

Join 3,299 other followers

%d bloggers like this: