A couple of week ago, I noticed that the UK Met Office website contained the following statement
Q. Where can I get the raw observations?
A. The raw sea-surface temperature observations used to create HadSST2 are taken from ICOADS (International Comprehensive Ocean Atmosphere Data Set). These can be found at icoads.noaa.gov/. To obtain the archive of raw land surface temperature observations used to create CRUTEM3, you will need to contact Phil Jones at the Climate Research Unit at the University of East Anglia. Recently archived station reports used to update CRUTEM3 and HadCRUT3 are available from the CRUTEM3 data download page.
On May 11, 2009, I sent an inquiry to John Kennedy maintainer of this webpage asking him to “provide me the archive of raw land surface temperature observations as you hold them at the Hadley Center.”
Kennedy replied the same day:
Most of the station data was given to us by Phil Jones under conditions that don’t allow us to redistribute it. If you want the full archive, you will have to contact him.
I sent the following FOI request to the UK Met Office:
I request the “archive of raw land surface temperature observations used to create CRUTEM3” as held by the Hadley Center (referred to on your webpage http://hadobs.metoffice.com/indicators/index.html ) under the FOI Act or other applicable legislation.
In my request, I did not include any of the flowery salutations that some readers have recommended. I did not, for example, say “Thank you, John” or “thanking you in advance, Mr Hadley”.
I received the following response today:
Re: Environmental Information Regulations request
Your email dated 11 May 2009 has been considered to be a request for information in
accordance with the Environmental Information Regulations Act 2004.You requested the archive of raw land surface temperature observations used to create
CRUTEM3, as held by the Hadley Centre (referred to on webpage
http://hadobs.metoffice.com/indcators/index.htmlThe Met Office does not hold this information. The information we hold is the value added
data, which is data that has been quality controlled and where deemed appropriate, adjusted
to account for apparent non-climatic influences.I hope this answers your enquiry.
Kennedy had previously said that “most of the station data was given to us by Phil Jones under conditions that don’t allow us to redistribute it” indicating that they held it. The two answers seem a bit inconsistent, though I guess there are ways to split this hair. However, I’m puzzled as to exactly how they split this hair. They admit to holding some information and seem to have left an opening for a further FOI request, which I’ll pursue.
Oh, BTW, don’t think that my request had no impact whatever. If you go to the Hadley Center webpage linked in my previous request – the one where they supposedly answer the question:
Q. Where can I get the raw observations?
that part of the webpage has been deleted. 🙂 It now reads:
Q. Where can I get the raw SST observations?
A. The raw sea-surface temperature observations used to create HadSST2 are taken from ICOADS (International Comprehensive Ocean Atmosphere Data Set). These can be found at http://icoads.noaa.gov/.
Climate Audit 
126 Comments
Isn’t this publicly-funded data? How can Phil Jones hold in a dungeon somewhere?
That should be “hold it in a dungeon..”
A brilliant example of the scientific method in action, climate style.
You should make a FOI request for details of the conditions imposed by Phil Jones.
It wouldn’t surprise me if there were no conditions written down, just a verbal “Don’t give this to
Steveanybody”Would it do you any good to request the exact procedures used to add the value to the raw data? Admittedly you’d have to work backwards, but that’s been done before around here anyway.
This is quite outrageous, and must be pursued with the greatest vigour.
Ask him for his “value added data” and for the “adjustments” he’s made to the raw data to achieve it. Also, ask him for a copy of his communication to Phil Jones where he agreed to the conditions.
This may be an ignorant question but why not ask Phil Jones for the data? Because it is the value add procedures that you are after?
thanks
Re: Eric (#8),
This has a long history and is well documented on this site. Phil Jones has been asked and refused. FOI requests to CRU have failed.
Re: Steve McIntyre (#10),
Just in case anybody has missed it, there is a little bit of a scandal in the UK at the moment regarding MP’s expenses. This was initially brought about by a FOI so MP’s might be especially sensitive and helpful to constituents who complain to them about somebody else stonewalling a FOI request.
PRICELESS !!!
What could they be afraid of??
I believe part of the problem is there is no “the data”. Every time they update, the data have changed because some stations have not been updated and others that were not up-to-date last time have been updated, and other stations have been adjusted or corrected. There is also no “the list of stations used” for the same reason. It changes constantly.
Steve, maybe they can stonewall you because you are not a British Citizen. Would you like me to follow up your F.O.I. request with others?
Re: Don Keiller (#12),
If you don’t get anywhere then get the press involved. Particularly the likes of Christopher Brooker. The Met Office, CRU etc are all funded by the UK taxpayer.
That actually seems like a reasonable response, and an invitation for further FOI.
They have the post-Jones data only. Jonesification will remain a black-box at present, but the complete output of his process may allow insights as to what his process is.
I have had an exposure to some aspects of litigation so I would ask any lawyers who read this blog to answer. Would a judge accept such quibbling? My experience would be that any litigant who tried to get away with such quibbles would be taught a very quick lesson by an angry judge.
Re: TAG (#14),
you’re 100% right about that.
So the initial information that the data was available has softly and silently vanished away? What a shame that you weren’t snarky in your request because….
I could then have said that your snark was a boojum.
JF
I’ll get me coat.
And the section were they instructed people how to ask for raw observations has been deleted?, amazing, I have no words, at least polite words
Re: rafa (#16),
I looked in the Wayback Machine for a cache but there isn’t one. The page presently reads:
I don’t know whether this is a further change today or whether I missed it – I searched on “raw observations” and “jones” – and might have missed this wording. In any event, the present version is even more cynical.
The MPs should have hired Phil Jones to keep track of their expenses.
Re: Steve McIntyre (#18),
Google has a cached May 12, 2009 version here
FWIW:
http://www.ico.gov.uk/
http://www.ico.gov.uk/Home/what_we_cover/environmental_information_regulation.aspx
If we at least had the adjusted station data, wouldn’t that tell us which stations were used during what period, and in most cases allow us to find the corresponding raw data from other sources, and then at least to see how big the adjustments are, if not how or why they were done?
The station list that Willis Eschenback successfully FOI’d out of Jones a while back (“CRU Reveals Station Identities”, Oct. 1, 2007) contained a lot of airports that presumably don’t really go back to 1850 or even 1900. It perhaps was the list of then-current stations, not the entire history of stations. The list includes Columbus, OH, for example, with coordinates at the airport, which is where the official Columbus station has been since 1948. But before that, “Columbus” was on a series of rootops near the center of the city, which is probably what he used. Having the adjusted data would allow this to be determined.
The paradoxical thing about “value added data” is that it always proves that things are “worse than we thought”.
Re: “The Met Office does not hold this information. The information we hold is the value added data, which is data that has been quality controlled and where deemed appropriate, adjusted to account for apparent non-climatic influences.”
To my mind this seems to cover it:
“The regulations apply to most public authorities, but they can also apply to any organisation or person carrying out a public administration function, and any organisation or person under the control of a public authority who has environmental responsibilities. This can include some private companies or public private partnerships, for example companies involved in energy, water, waste and transport.”
From:
http://www.ico.gov.uk/what_we_cover/environmental_information_regulation/the_basics.aspx
Om a quick read I don’t see anything in these that would disagree:
Re: exceptions:
Click to access introduction_to_eir_exceptions.pdf
Re: commercial interests:
Click to access awareness_guidance_5_v3_07_03_08.pdf
More informed opinion sought please!
Google cache
http://209.85.229.132/search?q=cache:ZpWtnFWcZOIJ:hadobs.metoffice.com/indicators/index.html+Air+temperatures+over+the+land+are+measured+at+around+2200+land+stations+every+month&cd=5&hl=en&ct=clnk&gl=uk&client=firefox-a
Steve, three weeks ago, when you posted the original link, I wrote to Phil Jones as follows:
I’m sure that you all will be shocked and surprised when I reveal that Dr. Jones has not answered …
w.
Re: Willis Eschenbach (#25),
We only need someone to write in and tell us you were not polite enough.
Re: Gerald Machnee (#27),
The request was extremely snarky. Willis said:
Jones recognized that comment as the ultimate snark. No wonder he refused.
Re: Steve McIntyre (#28), And of course that was pretty snarky.
Re: Willis Eschenbach (#25), Willis, maybe the thing to do is contact the University Chancellor, Sir Brandon Gough, and ask him how one may obtain the data.
One might also ask him whether the University has has a commitment to actual dissemination of knowledge, and policies on data transparency, and whether a record of prejudicial withholding of data is consistent with the Principles and Mission of the University.
Email of Sir Brandon Gough: c.kreetzer@uea.ac.uk (probably his office secretary)
The website of the Chancellory: http://www.uea.ac.uk/vco/Chancellor
Re: Pat Frank (#33), thanks for the suggestion. I have sent the following email to Sir Brandon Gough:
—————————–
Dear Sir Gough:
On the UK Met Office Website it says (regarding the CRUTEM3 temperature dataset):
I recently (three weeks ago) sent the following email to Dr. Jones, viz:
I have received no reply.
This is by no means the first time that Dr. Jones has refused to reveal this publicly funded data. When he was asked a few years ago by Warwick Hughes for a copy of the same data, Dr. Jones famously replied;
The answer, of course, is that he should provide the data because that is how science progresses. Unless Dr. Jones’ work can be replicated, and any possible errors identified, it is not science. It is merely anecdote. Impressive anecdote, with the name of your University unfortunately attached to it … but anecdote nonetheless.
Under the assumption that you and your University are interested in science rather than anecdote, perhaps you could have a quiet word with Dr. Jones and explain to him how science works. Here’s the simple version:
.
a) One scientist makes a claim.
.
b) Other scientists try to find something wrong with his claim. If they cannot find any errors, then his work is accepted.
.
Dr. Jones doesn’t want his work brought into the light of day because he is afraid someone will “try and find something wrong with it”??!?? Doesn’t he understand that the finding of errors is a central part of science? If Dr. Jones is not willing to allow other people to examine his work … then what is he doing claiming to be a scientist?
And if your University allows this kind of rubbish to happen in the guise of science … then what is it doing claiming to be a University?
The days when a scientist could get away with this kind of manipulation are long gone. The eyes of the internet are on all scientists, all universities, and particularly all those who falsely claim to be either scientists or universities. Dr. Jones is doing both his reputation, as well as the reputation of your University, immense harm through his refusal to follow the established scientific norms and practices. Due to his claim that science is best served by his hiding his data, his name is a standard joke in many circles … and the name of your University has not escaped laughter in that same regard.
Many thanks for your assistance in this matter. It is an important issue, which which cuts to the heart of whether your University truly wants to be a scientific institution … or not. The choice is yours to make.
Best regards,
Willis Eschenbach
Solomon Islands
—————————–
We’ll see what that stirs up …
w.
Re: Willis Eschenbach (#35), Willis, did you _really_ send that letter? My god, if Sir Gough loses his hair it’ll be your fault. 🙂 Good job. This is sure to be fun, no matter what.
Re: Willis Eschenbach (#35),
As a point of note, not replying to an email does not constitute a refusal. It’s quite possible he never received the email (might have gotten eaten by spam filters, deleted by accident, etc). You’re attributing a motive based on his actions towards others, which may or may not be true.
Just because Jones told them not to release it, does that mean they must obey him? It is from one gov’t dept to another! It isn’t private property nor is there individual information nor commercial data in it.
Steve, you should be more thorough in your search you didn’t even look in the expanded FAQ sections
LFAQ section – Less Frequently Asked Questions.
Or perhaps you could try ATAQ – Allowed To As Questions.
Try the secret section OCCAQ – Only climatologists can ask questions.
There is the possibiltiy that it’s been refiled under ABSMCAQ -anybody but Steve McIntyre can ask questions.
—
Dear Steve,
Due to the extemely sensitive nature of the requested information, certain procedures must be followed.
File form ES109F5 with the minister of climate in triplicate, forward one copy of your completion of certified climatology audit training form to the minister of factual representation – don’t forget your federal citizen ID number and last 3 tax filings and we’ll get back to you.
Truly faithfully yours,
Michael Moore,
Minister of Logic
Whilst the circularity and evasiveness of the Met Office is mildly entertaining, I am of the view that classification of such data as “belonging to National Security” is not very far away.
Once policy is set in motion (very close now), hard data pertaining to it is always classified. National Security exceptions to FOI are self-evident. Security of energy supply, patents relating to “green” technology … the list will become endless – and actual data sets are part of this.
Re: ianl (#30), That would go right along with arresting people for taking pictures of trains…oh, wait, they do that in England…too late.
Re: Craig Loehle (#31),
Well, yes … I was following the trend here in Aus, where a long and growing list of websites is banned through the simple procedure of prosecuting any ISP that allows its’ users to connect.
It is claimed as mostly for child abuse sites, but the actual list is a secret, of course. We have no way of knowing which sites are being added (apart from one prosecutable data leak – which has happened :)). Apparently, all you see is a message stating ” xxx page is unavailable”. So far, none of the links posted on CA here have shown this. I suppose I will post it if (when) it occurs.
If one attempts to purchase a CD of Govt-compiled topographical data, the degree of personal information required to be divulged is sufficient for a thorough, prior “security check”. And this is for simple topos (hard copies are available in most newsagencies, anyway) !! Try hard, reliable, gridded temperature measurements against a backdrop of anti-carbon policy.
You know, if the data access is on “double secret probation”, and likely to get worse, maybe the best way to attack the problem is to move forward with other data.
For example. There are private networks of weather stations already in place, such as is being demonstrated on Weather Underground.
Perhaps with a bit of education on siting, we can create a citizen database of climate data, based on private stations that are setup to meet standards. Certainly there seems to be a worldwide interest now in weather stations.
Then once a month, the citizen network publishes their temperature data. My educated guess is that we’ll see some divergence from CRU (and GISS), and at some point people like Jones will have to explain why. Demands for data would then be harder to resist.
This is actually a doable project. Education, signup, clearance of stations, an automated upload to web database, and then some collation and calculation.
I think it could run circles around any existing government station network, and we’d be almost guaranteed not to have them be a majority of airports. It could be called PubCRUT for Public Citizen RUn Temperature.
Re: Anthony Watts (#36), have you created a US-only average from the stations deemed “best” (CRNS Class 1), the same for Class 2, and so on, and see what effect those different classifications have had on the US average temperature? Perhaps the climate change signal isn’t really there – and using only the Class 1 stations would prove it.
Re: Gary Strand (#41),
Yes we have, working on more detailed analysis too, but not ready to publish yet.
Re: Anthony Watts (#36), one other thing – have you heard of CoCoRAHS? Not weather stations, just precip.
Re: Anthony Watts (#36),
(Chuckles) Some thirty odd years ago up to recent years,suggestions that volunteers, including former military and civil meteorologists, would be useful in reporting weather, especially special observations of severe weather, would be tremedously in filling in the gaps between official observation stations was met with considerable derision, contempt, and scorn. Today, NOAA is falling all over itself to recruit volunteers to the COOP and other programs. Some former COOP observers are disillusioned and no longer as enthusiastic about the government’s increased involvement and interest.
Anthony, are you acquainted with the Community Collaborative Rain, Hail, and Snow Network (CoCoRaHS) sponsored in part by NOAA?
Re: D. Patterson (#44),
Yes I know about CoCoRaHS.
It’s mainly for precip forecast verification.
Re: Anthony Watts (#46),
Yes, it’s a hydrometeorological network with more than 12,000 observation sites. Imagine if a significant fraction of those sites were equipped for the observation of additional conditions. Equipped with wireless instrumentation, WIFI, Webcams, and Web access, they or another network like them might look something akin to Weather World ‘Round.
http://www.davisnet.com/weather/cool/world.asp
Re: D. Patterson (#47),
Yes on on my third try selecting from the list they offer, I found a siting violation from people who should know better:
http://weather.glendale.edu/
At least it wasn’t in the parking lot like UofA.
This is why siting would have to be drilled in to the observers, and stations would have to pass a basic siting exam before inclusion.
Re: Anthony Watts (#36), Through Mathematica one can access most of the world’s weather stations with up to the hour currency. I would love to see Hadley circumvented.
Re: Craig Loehle (#54),
Yeah Lubos has done that on more than one occasion. I agree, I think we are at the point where we can start bypassing HadCRUT and GISS, and work towards a grand unified temperature dataset based first and foremost on quality.
Re: Craig Loehle (#54),
I’d love it too. The most accurate data set would be done by bloggers–so to speak.
Thank God for the internet!
Re: Anthony Watts (#36), Anthony, PUBCRUT is a great idea, I had been thinking of something similar. It could have nice clear rules – only rural sites, only publicly available data, no adjustments. I am sure there are people around with the time and energy to set it up.
On the Met Office question, note the final FAQ:
Re: Anthony Watts (#36),
You have my vote and my commitment. I want to setup a weather station in my yard, which I should be able to have 100 feet away from any structure or asphalt/concrete. I don’t know which to buy either. These are the things people need to know.
Re: Jeff Alberts (#68),
Jeff, I’ll write up something in a few days to help you and everyone else with the same questions.
I can say this now, whatever you do, do NOT buy an Oregon Scientific weather station such as the WMR-968. You truly do get what you pay for.
Re: Anthony Watts (#86),
Hehe, I got one of those OS thingies using Credit card points before closing one of my cards. It’s been pretty to watch, but fairly useless for accuracy, especially about what the next 12 hours holds weather-wise.
Re: Anthony Watts (#36),
As a bare minimum, century-long records are required to provide any serious statistical indication of climate variations on mutidecadal scales. The charming idea of starting a PubCRUT of new, impeccably sited stations, strikes me as impractical. Maybe I’m just too old to wait!
Despite micro-siting problems well-documented by you at the majority of USHCN stations, there are more than enough effectively uncorrupted records scattered throughout the lower 48 states to give great confidence in claiming that, on the whole, US temperatures in 20th century were esentially trendless. At +1.3K, 1934 was far and away the warmest year, followed by 1921 at +1.0K, as a geographic average estimate. The el Nino of 1998 did not produce exceptional temperatures west of the Rockies, and that year vies with 1931 for third place. What spurious “homogenization” of records corrupted by intensifying urban heating, land-use or instrument changes, and station moves has done to the geographic average is obscure the fact that a cooling swing starting in the late 50’s culminated in 1979 with the coolest year of the century, comfortably below 1917. And 1979 is the year when satellite observations began!
The whole AGW ruckus opportunistically seizes upon corrupted records in the ROW and the sharp swing from the globally coolest year (1976) to 1998 to provide baseless projections of an apparent 30-yr “trend” into the indefinite future.
BTW, on your recent visit to China, did you manage to unearth the rural station data that GHCN has buried since 1990? Even an up-to-date record from several such stations would help break the reliance upon China’s burgeoning mega-cities for regional temperature indications.
Re: Anthony Watts (#36),
Anthony:
What a wonderful idea. Kind of like GNU for climate data and methods.
I would suspect that most folks interested enough to be part of WU would be willing and quite forthright in complying and documenting siting requirements.
Where would we compile and collate the data?
What minimum data would we need to collect?
May I suggest a thread at WUWT?
Anthony Watts:
June 4th, 2009 at 9:19 pm
That is a great suggestion- I will put one up…
Why depend on Jones? Why not access the station data that he’s used, listed at this site, Jones’ 2006 paper, and do the calculations?
If the end result is vastly different from Jones, he’ll have to respond.
Re: Gary Strand (#40), No offense Gary, but if the data at the stations were also available with the station names and ID’s, your suggestion might be worthy of follow up.
Re: Gary Strand (#40),
The page that you link to exists only because of the efforts of CA readers and was produced after a prolonged FOI effort attempting to get data. After we got this page, I quickly found that they did not include full identification numbers for the listed stations that provide an unambiguous link to a primary source. The partial WMO numbers, names and coordinates all provide material for a concordance, but the spellings are frequently inconsistent and the lat-longs are sometimes inaccurate and the country IDs have varied, so that the concordance is time-consuming. I did it some locations.
In Canada, CRU uses national data that has not been incorporated into GHCN. It appears to tie into available national records again requiring a new process of collation and learning how to access another database. It could be done, but, as I recall, I had trouble scraping the data from this new format and gave up after a while.
Jones uses Australian data that is not public. I applied for the data but got no answer. I didn’t follow up as I was working on something else by then.
If I had trouble with Canada and Australia, then my surmise was that I’d have trouble elsewhere. I’m not averse to spending some time doing a certain amount of clerical work. I do it all the time. But it’s not a small job for 6000 or 8000 stations. In doing so, you create the risk of introducing your own collation errors.
But, back to your point, I don’t see why I or anyone else should spend perhaps several months doing this sort of clerical work merely because someone who is being paid by the public is refusing to produce data that the public is interested in.
At the end of the day, there isn’t a good reason for Jones not to archive his station data as used.
If Jones wants to play this sort of cat-and-mouse game, so be it. If I have an opportunity to pursue some new FOI avenue, I will. And if I don’t, undoubtedly someone else will.
If the Community is content with the impression that this gives the public of their commitment to openness and transparency, then so be it. My recommendation to climate scientists has always been not to get into this sort of dispute. The public has no sympathy for it. It may play well at climate science meetings and workshops, but people who live in the real world don’t get it.
And fortunately or unfortunately, the blogs are a way of putting a little sunshine on this. So rather than get frustrated with the obstruction, I’ll publicize the obstruction and let the public judge.
Re: Steve McIntyre (#55), you’ve got folks here that would be willing to chip in their time, I should think, to assist in collecting the station data.
If Jones is being a roadblock, seek another route.
Re: Gary Strand (#58),
I am.
In case you haven’t noticed, I’ve sent a number of FOI requests to UK institutions, making it harder and harder for them to continue withholding the data. CRU may have inadvertently sent another institution a copy of their data and that other institution, e.g. the Met Office, may have a different policy on stonewalling FOI requests.
Re: Steve McIntyre (#59),
If my memory serves me correctly, and it might be faulty so don’t quote me. At least in the US the creator of the record holds the keys, in terms of FOIA release. If CRU created the information, then they are the only ones who can authorize its release. If they provide that information to another agency, that other agency upon receipt of a FOIA for it will have to contact CRU to get permission to release. Or, more commonly the recipient agency will deny the request and send you off to CRU. (I handled a few FOIA requests for a government agency with whom I was then employed and that is what we were instructed to do.)
Re: Gary Strand (#40),
That would be interesting exercise for sure. Jean S had IMO a good suggestion earlier,
http://www.climateaudit.org/?p=1187#comment-88510
mostly we are interested in trends, so how about
1) for each station take the first difference ( say, of the annual averages)
2) take the median of the first differences – here’s an estimate of local temperature trend
3) average the trends of 60*n well-distributed stations
This is not as efficient estimator of the trend as OLS (under normal errors) , but it tolerates quite many step functions in the underlying error processes ( location change, incorrect homogenization, ..). CIs for both estimates would be quite easy to calculate (again, under i.i.d normal errors, Gaussian ARMA(1,1) processes, etc. ), and any conflict would indicate problems in the traditional approach.
This does remind me of the section in Douglas Adams’ The Hitch Hiker’s Guide to the Galaxy, when Arthur Dent’s house is being bulldozed for a by-pass. Dent protests he knows nothing about it, whilst the council official states that the “plans were on display”. Dent has to go into the basement of the council offices, with a torch (as the lights had gone). He eventually found the plans: on display, at the bottom of a locked filing cabinet, stuck into a dis-used lavatory with a sign on the door saying “Beware of the Leopard”….. link
This refusal to divulge raw data might not be so straightforward. There are possibly countries that have made data available to PDJ on condition that he does not pass it to further parties in a way that can be attributed to the providing party, this limiting liability for errors of fact, observation, malpractice or interpretation.
Having had experience years ago, I can attest that PDJ performed certain data selections that could be interpreted as selective bias, presumably (but not able to be proven without an admission) for the purpose of promoting a certain pattern of change.
For reasons flowing from that, I simply do not use HADCRU data and I strongly recommend to others that they do not either. If PDJ can boycott the distribution of data, readers can boycott its mention and use in Science. That would be equitable and just – and not an uncommon response.
I am directing my efforts at station scale (not at the huge global scale) to obtaining the raw data paper and email sheets that passed from from observers to offices.
In any case, a conscientious scientist ought not use figures when manipulation is reasonably suspected and whose final adjusted provenance is unknown.
Re: Geoff Sherrington (#49),
See:
Why does CRU have a confidentiality agreement with Germany?
by Steve McIntyre on May 31st, 2007
Re: Geoff Sherrington (#49),
There are also some issues about meteorological datasets which a nation and its NMS regard as sensitive with regard to national defense. Forecasting contrails, upper air conditions,….
RE: Anthony Watts (#36)
I’d sign up in a heart beat. When do we start?
The old page in the google cache:
http://tr.im/nxfK
My own guess as to why CRU is so adamant about not disclosing their data is that the reason is commercial rather than scientific.
They’ve been paid a lot of money over the years to produce a temperature index. And it’s enhanced their brand a lot.
My guess is that they’ve used this as a profit center to fund non-CRUTEM3 activities. Gavin Schmidt said that GISS spent something like 1/4 man-year annually on producing GISS, which he held out as an excuse for poor quality control. Schmidt observed that GISS did not actually collect any data, it relied on GHCN for data, and any errors should be blamed on GHCN.
My guess is that CRUTEM doesn’t take much more time. They get most of their data from GHCN. They get some from Canada, some from Australia separate from GHCN – which for some reason doesn’t manage to update GHCN. They get some from USHCN.
They probably get current data from MCDW, if GISS is any guide.
I did some posts in 2007 cross-checking CRU gridded versions against station data in areas where the stations were isolated (e.g. Barabinsk)and the CRUTEM series tied directly to the GHCN station.
My guess is that the entire CRU program can be reduced to a very small number of lines of code.
GHCN’s collection of rural stations after 1991 or so is almost non-existent. The issue in this field is whether the dominance of MCDW stations (mostly airports) after the early 1990s has introduced a bias.
Re: Steve McIntyre (#61),
This is an interesting and very plausible reason. Which begs the question: given both A. Watts and your own continuing involvement in collating, analyzing, and dissecting temp data, it would seem to be not much additional effort to offer your own gridded temperature dataset based on publicly available station data. In doing so, you could establish your own set of gridding routines, and back them up with code that could be run by any user on either your raw data, or their own. This could offer the possibility of becoming a benchmark, if your data were easily available, well documented, and cheap (or free).
Really, this isn’t a “if you’re so great, why don’t you do it yourself” post. It’s a “you’re already doing all this work now, why not show up some people with something better than what they are protecting so zealously” post.
Re: Joeshill (#64),
First, there’s a difference between analysing a business and running a business. As an analyst, you’re entitled to criticize a business or suggest that you sell their stock, but no one says, if you’re so smart, why don’t you set up your own bank or automobile company or airline.
Personally, I don’t have the faintest interest in trying to manage a proper collection of temperature data on a monthly basis, let alone on an unpaid basis.
Third, the world doesn’t need another temperature index, it needs the people that are doing them and who are being paid to do them to do them right. Which includes more openness and transparency.
Fourth, even if you produced another index, you’d be arguing all the time with the other indices. They’d say that you did something WRONG!!!
Fifth, I don;t share the enthusiasm of many readers for do-it-yourself things. If a temperature index is important, then it should be done professionally. My concern in this respect is that CRU and GISS are not doing the job professionally. Ross McKitrick observed long ago that a temperature index should be produced by the same sort of people that produce a Consumer Price Index.
It’s a clerical job, it’s not a job for high-powered people like Phil Jones and Jim Hansen.
Re: Steve McIntyre (#74),
This isn’t an argument. Really. Just discussing.
1. Yes, you are entirely right. There is a world of difference. But I think of it more like the difference between a pundit who analyzes the market and someone who implements their analyzes of the market to sell a product.
2. No argument there. Your time is your own, and if it doesn’t interest you, my hope would be that it interests somebody.
3. Yes and no. The world needs a Open and Transparent Temperature Index. With proper quality assurance, and version/correction tracking, and the ability to reconstruct the index as it was calculated at any point in the past. As in “I need the raw data and intermediate sets and the algorithm that was use on May 11, 2010 to calculate the April 2010 Mean Gridded Temperatures, and then I need the raw data and intermediate sets as they currently exist to calculate the April 2010 Mean gridded temperatures so that I can see what has changed in the raw data, what has changed in the algorithm, and how this has affected the intermediate datasets. (I see a huge value in this, but I don’t know if there’s a huge economic value, which probably drives the amateur implementations that exist now, rather than a professional implementation that could be.)
4. People could claim wrongness, but the beauty of an open system is that either the claims have merit and algorithms can be changed so that everyone can agree on a correct implementation, or the claims have no merit and are proven wrong. Where the claims come down to a legitimate difference of opinion, the project can be forked to include multiple implementations, each open.
5. I agree. It is a clerical job, and when it becomes economically viable, somebody will step in and start doing it professionally. (I regard clerks as the professionals that provide implementation that scientists don’t generally have the interest to follow through with.)
Re: Steve McIntyre (#74),
Steve, I think you are short changing the CPI. In fact, there are adjustments made to the CPI that are more complex and controversial than anything in HADCRUT and GISS – just google “Hedonic Price Regression”. Of course, the CPI actually has real and direct economic consequences, so the the people responsible make sure they are accurate and disclose their methods.
Has anyone looked at MADIS and reviewed the “automated QC” adjustments made by NOAA?
Re: D. Patterson (#62),
I think MADIS data QC flags the records as failed, but do not alter them:
“Quality Control (QC) of MADIS observations is also provided, since considerable evidence exists that the retention of erroneous data, or the rejection of too many good data, can substantially distort forecast products. Observations in the ESRL/GSD database are stored with a series of flags indicating the quality of the observation from a variety of perspectives (e.g. temporal consistency and spatial consistency), or more precisely, a series of flags indicating the results of various QC checks. Users of MADIS can then inspect the flags and decide whether or not to ingest the observation.”
The failed QC data is still there and can be used or visualized:
I have sent the link to this to the UK Daily Mail, it is one of our more vociferous national newspapers , and following the MP’s FOI issues , they are now pursuing the BBC over their hiding of salary details behind the FOI act. As it is topical , they may jump on the bandwagon.
http://www.dailymail.co.uk/home/index.html
Oops, someone didn’t close a tag…
Houston, we have an anomoly, can you adjust it? 🙂
Re: D. Patterson (#70),
Only if you ask nicely! ;P
The section on “How not to do science” in my Introductory science course keeps getting longer and longer.
Re: 59
I don’t think that individual institutions have a choice in this – I think their policy can only be to comply with legislation:
http://www.ico.gov.uk/what_we_cover/environmental_information_regulation/legislation_in_full.aspx
I think future FOI requests may benefit from being copied to the Information Commissioners Office along with copies of previous requests. My reading of the legislation is that both the obligation and the spirit is one of compliance.
Steve – can I suggest you read this:
Click to access environmental_information_regulations_code_of_practice.pdf
About 30mins worth – I don’t see that it leaves any “wriggle” room. IMO a request under the EIR direct to the Chancellor of UEA should provide you with the information you require within 20 days. I would also add that the Met Office appear to have failed in their obligations to transfer the request to UEA.
Re: curious (#75), It sounds nice but the history of FOI requests is that most of them are ignored or stalled–in spite of what the law says.
Re: Craig Loehle (#79), Sure, but that is a whole lot harder to do if there is a CC to the Information Commissioner and a CC to a public blog. Also the EIR looks stronger in that it places an emphasis on the spirit of the Law. My view remains that a request to the Chancellor transparently CC’d to the IC and CA blog will secure the data. In the link I gave in 75 IMO it addresses all the issues very clearly. If it were sent today we’d know in 3 weeks.
Re: curious (#75) and others,
Ultimately we all have to obey the law, but are still entitled to due process. FOI is fairly new in the UK and by no means all the ridiculous reasons for refusal have been tested at law, so almost any request could be resisted to the House of Lords if the public authorities have the nerve. Remember that our MP’s have spun out their pathetic case for nearly four years. They and other public authorities can spend huge amounts of taxpayers money resisting disclosure on the flimsiest pretext. CRU and the Met Office may have just as much reason as our MPs not to disclose information.
Most people reading this site will have worked out that there is often a simple reason people do not want to be open and transparent.
Re: curious (#75),
Addressing a request to the chancellor of a British university would seem to be pointless rather it should be addressed to the vice-chancellor.
* UEA has instructions on its FOI requests page: http://www.uea.ac.uk/is/foi/foi_requests
* Under FOIA 2000, UEA is required to have a Publication Scheme: http://www.uea.ac.uk/is/foi/pub_sch
* The PS has a section about Policies: http://www.uea.ac.uk/is/foi/pub_sch/policies
** Research, Enterprise and Engagement Office (REE) has a list of Policies, etc.: https://www.uea.ac.uk/ree/documents
*** REE “Guidelines for Good Practice in Research” is a guideline, not policy: https://www.uea.ac.uk/polopoly_fs/1.34534!policy-good-research-practice-apr2006.pdf
Because writing to the investigator has failed, I suggest that FOI requests be directed to the University’s “Information Policy and Compliance Manager” to ensure that the requests and replies will appear in their FOI records.
In 2006, there were 77 FOI requests to UEA during the first full year of the act. So there is someone processing the requests there.
I should also say that 14.ii of the Guidelines to my mind will give full weight to an email request. Perhaps 30secs of cut and paste with the Guidelines attached as a .pdf?
Re: curious (#82),
look, Willis tried to FOI the information from Jones a couple of years ago. Please look at the relevant Category in the left frame for the correspondence. David Holland’s been stonewalled on other matters.
I do not think that UEA-CRU refused the requests because of the failure to say some magic word or because some guideline wasn’t copied. Their intention is to stonewall. IF one reason doesn’t wash, they’ll try another. They don’t believe that anyone’s going to bother taking them to court and they’re probably right.
Look at the experience with the UK Met Office and John Mitchell’s IPCC Review Comments. They made absurd and untrue answers to avoid producing the information.
If someone thinks that an FOI request adding some magic word will do the trick, then send an email making the request to UEA-CRU. Aside from finding out for yourself, there’s a difference between me and/or Willis and/or David Holland making the request and a lot of people. And it will make them think a little more about whether they’re really following the right course of action in keeping their data secret.
What possibly would you gain by not allowing access to something?
Is it official government property? Are their things that are better not seen?
What possibly could cause those chaps to shy away from sharing?!
Ok Steve – only trying to help. In the time you did that post you could have sent the request so I’m at a loss what else to suggest.
The Guidance is quite clear on issues such as non discrimination according to applicant etc. but you appear to have given up. What do you want us to do – apply and then forward to you?
Re: Court – it is the IC’s job to ensure compliance:
Source: http://www.ico.gov.uk/about_us.aspx
Willis – if you are reading this I think your letter to the Chancellor and the previous one to Mr Jones miss the mark. The IC and legislation are clear about the procedure – make a direct request under the legislation and they supply the data according to the criteria laid down. Read the guidelines I referenced above – there is no doubt that public interest and presumption in favour of disclosure are at the heart of the regulations.
Re: curious (#85), I have made a proper FOI request and I was stonewalled. What further action are you proposing?
w.
Re: Willis Eschenbach (#93), Hi Willis – read the link I give in 75. I do not see how a request under the EIR can be refused. My suggestion is you make a request direct to the Chancellor of UEA cc’d to the IC (and publicly to CA if you wish) attaching a copy of the guidelines. If you wish, highlight or reference all the relevant paragraphs (there are many of these) and, again optionally, attach your previous request(s). Make it quite clear your request is not about the process or failed previous attempts but that it is for the data. Including reading the doc. I’d say 30mins should cover it. As another poster pointed out FOI is a hot topic in the UK at the moment so if it is bounced I think there will be journalistic interest. It could be that you find a journalist would even submit the request to test the process.
Dave at 88 –
Quite – some examples:
http://www.ico.gov.uk/tools_and_resources/decision_notices/2009_05.aspx
Phil at 89 – As Willis had been in touch with him he will be aware of the issue and expect he’ll pass it to the appropriate person for action in accordance with their oobligations:
http://www.uea.ac.uk/vco/Chancellor
You don’t see the director of the CPI report spending 98% of this time on other work. Whether the CRUTEM index is hard or easy, Phil Jones cannot possibly spend much time actually working on the data, given his participation in IPCC, attendance at meetings and workshops, coauthorship of academic papers, his university duties etc. Nor does Hansen.
But I’ll bet you dollars to doughnuts that they get a lot of money from funding agencies because the output is visible and they use that money to fund the things that they want to do – like academic papers or travel or whatever.
They don’t want to see how little work they actually do on the data.
People who talk about volunteers running around collecting data misunderstand what CRU and GISS do – they don’t run around collecting data. They get a few downloads a month from different places, run a few simple programs – they only appear complicated because the programs are written for antiquated machines.
I worked through GISS for a while, but it’s easy to get tired of it. I did quite a bit of concordance on CRU as well.
I’m probably not all that far away from being able to automate both indices from a home computer. But it takes time to dot the i’s and cross the t’s. For example, I’ve just spent a couple of days figuring out GISS binary data, most of which I couldn’t handle readily before. This was partly me – I hadn’t worked with binary data and wasn’t handy on it. So I had to ask someone (Nicholas) to help me with the files. NOw I’ve figured it out and my productivity is much better.
It’s only a matter of time before we have an R-emulator of GISS.
Why anyone would want to emulate GISS in any respect is totally beyond me. 😉
Sounds like a great plan, curious, and one you know the ins and outs of. Since it would only take you a half hour (I’d require much longer just to download the regs, Solomon Islands connections are slooooow), I invite you to go ahead, and let us know how it works out. Me, I’ve beaten my head against that wall enough. I wrote to the Chancellor … you can file the UCI or whatever it is called.
I seriously invite you to do so, and to report back on the outcome. Become a player rather than an observer.
All the best,
w.
http://www.ico.gov.uk/complaints/freedom_of_information.aspx
Seems to be a well established process for pursuing a complaint about how you’ve been treated, Steve. I think this is important enough to follow through as a test case, and any publicity we can get about ANOTHER abuse of FOI in the British govt can only help.
I used to be someone who hated litigation and saw it as a detriment to society. I no longer feel this way, its the people who violate rights and property who are damaging society and the only way to minimize the damage, short of a .44 magnum, is to pursue all legal options.
I started an email chain to Briffa & Jones three weeks ago following your earlir post, pointing out that this data should be made available. I copied in the business managers at the university, their FOI officer, press office the vice chancellor and chief executive. I left it a week then added some Met Office addresses and the Office of Climate Change Minister and the CE of the Royal Meteorological Society.
.
I got a response from the RMS secretary telling me he was on hols until about now. I had a couple of other out of offices, so a week later I added the BERR Ministry, and the Met Office came back a few days ago and said that they’d drawn it to the attention of someone at Hadley. I sent them a followup thanking them for their action and drawing their attention to this thread. I’ll leave it a little while to see if they come back with anything, and will then forward all that info to the fuller email list (19 addresses so far)
.
I plan to expand the number of government addresses, and then start adding in media until hopefully someone feels embarrased into giving Jones a kicking. Included in the original email was a link to one of the recent threads on this site for information Another Try on the Wall. Everyone reading this thread should understand that the FOI requests have been blatantly abused and whilst the Information Commisioner might be interested and that is worth pursuing I think a bit of direct action would help.
.
Steve would be interested to see if you’ve had (m)any hits from uea.ac.uk, rmets.org, mets.org, occ.gsi.gov.uk, berr.gis.gov.uk addresses as I would like to be able to relate this in the next one.
.
Ian
Ian
IIRC the Met Office is part of the Ministry of Defence, so you might want to contact the responsible minister in that department. Also UEA, as a university, is part of the Department of Innovation, Universities and Skills. Or at least it was until yesterday. I think they are reorganising it all. Bear in mind also that in the next 12 months there is likely to be a change of government in the UK.
While the Met Office may be part of the MoD, I was not aware that there was a security classification called “Because Dr. Jones Says So”.
Re: Mike Lorrey (#100), Shows how little you know…
Steve mentioned how this temperature index has improved CRU’s brand, and it probably has. But I think the stonewalling we now see, is doing the opposite; it is destroying both the brand and credibility of CRU. They should really reconsider this. It takes a long time to build a brand, but a short time to wreck it.
Isn’t it time soon, for Dr. Jones to wake up and give Steve a call?
I was involved last week in formal discussion of an FOI request at my own UK academic institution. There was no question whatsoever of ignoring it or stonewalling. The only discussion was on whether we should answer the question actually asked or a slightly more general question which the inquirer probably meant to ask. We decide on the second course and are now preparing a complete answer.
Individuals are often inclined to ignore FOI, but FOI officers typically take these matters very seriously. Doesn’t guarantee you’ll get the data, but you should at lest get a proper explanation if they can’t/won’t release it.
HOT POTATO!!
The dancing around makes them look suspicious. They do it to themselves!
Anthony, #36
I have been thinking a bit about your proposal. In the process, I have become more forgiving of the flaws you are noting in your study.
Let’s assume for the moment that all us amateurs got together and started the network now. We are currently enthused about the need for a “better” temperature network, so things will probably go swimmingly for a while. I get my station set up on my 4 acres next to a power substation site of 40 acres, mostly small trees with a small substation in the middle, and my neighbor’s 6 acre miniature cattle farm. There is a development on my south property line, but I can locate instruments favorably.
Other members happily volunteer and locate stations in favorable spots. All is well. Well, no. I am now 68 and in, say 4 years, decide it is time to move to an apartment. The station is still in place, but the new owner develops the property. It now becomes the site for 24 houses. Looks to me like this will mean the end of the station record. How to deal with the relocation issue is going to take major effort on someone’s part, especially because this will be typical of many stations. Even if a location is not developed, the new owner of a site may not care, or may not want the site there, especially if it means work or inconvenience for him.
I seem to remember that a few years ago it was noted that the average U.S. resident moved every 7 years. This figure is probably out of date now, but it indicates the potential magnitude of the network update problem.
One solution is to locate stations at facilities where station maintenance and reporting can be part of the activity of the local government organization using the site; Fire stations, for example. You would be more likely to retain continuity of site location, but then you have to be able to deal with parking lot expansions, barbeques ….. Is this beginning to sound familiar?
I guess the point of this post is that I am beginning to understand some of the location choices made over the years for USHCN. The problem is complex because not only must good sites be selected, but they must be maintained and the results reported. Compromises will be necessary as USHCN has discovered. Those compromises will affect the data gathered. Better perhaps to rescue the existing data from “corrections” and recognize the resolution and precision (or lack thereof) of the present system.
Having said all that, I look forward to identification of additional problems I have not considered and proposals to solve them.
Re: Stu Miller (#106),
Cemteries…
Pro
Some long-term continuity of site parameters
Con
Lack of technical infrastructure: electrical power, wifi, Internet access
Re: Stu Miller (#106), thanks for pointing out the difficult and non-trivial nature of keeping an observation site going and well-maintained. Signing up volunteers is easy – keeping a continuous record suitable for climate studies (30 years minimum) is not.
There are some important points I want to make and will illustrate them with respect to my request to Ofcom to see the complaint against the ‘Swindle’ programme allegedly made to them by the IPCC, and also to be told who signed it. Why I say “allegedly” and why I think it is important to have the “IPCC” complaint disclosed is partly explained in the complaint, to which I have linked above, but is not important to this thread. In UK law you do not have to say that the request is made under FOIA or EIR, or why you want the information. As, so far, every important release has at least required the Information Tribunal (IT) to rule on it, I think you might as well put in the reasons, as the Information Commissioner’s Office (ICO) and the IT will need to know them to consider the ‘public interest’ test.
Ofcom first said they would not release the “IPCC” complaint because, under section 393(1) of the Communications Act 2003, they need not. This section says Ofcom need not release information given to it by a business. In legal parlance this section was obviously not ‘engaged’ as the IPCC does not fit any normally accepted definition of a business.
This is the first point.
Under section 38 of the DCA Code of Practice issued under section 45 of FOIA , a public authority should (but is not legally required to) treat any hint of a complaint in writing, to their response to an information request, as a “complaint” and institute an ‘internal review’. With CRU, the Met Office and others, as contributors to this site know, one may have to be very blunt rather than hint, to get them to accept one is complaining. However, merely pointing out politely that the IPCC is not a business, was sufficient for Ofcom to start an “internal review”, which took 113 days. This would be within the law but contrary to the Code of Practice for FOIA. However, under the Environmental Information Regulations, 40 working days is the maximum allowed.
After this long and supposedly careful consideration, Ofcom, the British Communications regulator, gave a second reason for refusal that was also not ‘engaged’ without addressing the issue of the failure to classify my request as environmental information which had also been strongly made in my reply. Ofcom said the information I had requested was already available on the Internet. It is not. At that website the IPCC complaint is not shown, nor does it make clear who sent or signed the complaint. It does however say, if you look through all of its pages, that it was “co-authorised” by Pachauri, Houghton, Parry and Watson.
This is the really important point.
At the foot of the Ofcom “review” letter it stated,
Ofcom had, I thought, obviously just made another mistake, and hitherto it had been my general understanding of legal procedure that you should exhaust all the opportunities for a resolution of a dispute before applying to a superior authority. Indeed section 50(2)(a) of the FOIA says so.
However, and this is the trap, section 50(2)(b) says that there must not be an “undue delay” in complaining to the Information Commissioner’s Office, which the ICO arbitrarily say is two months from the public authority’s final reply. This section is poorly drafted in that it does not differentiate between delays from the complainant and those from the public authority. My correspondence to Ofcom added 20 days including drafting my complaint to the ICO. Ofcom’s replies to me added 92.
On this basis the ICO has thrown out my complaint against Ofcom. They say they have a rule that they consider the first letter from a public authority announcing the result of an internal review as their final reply. If any reader finds this rule on the ICO site for “members of the public” please let me know. This is not the end of the matter but demonstrates that the ICO will try as hard as the public authorities to dissuade us from pursuing matters. The ICO are reviewing their decision but I am resigned to having to go to judicial review if they do not reinstate the complaint. To avoid this hassle, you need to complain to the ICO within two months of an internal review reply or if you get no reply to a request or a follow up within 2 months of the request.
As for writing to the University Chancellor (actually it’s the Vice-chancellor you want) or Ministers of State, in my experience it is an absolute waste of time. FOI is a pain in the neck for them and they will not intercede on our behalf. As we know from dealing with Jones, Mann, Chuine, Steig, Santer and others, they probably all know that the truth will out and delay is their best defence, hoping that by the time we get the information they will have moved on, leaving us a yet more complex study to reverse engineer.
The FOIA will only work easily for us when someone has done the groundwork to get case law established. First, and this is why it is worth going all the way with Ofcom, we need to establish that virtually all information about climate change is ‘environmental’. This will box in CRU and the Met Office and would not allow the ICO to use the section 45 Code of Practice so easily. Ofcom have no obvious interest in protecting the IPCC, they just don’t want to be shown to have been conned by Dr Pachauri and friends. They are an easier target than CRU and the Met Office, who might well go to the House of Lords to prevent the full disclosure that we need.
Re: David Holland (#112),
Very informative. Thank you for that, David.
The “tricky-dickie” regulations are clearly written by bureaucrats well versed in the Humphrey Appleby “masterly inactivity” view of life. That politicians encouraged them in this is self-evident.
Next is to classify this data as of National Importance (security of energy supply, food supply etc). Watch this space.
Hi Willis (and anyone else seeking data)
I had a spare half hour so I put this together:
You, or others, are welcome to use or modify it as you see fit. The Guidelines I referenced are those linked in post 75. On my machine they are about 72KB downloaded which, compared to this CA page at 356KB as downloaded just now, should be manageable even on a slow connection.
Noting AnonyMoose at #78, Mike at #97 and Jonathan at #103 I suggest they are sent to the Chancellor by email with a “read request” and a paper cc to the UK ICO:
https://www.ico.gov.uk/Global/contact_us.aspx
Hope that it is of some use and counts as a contribution to the debate. If it is sent today the data should be available before the end of the month.
Re: curious (#113),
As I remarked earlier the correspondence should be to the vice-chancellor who is the top administrator at a british university, the chancellor is just a figure-head. For example, when I was an undergrad the chancellor of my university was HRH The Duchess of Kent, at Cambridge it’s Prince Philip, at Oxford it’s Chris Patten, the former governor of Hong Kong.
Re: Phil. (#114), Hi Phil – As I replied in 91, based on Willis’s post of 35, I suggested the Chancellor for continuity but I’m sure the VC would do just as well. So long as it reaches a senior figure and they are told it is an EIR request all they have to do is read the guidelines and then pass to the appropriate person for action.
Re: curious (#117),
The point is that the chancellor isn’t an administrator and usually isn’t present on the campus for long periods of time so it would be completely unproductive sending it there and would likely lead to delays.
For UK universities the vice-chancellor is indeed a better person to address than the chancellor, but I suspect that in fact the right person might be the registrar.
I just called Ms Kreetzer who is the contact listed in Pat’s post at 33:
http://www.uea.ac.uk/vco/Chancellor
She was very courteous and efficient and tells me the email is monitored and that she will pass any request on to the appropriate persons. For info. she is also listed as the PA to the VC.
That covers all the “spade work” – over to our data seekers now.
Re: curious (#119)
I don’t mean to be unkind or dampen your enthusiasm, but as I mentioned in #111, in my experience, we are just wasting time thinking we can appeal to a higher authority, within the public authority that we are dealing with. As I explained we risk running into trouble with the ICO. Have you read it?
I also have complaints with the ICO against Defra, UEA, Reading, Oxford and the Met Office resulting from requests which have been discussed here. I could not believe that their senior management could possibly condone what they were doing. I wrote to Hilary Benn at Defra, Bill Macmillan VC of UEA and my MP who raised the matter with the Minister of State for the Armed Forces.
A Defra official replied for Benn with the remarkable statement subsequently confirmed as authorised by the Government:
In my letter to UEA’s VC I complained not only at absurdity of their Director of Information claiming that no information on climate change was ‘environmental’, but also that they clearly had no proper publication scheme in relation to their work on climate change and also that they were in breach of the Data Protection Act. The latter breach was on account of misusing my personal data to collude with the Met Office and Reading to “resist” my information request by applying section 36 of the FOIA. Both the Met Office and UEA used Ministerial certificates to apply this exemption which they could not do if they accepted the information is covered by the EIR.
If you have not read the Aarhus Convention, and searched the Greenpeace High Court Judgement for “Aarhus”, please do so. Please also read section 3 of the EIR and tell me how you think UEA and the Met Office are going to persuade the Court that email discussions between Caspar Ammann and Keith Briffa on how to respond to criticism of the Wahl and Ammann papers is not ‘environmental information’.
The UEA VC passed my letter back to his Director of Information, who said “go to the ICO”. The Minister of State for the Armed Forces told my MP:
Now if you can tell me where in the ‘Principles Governing IPCC Work’ or its Appendix A it says this please tell me.
Regrettably, “you ask – they tell” only applies if they want to tell. The reality is, you ask and they say no or give you the run around – then you ask for an EIR r.11 reconsideration ( if its climate change it is environmental) – they still say no – you complain the ICO.
The ICO’s first instinct will be to find some excuse for not giving you a ‘Decision Notice’ which five judges thought would prevent you going to the Information Tribunal. However three out of five Law Lords said in some cases you can, so you complain to the IT but, to be sure, also put up £50 for a judicial review (within 28 days) in case the IT say they will not consider your complaint. If the IT won’t consider it, because they don’t think the Lords ruling applies, then you have to take a deep breath and go to judicial review.
If the ICO does accept your complaint and rule in your favour, UEA or the Met office can spend your tax to appeal to the IT, and as the BBC just did (and lost) as far as the Lords. On the slightly bright side if you have any case at all you will probably get a ruling, if you ask, that you are not responsible for the other sides costs.
Now can you see why no one is going to hand over any climate change information without a fight?
Hi David – I have some experience of the type of run around and blocking that can go on. And I know it can drive one to distraction. However as others have noted using the tools in place to provide access to public information is the only way forwards. I picked up on this post by Steve for these reasons – IMO these issues go to the heart of democracy.
Steve’s request was bounced by the Met Office as they do not hold the data he is seeking. IMO they should have referred the request on to UEA as the organisation (not individual) responsible for the data. They didn’t, but that doesn’t change the UEA’s responsibility to comply with legislation. I haven’t seen the whole history to your attempts but I can well understand your frustration. Despite this I would still encourage you (or others) to apply today for the data. Not correspondence etc etc – just the data that Steve requested in the lead post. There is still 30mins to close of business in the UK so the request could be filed today in less time than these posts have taken to type and read. Kind regards C
Re: curious (#121),
Are you a lawyer? I am not but have spent some time reading the Acts, Regulations and case law.
Have you read the ICO guidance note on vexatious or repeated requests? The right way to use the tools in place is to follow the procedures laid down. Multiple requests may get identical replies for a while then a FOIA section 14(1) or EIR regulation 12(4)(b) refusal when the public authority recognises that it is a coordinated affair. This would not be good for Steve or any of us. You will not bully UEA and the Met Office. Only the ICO, IT or the Court will do that as our MPs found out.
There are plenty of requests to make but each one should have one requester who is prepared to press it professionally and resolutly according to the procedures, which will inevitably mean judicial review at some point because UK law is not compliant with European Law. The key battle is to get a the ICO, IT or the Court to enforce EIR regulation 4 in conformance with Directive 2003/4/EC and Aarhus.
If there are volunteers in the UK who are prepared to cooperate in learning the procedures rules and jargon that are needed to succeed in EIR/FOIA I would be more than happy to be part of a private self help group. I am convinced a professional effort could radically change the way climate science is undertaken here. However a multiple request campaign is not a good idea.
Incidentaly, if the data Steve is asking for is ‘environmental’ (and I am sure it is) and the Met Office actually do have it (which is possible on previous performance), some official is risking a £5,000 fine.
Hi David – lets be clear – I am not advocating a “vexatious” campaign. What I have done is supplied links to the published background info. for those seeking data. To put it in context, Steve’s column-heading post states he intends to make further FOI’s:
My point is that the EIR is the applicable process, a fact the Met. Office are in agreement with:
Hence the links I provided to Steve in the earlier posts. Then Willis picked up having sent an enquiry to Dr Jones re: process, and a communication to the Chancellor of UEA following on from Pat’s post at 33. As I stated IMO these miss the mark – the EIR info. in the Code of Practice I reference in #75 explains the process and the criteria to be applied. IMO, as I read it, the Met Office had an obligation to pass Steve’s request on to UEA but didn’t. So, no big deal – for those who want the data simply make a direct request to UEA.
It is not a difficult thing to do and I do not understand why those seeking the data do not do it. Nor do I understand why you are of the opinion one has to be prepared to continue to court etc etc – the Met Office have acknowledged that the request is valid under the EIRs.
My reference to journalistic interest was because there is the additional issue of proper working of democratic instruments. As far as “vexatious” is concerned – from what I can make out no one has asked UEA for this info. using the EIR and hence I think to define a first request as “vexatious” would be perverse indeed. Apologies if this has happened and I’ve missed it.
So the info. is there – if you or (note: “or” not “and”, not “and everybody else who’d like to stir things up regardless of whether they want the data” ) others want the data, read the guidance, use the process and await the response. I suggest a cc or note to CA would prevent/limit duplicate requests and I would also say my reading of the regs. is that the request should be made by the person seeking the data. If a request had gone in yesterday IMO there would only be 19 days to receipt. However, in answer to your query, I’m not a lawyer so feel free to seek more qualified opinion if you wish.
Re: curious (#123),
That is what EIR r.10(a) says but goes on to say “or (b) supply the applicant with the name and address of that authority”. The Met Office complied with r.10. One could quibble about the address but the Met Office know who Steve is and that he knows who Phil Jones is.
Unfortunately the ICO has just emailed me to say they will not reopen the Ofcom complaint so I must break off this interesting discussion and return to to that battle.
Re: David Holland (#124),
Oops! Ignore the last sentence in 124. The email came in while I was typing the last comment and I just glanced at it in my preview window. The bit that caught my eye was the bit in bold in which the ICO restated their previous position. Only right at the end do they say they are taking the Ofcom complaint forward and will update their advice to complainants.
However curious, I think we have beaten this thread to death.
Regarding access to Government data:
http://news.bbc.co.uk/1/hi/technology/8096273.stm
Sir Tim [Berners-Lee]refused to be specific about which items of data he would help make public, but said he was interested in suggestions from bloggers about what they would like to see.
One Trackback
[…] is crap, and modern poetry is crap. Unfortunately, modern science is rapidly becoming crap. As Climate Audit shows, global warming “science” has abandoned that inconvenient stuff about replicable […]