OK, folks, guess what. I’m now in possession of a CRU version giving data for every station in their station list .
In their refusal letter, the Met Office described adverse consequences of disclosing CRU station data, an event that apparently would let loose the Four Horsemen of the Apocalypse.
The Met Office stated:
Some of the information was provided to Professor Jones on the strict understanding by the data providers that this station data must not be publicly released. …
If any of this information were released, scientists could be reluctant to share information and participate in scientific projects with the public sector organisations based in the UK in future. It would also damage the trust that scientists have in those scientists who happen to be employed in the public sector and could show the Met Office ignored the confidentiality in which the data information was provided.
They continued:
the effective conduct of international relations depends upon maintaining trust and confidence between states and international organisations. This relationship of trust allows for the free and frank exchange of information on the understanding that it will be treated in confidence. If the United Kingdom does not respect such confidences, its ability to protect and promote United Kingdom interests through international relations may be hampered.
Competitors/ Collaborators could be damaged by the release of information which was given to us in confidence and this will detrimentally affect the ability of the Met Office (UK) to co-operate with meteorological organisations and governments of other countries. This could also provoke a negative reaction from scientist globally if their information which they have requested remains private is disclosed.
And that’s not all. There’s more:
to release it without authority would seriously affect the relationship between the United Kingdom and other Countries and Institutions.
CRU was a less dramatic but still very clear about the consequences:
we feel that there is a strong public interest in upholding contract terms governing the use of received information. To not do so would be to potentially risk the loss of access to such data in future.
Just to prove that I have actual CRU station data, here is the 60th series (Lund Sweden), covering the period 1753-1773: sensitive information indeed.
1753 -1.8 -1.3 3.7 7.6 11.5 14.6 16.7 15.9 13.4 9.9 3.2 -3.0
1754 -1.0 -1.4 -1.2 5.6 12.9 15.2 15.1 15.5 11.9 10.1 4.7 1.9
1755 -3.8 -5.3 0.8 7.9 12.0 17.8 18.2 15.4 12.1 8.4 3.7 2.0
1756 1.9 2.3 2.5 4.2 9.9 17.6 19.4 15.6 14.1 9.2 1.8 -0.1
1757 -2.8 0.7 1.4 8.2 10.7 18.2 21.4 17.6 13.6 5.2 6.0 1.3
1758 -3.9 -2.0 0.7 3.3 13.9 16.7 16.0 16.8 11.8 6.7 4.5 1.0
1759 2.4 2.3 3.4 6.2 10.2 17.4 20.1 18.1 13.1 9.1 2.1 -2.0
1760 -4.0 -1.0 0.7 6.1 11.8 19.2 18.2 17.1 15.3 8.5 4.0 2.5
1761 0.6 1.2 5.0 6.8 12.9 18.0 17.3 18.3 15.2 6.3 5.1 -0.6
1762 1.1 -0.7 -1.8 8.1 11.5 17.0 17.4 14.2 12.4 4.8 4.1 0.5
1763 -3.9 0.5 0.5 4.6 11.2 14.9 17.8 16.9 11.5 7.7 2.8 3.0
1764 -0.1 3.0 1.4 5.6 12.5 13.6 20.5 16.3 11.8 7.6 2.4 0.1
1765 -0.3 -2.3 2.9 6.9 10.2 15.3 15.9 16.9 11.9 9.0 4.5 -0.2
1766 -1.8 -2.7 2.1 8.1 11.9 17.3 18.8 17.2 13.8 8.7 5.8 -0.9
1767 -6.1 -0.6 2.1 2.7 9.8 13.9 16.4 17.3 15.0 8.9 6.4 0.4
1768 -5.5 -3.1 -2.4 5.1 10.7 16.3 17.9 17.1 12.5 8.2 4.9 2.1
1769 0.6 -0.5 2.3 5.7 11.3 15.6 17.6 15.9 13.6 5.2 2.6 3.2
1770 -2.3 0.0 -2.9 4.5 11.5 15.1 18.1 18.1 15.4 10.5 2.5 1.5
1771 -3.8 -3.8 -3.9 2.2 12.4 18.0 17.2 15.1 12.5 10.1 2.8 2.5
1772 -1.6 -2.1 -1.1 4.7 10.0 16.2 17.8 17.1 13.6 11.0 7.2 2.9
1773 1.0 -0.9 1.6 7.3 14.1 15.9 18.1 18.0 14.5 11.2 5.0 2.6
It’s hard to imagine that my being in possession of CRU station data would “damage the trust that scientists have in those scientists who happen to be employed in the public sector”, interfere with the “effective conduct of international relations”, “hamper the ability to protect and promote United Kingdom interests through international relations” and “seriously affect the relationship between the United Kingdom and other Countries and Institutions.”
But that’s what the Met Office says.
Given such dramatic adverse consequences, I wonder what they’ll do. Will they investigate? I wonder what the form of investigation will be. Will they do it Jack Nicholson-style?
I’ll keep my eye out for secret agents from MI-5. The data’s in a safe place, but I doubt that I would bear up well under waterboarding.
And by the way, just because I’ve got a version of the data doesn’t mean that I’m going to give up trying to get the data through FOI. Quite the opposite.
Climate Audit 
262 Comments
That is so savvy. A brilliant position to be in really. Their excuses in continued opposition to the FOI request will be ultra asinine in the face of your already having the data they tried to claim was sensitive and in need of supressing.
Hat tip to the Mole. A few more of them spread around various agencies and the con job will be over.
Do we get to play 20 questions to figure out who the mole is?
Please don’t. Given the Met OFfice’s serious view of the consequences, it could compromise the mole’s career. I ask that people don’t indulge in such speculations.
In any event, I got it anonymously. I will say this much – the data didn’t come via Georgia Tech, it came from the UK.
There was a hole in the bucket, dear Henry, dear Henry …
And I bet you didn’t even have to empty the tip jar to get this well kept, secret data of strategic value for the Nation.
That was a joke, of course. But it brings up an interesting question. What would happen if you made an ultimatum that you’d release the data at a certain date as is? And send send them a copy of the data, suitably scrubbed to avoid anyone identifying where it came from. Tell them you’d not release it provided they’d verify that this is the correct data with a few minor changes which you’d inserted as tests and which they’d return to prove they’d verified it. If they wouldn’t verify, you’d assume it was not the correct data and would feel under no obligation to not release it.
The purpose of this process would be to prevent them from denying later that any results you got from playing with the data was valid. There might still be a he said – they said situation, but ultimately your results could be verified without release of the data.
You know, not everyone in every organization believes in everything the organization does. This is why we have leaks in the White House and people like “Deep Throat” that provide evidential tidbits with guidance like “follow the money”.
Steve has shared this data and the source with me, as a way of verification, and I can vouch for both the validity of the data and of the source ip address. It truly comes from deep within the organization. – Anthony
Beware the ‘canary trap’, data that is ever so slightly jiggered so as to make the individual source apparent. Say, changes in the sixth digit of otherwise innocuous date, or extra whitespace, semi-colons instead of commas.
Refs:
(1) T. Clancy; The Hunt for Red October
(2) Slashdot; http://yro.slashdot.org/story/09/07/24/2243209/IBM-Seeks-Patent-On-Digital-Witch-Hunts
The canary trap is used by major atlas publishers such as The Times Atlas, with the use of fictitious villages or slightly wrong longitude-latitude values.
How do you know that you have the actual dataset?
Given Jones’ unflagging determination to keep this data away from unbelievers, unpleasant legal threats may not be out of the question. I hope the leak is rather broad so that it’s pointless trying to contain it that way.
“Beware the ‘canary trap'”
Heh, I know for a fact that this is done. Had someone back when daisy wheel printers were first coming out ask me if there was a way to modify the font so the output could be traced to a specific unit. Said they had already done this with the typewriters in the office so any “leaks” could be traced back to the typewriter that produced the document but they were having some trouble with these newfangled printers (no, it wasn’t a government agency).
“I hope the leak is rather broad so that it’s pointless trying to contain it that way.”
Brings BitTorrent to mind.
I think we should take donations to protect “deep throat” he is going to be sacked once this is out.
Can I speculate that it is William Connelly?
Re: steven mosher (#13),
Speculate all you want but don’t expect a conversation.
The real message is that Jones’ stonewalling can’t last much longer. Either he tells us what the data is and what the methodology is for the purposes of replication or the negative publicity will simply drown out everything else. He’s got everything to lose.
Steve
There is a danger you may have won a battle but will lose the war. The CRU/Met Office pride themselves on getting data which are not freely available in the public domain directly from National Met Services. As Anne T. Cyclone has said elsewhere many services charge for data and will not want to lose a revenue stream; it may be wrong but it is a fact. If, as a result of data being transferred by a mole, the CRU loses the confidence of data providers we will be left with only the sanitised Hansen data set.
Re: Ron Manley (#14),
Most of the revenue seems to be in short-term weather information. The main revenue from climate would seem to be carbon credits and other AGW rich-to-poor money transfer schemes, for which there is motivation to filter data through AGW-promoting organizations.
Re: AnonyMoose (#38),
“As Anne T. Cyclone has said elsewhere many services charge for data and will not want to lose a revenue stream; it may be wrong but it is a fact. ”
Agree, BUT:
The first i start to think of is the AGW “scientists” and “AGW-promoting organizations”.
And unfortunatley is Met office more of a “AGW-promoting organizations” than a public servant. Which explain their position.
Re: Ron Manley (#14),
Anne T. Cyclone has not revealed any specifics about the kinds of data that are charged for and whether the charging has been an issue for denying an FOI. Maybe she will spin something up soon but so far, no cigar. I do not like these myths that get started by a lack of details – like John V demolishing the Watts team efforts on CRN evaluations.
How could the data providers be concerned with confidentiality when CRU makes their data public – after some intermediate adjustments. Are you saying that the raw data is more valuable than the adjusted data?
I would summarize the situation at hand of a government agency withholding information as akin to the US government withholding presumably secret information and then invariably having it leaked to the NYT or the Washington Post. The newspapers then have a hand up in influencing policy and the government then looks stupid, stupid, stupid.
I am somewhat surprised that we do not have more defenders of the consensus here defending and excusing the withholding of data.
I am sorry but not surprised to read that the only way to get data from CRU is via an anonymous source within the organization.
I hope you were able to get the raw data along with value added product. Dissecting Jones’ methods would be interesting reading for the next few years.
“Re: Ron Manley (#14),
Oh really? So we expect the nations of the world to spend trillions of dollars fighting climate change when they won’t cooperate to measure temperature to see if it is happening?
Spare me, but that is so ridiculous.
At least now, knowing the locations of the sites used, one could make a map showing the coverage – much as GISS uses.
One could also to see exactly just how much overlap there is between the two; see which areas HADCru and GISS cover or miss.
#13 HAHA!! We can but hope!
The UK government is like my garden. No matter how many I kill the moles keep coming.
I’m really puzzled as to what you’re hoping to achieve. Given that the surface based data seems to fit pretty well with the satellite data what grounds are there for believing that there are any significant issues with the ground station data? Just read the notes and look at the final plots here: http://www.woodfortrees.org/notes
I’m also somewhat concerned about the contempt in which posters here hold international and commercial relations. Sometimes countries/companies will only agree to share information if it is guaranteed to be kept confidential. It is almost a certainty that this leak which you have publicised will be taken seriously because otherwise it might endanger future exchanges of information. In addition, you do realise that the UK Met. Office falls under the Ministry of Defence, don’t you?
Re: Nigel (#20),
I’m somewht cncerned about the contempt in which the warmists hold good scientific principles of openness and replicability.
Re: fFreddy (#32), Touché
Re: fFreddy (#32),
If the four major indices didn’t pretty well agree, then you might have a case for demanding openness at all cost. But as HADCRUT GISS, RSS and UAH show basically the same shape: see here: http://www.woodfortrees.org/notes under shifting the sources, I find this whole exercise a storm in a teacup.
Perhaps you don’t realise that some organisations sell their data to the public and businesses, but also make it freely available to others on an exchange basis. If they know that the UK organisations will release their confidential data simply on demand via a FOI request then they may be reluctant to share data with the UK in future. So there is nothing sinister in refusing the FOI request.
It is regrettable that not all information is freely available: I deplore that so many interesting publications based on publicly funded research are hidden behind paywalls.
Re: Nigel (#20), I’m really puzzled as to what you’re hoping to achieve.
He can look for suspicious post-processing.
Re: Nigel (#20),
The reasonable agreement of Hadley and GISS surface-station temp indices with satellite data is achieved at the cost of an abrupt change in index-series behavior ca. 1980. (This matter was broached on an earlier thread here at CA.) The question still outstanding is how these anomaly-sausage indices agree with intact records from the pitifully meager set of stations scattered unevenly throughout the globe that span the entire 20th century. That and longer time-frames are the heart of the “climate-change” issue.
Re: Nigel (#20),
If I am supposed to be concerned about “… between 1.3 and 1.7°C per century.” (from your woodfortrees link), then I think it is okay to be concerned about the .4°C difference between the indices; and why that difference might be there.
George Smiley would be pleased.
I am so reminded of a comedy sketch in the 60’s.
“Prime Minister to Parliament: In this country operates the finest Secret Service in the World (hear, hear from the assembled MP’s), Unfortunately, it belongs to the Russians. “
The true irony would be the AGW community’s response if Steve McIntyre’s analysis of the data does indeed show errors in the HADCRU3 processing.
I can already hear the howls of people dismissing the analysis because the data isn’t available to check.
Re: Charlie (#23),
A truly delightful situation !! 🙂
Great stuff!
In all seriousness, I would recommend making multiple copies of the data (stored in various locations) and any analysis work that you undertake as the UK Government were obviously very keen that you shouldn’t have it.
I didn’t know Richarad Armitage was working in the UK.
What the Met Office says is lunacy, pure fantasy of delusional minds. Do they imagine themselves to be the caretakers of vital information on which the entire security of the planet depends on? Weird minds over there?
Good for you Steve!
An old proverb goes: “Never tell secrets in front of servants”.
Great news indeed.
Hopefully, you and/or someone else you trust can use it to provide an “unbiased” estimate of historical annual temperatures. Or, perhaps, you may discover that the data is too fragmented to actually provide a “valid” estimate having the needed accuracy.
I’m really looking forward to your findings — as the historical raw temperature record is the most important data relevant to Climate Change hypothesis testing.
That the Met Office thought such “entrusted” data could be kept secret from the public is truly naive.
I do hope someday you’ll reveal the source of this entrusted cross-my-heart-hope-to-die data which no one is supposed to have. Can you give a hint? I promise I won’t tell anyone.
We’re definitely in the end-game on this.
The FOI requests flooding in will either produce the list of “sensitive” nations, or will break open the safe.
Steve already has the data, and can then directly identify what, if anything, is so sensitive about the sensitive data.
Having the data is simply a valuable shortcut. The long way around: work out all the knowns, “subtract” from Jones’ results, and you’re left with the “sensitive” data.
Either way, bright light is the best disinfectant.
Allen 63
ditto
Hi Steve
First time poster (I think, subscribe to lots of blogs :)) . I’m not surprised by this development, I’ve never seen ‘the truth’ reliably hidden. Data may or may not prove to be unreliable but this is bigger than the data, much bigger.
I’m very impressed with your determination and stats knowledge – keep going.
Regards
Michael
I’ve been following this John Le Carre adventure via your posts at Anthony’s. Congratulations. Always nice to see officious–in this case possibly dangerously officious–asses get their due.
If not the Four Horsemen of the Apocalypse, what if it starts raining frogs? Will that be blamed on global warming?
I have to agree with the second para here Nigel (#20).
The data is not secret. The data has commercial value to some organisations. It was presumably purchased by or provided free to the university. Peter Webster at Georgia Tech or a mole in UEA releasing the data to the world has made the data worthless. The organisatios providing this commercial data will I suspect be pretty hacked off. There only course of action that will restore value to the data is to no longer provide it to the leaky repository.
Consider this:
Mr.McIntyre had data relating to uranium deposits which he was selling to interested parties in order to pay his mortgage.
Being of national importance, it is provided to a university for future power generation planning research, on the basis that the reserve figure may be used but the detail is not released.
A university staff member – an ordinary human being, not a CIA operative, not a person who naturally likes keeping secrets; science should not be secret – agrees to the terms.
A mole within the university releases the document to the world at large.
Mr. McIntyres mortgage no longer gets paid.
.
1. Will Mr McIntyre provide further data to the university?
2. Will Mr. McIntyre be a happy bunny?
3. Will other souces of commercial data be happy providing the university with their data?
4. Has energy planning for the future been enhanced by this action?
.
I think, if you are honest, you will agree that the answer to these 4 is NO
I sincerely hope the data providers will not see the current state this way. If they decide not to provide data in future then CRU temperature data will have been compromised. Perhaps this is what is intended with this campaign?
.
Mike
Re: thefordprefect (#35),
Seems to be stuck in moderation!
Re: thefordprefect (#35),
Ok thefordperfect. here you go.
You don’t know this. Some of the data may have been supplied by military organizations or organizations
controlled by MODs. This data could be marked as “Secret.” in the past some meterological data has been
classified. In any case to prove your point please supply the data in thr form it was supplied to Jones so
we can test your claim.
Speculation. This is the point of the FOI. we want to see the agreements and the organizations. Some organizations
may be charging for data that they have no right to charge for. Further, some of this data may already be in the
public domain. If it is, then any contract people have signed to buy it or contract they sign to protect it may be
a sham contract.
This is an interesting claim. If such a claim were true we can expect the organizations who derive compensation
from the data would present SteveMc with a nice letter from their lawyer protecting their financial interests. That
is if it has commercial value we would expect them to excercise due dilgence in protecting their interests. If they
fail to do this, what does that say about the value of the data? there is much much more to say here. but for now
let’s see how you respond to this trap. be careful ford, I might know more than you.
One, this assumes that they actually exist. two this assumes that they actually have a contract that prevents
disclosure to “non academics.” Three this assumes they consider Mr MCIntyre to be a non academic. Four,
this disregards the fact that CRU has already given the data to non academics: MET. Met reports into MOD
which is decidely non academic. Who knows maybe the MET is feeding the data to the british air force? hehe.
There only course of action that will restore value to the data is to no longer provide it to the leaky repository.
We assume here that Mr. McIntyre has rights to this data. For example, he could have been granted access
to lands, say public lands, under the condition that he make his data freely available. So, I would have
to look at the WARRANTY Mr. McIntyre proffers. Does he warrant that he has the right to sell the data.
Or is he in violation of some prior agreement? Does his contract demand that I place his data under some
form of document control? Does it require me to report any inadvertant release of the data? What is the
period of the contract? that is, must I hold the data secret from now until I die? Survivorship? etc etc
Well, here too we would have to look at the university policies and the national policies WRT accepting data with restrictions. In some cases organizations are prevented from RECEIVING confidential data. This is why I FOIed
CRU and asked them for their policies and procedures for receiving and CONTROLLING confidential or controlled
data. Do they have a document control process? a document marking process? Employee training?
Does the staff member have the right to sign the agreement. Funny story, I once ran into a contract that was
signed by the owners son. Jones Jr. Nobody caught the fact that Jones Sr. had Jones Jr. sign the contract.
Jones Jr. was a minor. ( haha not a miner) Anyways, I FOIed CRU and asked for their proceedures. Who has
the right to put CRU under contractual obligations? I suspect if McIntyre gave data to a university he would
check whether the individual had the right to sign. Now, since Jones did some work with Hydrodata ( floods etc)
I suspect he had some aquaintence with the commercial value ( to insurance companies) of that data.
Re: steven mosher (#108), What a load of waffely twaddle!
If the data is “secret” then the scenario is worse!
“Speculation. ..we want to see the agreements and the organizations. Some organizations
may be charging for data that they have no right to charge for. “
You know that some of this data is not freely available.
I was going to go through your points one by one. But You have created a straw army of dross.
I await Mr. McIntyres report on his purloined data. It will be interesting also seeing what actions CRU/MOD/Georgia Tech/hadley take.
I really love your final demand
Don’t motive hunt
Have you actually read the comments on wuwt especially
Can it be determined from the data file how much the ‘official’ record is being altered?
I smell lots of rats here … cannot wait until analysis of the data can be completed. This is a best selling spy novel in the making!
people should be laughed out of the arena, laughed out of funding, laughed out of a job. A common shoplifter has more integrity
etc. etc
Re: thefordprefect (#110),
“Waffely twaddle?” I tried several times to utter those words without cracking up– couldn’t do it. Turn in your man suit.
Re: thefordprefect (#110),
You mean you’re perfectly happy to be told “It’s happening, we don’t need to show you any data. Just believe us.”?? How many bridges in Brooklyn do you own?
Re: thefordprefect (#110), Thanks for quoting me from WUWT (ie: “I smell lots of rats here … cannot wait until analysis of the data can be completed. This is a best selling spy novel in the making!”)
Please keep in mind, the “rats” I am smelling are those of the MET, and I am hoping that this “Spy Novel’s” closing chapters expose those rats and their misdoings. I just want to be sure that the small excerpt to my post from WUWT is not taken out of context.
Thanks!
Re: thefordprefect (#110),
You note comments on wuwt in discussing motives here; as a long-time poster, I would have thought you’d be aware that other sites don’t speak for Steve or this website.
Re: thefordprefect (#110),
Well ford point to the waffle. The points I raised are all points that you would see in any court case surrounding
the issue. Since CRU claim they are under legal obligations then the details matter. But, in you’re handwaving lets
let’s see what you got.
I know no such thing. I have not seen any evidence that people charge for the data that Jones
has. To know this I would have to see several things:
A.) the original data
B.) the data that jones has.
C. Records indicating that said data was in fact to someone.
Even here you have a problem. Let’s suppose that the WMO gives grants to countries to upgrade
there measurement systems. Lets suppose that the conditions of this grant is that the country make
the data freely available. Let’s further suppose that the country then sells the data. And suppose that
they give this data to Jones with commercial restrictions. I could very well know that the data is SOLD,
and still argue that CRU should release it. Why? because the seller has no right to sell it. In short.
I don’t know that the data is sold, and knowing it is sold would not be the whole story.
Hmm. I was kind enough to go through your poorly constructed hypothetical.
Purloined? That assumes facts not in evidence. As for actions.. What would you suggest we do to the mole?
Nice diversion. I try not to motive hunt myself. Whenever I do and people call me on it. I straighten
out my act. I don’t point at other people and say “he did it worser than me!” As to your question.
I scan the comments on WUWT. there are many people, skeptic cranks, who I just plain ignore.
I tend to look for the best arguments made by the most thoughtful people. I try to respond to them.
I will occasionally thump an idiot. ( usually with a youtube quip) I will read comments by Leif,
by EM smith with care. I read your comments with care. I read Jeff ID, ryan O, Bender, RomanM,
UC, Mr Pete, Lucia.. ( sorry if I left some out or offended some cranks)
Re: steven mosher (#123),
Mosh: speaking of youtube:
Re: steven mosher (#123), I could very well know that the data is SOLD, and still argue that CRU should release it. Why? because the seller has no right to sell it. In short. I don’t know that the data is sold, and knowing it is sold would not be the whole story.
Basically you are saying that the CRU and Met Office are lying For they have said it is commercial problems. (the met office have actually said they do not have the data (its now on their web site))
What would you suggest we do to the mole?
I would hope the mole is well down in his hole and does not loose his job or his university place be it in UK or US. But as I have said – If commercial interests force a head to roll, what should one do?
Mike
Re: thefordprefect (#133),
They have said no such thing. MET has said; “The Met Office are not party to information which would allow us to determine which countries and stations data can or cannot be released as records were not kept, or given to the Met Office, therefore we cannot release data where we have no authority to do so.”
In short, MET is saying, we are withholding the data because we have been told by CRU that the data has commercial
or industrial value, BUT the MET does not have the documents supporting this contention. So, I am NOT accusing
the MET of lying. The MET is reporting that it does not have the documents supporting it’s position. This is WHY
we did and FOI to CRU for the documents. basically, if 5% of the data is covered by agreements that would be
a good thing to know. That would let us raise funds to request data from those sources. that would let us petition
those sources. Now, Am I accusing CRU of lying? What did CRU say?
“Regulation 12(5)(f) applies because the information requested was received by the University on terms that prevent further transmission to non-academics”
Two problems here. First they represent the restriction as a restriction on release to a particular type of person:
“non academic”
1. they released the data to MET. MET is non academic. it is part of MOD.
2. SteveMc is arguably academic.
3. An academic who works in the commercial sector could have access to this data.
4. Several academics ( professors at universities like Webster) have requested the data without success.
So, I’m not accusing CRU of lying. To do that I would have to have the document in front of me and review that
document(s). I am saying that they are inconsistent. Maybe they don’t have the documents. Maybe they don’t
know how to read legal documents. Maybe they misspoke when they said non academic. maybe they meant
the data could not be used for commercial purposes ( that would be the USUAL use restriction in this type
of contract )
So, if you were boss of the mole what would you do?
Re: steven mosher (#135), I believe I was looking for temperatures local to Dijon (Dijon only goes back to 1958) for comparison to grape harvest dates when I first struck paywalls.
However, looking just now I could find no data! No time to continue but I did come across this rather excellent European source of all things climate:
http://eca.knmi.nl/download/millennium/downloadTemp.php
http://eca.knmi.nl/dailydata/index.php
Click to access ECAD_report_2008.pdf
Hunting around the site it seems good at first glance
Also some old CRU datasets are available from the waybackmachine. go first to the current page showing data/pages you want:
eg http://www.cru.uea.ac.uk/cru/data/temperature/
copy the URL and pase into the waybackmachine and you are presented with all the archived data they have.
e.g. http://web.archive.org/web/20010224010309/http://www.cru.uea.ac.uk/cru/data/temperature/
http://web.archive.org/web/20051210022317/www.cru.uea.ac.uk/ftpdata/
But mainly only the directory structure is shown from the second – useful for viewing personel changes perhaps Some data is available however.
The urls here are for a specific data. The root page of wayback allows other date access.
Re: thefordprefect (#144),
Thanks Mike.
[snip- by RomanM – OT political comment]
I predict legal action forthcoming. And this is still a large and unsatisfactory snarl. I’m curious if there really is any way for Steve and Anthony to “validate” this data in its entirety under current circumstances. I’m not saying its bad or incomplete data, but I don’t see any way to validate it in toto for accuracy and completeness against its source without some cooperation from Jones or Webster or the Met office.
If there are public algorithms and public results of applying those algorithms to this “secret” data, that one would be one way to get a check. But even that doesn’t strike me as totally conclusive — 2+2=4. . . but so does 1+3 and 6-2.
There was a Guy Burgess at Cambridge who once aroused my suspicions, but I did not think he was Met Office… Can’t remember.
Happy to look at verification/plausibility of a small subset of this country’s data. I’ve posted a few graphs lately showing how different agencies commonly produce different answers and would be interested in looking for a 100% match with one. I’m not worried about legalities because I’ve only your words on this blog referring to a mole and that might be quite a diversionary fiction.
Just happy. This is the Internet working to enhance intellectual freedom, as was its early purpose after Al invented it.
Consider yourself lucky. Burning at the stake is no longer fashionable.
I am shocked and appalled at the callousness and disregard for releasing sensitive information that Mr. McIntyre is showing here! The ramifications of this act could lead to serious national security issues and jeopardize the safety of the entire free world. What if the Chinese or a terrorist organization gets this info and uses it to hold us hostage to their evil designs on our temperature records? Water boarding is much to good for you Mr. McIntyre, I would prefer to see you sent directly to bed without your supper for this!
So the British invasion of Lund (Sweden) has been thwarted by your reckless attitude… 😉
The website at http://www.cru.uea.ac.uk/cru/research/grants.htm
lists recent and current grants that have been made to various scientists at CRU. It makes for interesting reading.
Prof. P.D. Jones appears on several grants that are from British public agencies. FOI requests could be made to these agencies for all data and letters or other communications that transmit data or discuss confidentiality of data.
One especially pops out: “Climate data and analysis for the study of natural variability and anthropogenic change” a grant of £99,785 with dates from 1 May 2007 30 to Apr 2008 from the US Dept of Energy. This would be a FOIA request and the DOE is very compliant with FOIA requests.
I do not believe that the DOE would take kindly to the concept of “confidential data.”
The mole shall be known as “Deep Cool.”
Re: John S. (#46),
Very good. Thinking over yours and a tentative “Deep Freeze,” I eventually set aside riffing on the infamous original, favoring either “Joe Cool” or “Mr. Freeze.” But I could live with Deep Cool.
I have been able to get a photo of the mole, which I have sent to Steve.
re 49
GCHQ will be tracking Steve and Anthony’s emails as we speak !!
Its worth noting that, as a prominent blogger, Steve is a journalist; and the mole is a whistle blower. As a member of the press, Steve’s purpose in examining this data is to investigate allegations, that have appeared in print, of government corruption related to the handling of this data. I’ve heard that there are certain rather fundamental legal protections afforded to both parties in such circumstances. I think its important to frame this properly for anyone so foolish as to go after Steve.
Re: mpaul (#51),
That is somewhat true in the US, but Steve is Canadian and the data came from the UK. I don’t know that such protections hold in either country. Better hope this record isn’t covered by the Official Secrets Act.
Re: Soronel Haetir (#59),
Alas, it is. The Met Office is part of the Ministry of Defence, and the OSA covers every last little chunk of information acquired as a result of employment. This mole has done a very brave thing, especially if (as I read it) the document was sent from an IP address within the organisation. If identified they risk suspension, prosecution and jail time (although the prosecution after at least one recent leak from the Foreign Office was dropped because there was no realistic probability of conviction).
Re: exMoD (#67), The data in question is from CRU at University of East Anglia. I would be surprised if this data is covered by the OSA.
I believe the gridded data was readily available from Met Office
Re: thefordprefect (#69),
Sorry, the way I read #6 was that the data had come from the Met Office (although it originated with UEA). If that’s not the case then the OSA probably doesn’t cover it. If it came via the Met Office, then it certainly will.
Either way, I think this act took a certain amount of courage.
Nigel and others who are worried about the release of this data. Will you agree with me that the person who supplied this data should be brought up on charges? will you? I most certainly will call for the person to be sacked. will you?
Now, what if the person pleads ignorance? I say we still sack them. will you?
WRT to SteveM. He is in receipt of data that may be covered by a confidentiality agreement. Is he obligated to keep it secret? Well, he’s asked for copies of the agreements. If CRU and MET had an interest in protecting the data, they now have an interest in telling steve exactly which portion of this data is covered by the agreements. They can’t just let him publish it without taking some action. If protecting this data was of such a great interest to them, THEN they should be contacting him and telling him which data exactly is covered?
But back to the mole. Do we all agree that this mole is a person we should never trust with data?
Re: steven mosher (#56), As I have said national security is not at risk. Most scientists HATE secrecy especially on something so trivial.
I do not think anyone has been defending secrecy. The majority have said it’s commercial, and and a commercial agreement has been made. Breaking this agreement puts the exchange of data at risk.
The mole is neither villain nor hero. He MAY be just foolish.
This data has already passed through McIntyre and Watts and ???? I think one can safely say any confidentiality/reciprocal agreement is finished. Hopefully the data flow to CRU will continue. But who knows?
It would not surprise me to find that your mole is Jones himself. I would hope no person suffers from this data release. But if someones head must roll to ensure free flow of data to CRU perhaps this will be necessary.
In the same way Peter Webster at Georgia Tech, who may be the mole, may have wrecked data exchange between his uni and UEA.
I suppose until the mole is revealed any data exchange between Peter Webster at Georgia Tech/CRU/and anyone else who have received this data will be at risk.
Let’s hope the leak was worthwhile
Re: thefordprefect (#68),
Thanks for taking the time to respond. Many people have made the unsubstantiated claim that the data has commercial value. Please note JONES himself has not made this claim. he has claimed, rather, that the data cannot be released to non academics. If there is a commercial agreement, then produce that agreement. It cannot be a secret about who charges for their data. Sales would suck if you did that.
Now you say that breaking this agreement puts the exchange of data at risk? Really? Let’s do a little thought
experiment you and me. Will you?
Let’s say, for sake of argument, that the Mole released the data a couple years ago. Under your hypothesis that
release would endanger future releases. Correct? So, If I could invent a time machine and go back in time
and release that data a couple years ago, we could test your hypothesis? Correct?
And, since this data has such a high commercial value you can expect that these agencies will be contacting
Steve and saying ” STOP” don’t release the monthly average temperature in Wherethefuckarewe!” especially the data
from 1910.. that shit was gold! man it was a 2C anomaly.. Here is an Idea, Maybe I go into the selling data
business. I will sell climate data from 1910 that shows it was really really cold. Any then maybe I’ll set up a competing data product that shows it was really really warm… And then I’ll be like an arms dealer!! selling data to both sides
of the climate wars… Like my old pal… http://en.wikipedia.org/wiki/Adnan_Khashoggi
Re: thefordprefect (#68),
Since CRU has demonstrated by its opaque response to scientific inquiry that it does not produce science, but advocacy instead, who cares if data continues to be delivered to them? I mean really, what do they produce that is of value, any value?
Re: thefordprefect (#68),
You seem to be stuck on what you think are the worst case scenarios, but that transparency is worth it (even though you say that the mole may be merely foolish??).
Best case scenario is that HadCru diminishes in importance because the methods used were not transparent and a valuable lesson is learned and further that the contributors see the errors of their confidentiality requests – if such agreements actually existed in contract form and were considered so important that the dog was allowed to eat them. We have one less temperature series from which climate scientists can pick and choose and forget to show sensitivity to a selection of the other series available.
Most likely scenario, in my view, is that what the mole revealed will not have any of the detrimental lasting effects mentioned here, not unlike the secrets that US government has until they are leaked to the NYT and WP.
Legal action is highly unlikely on a practical basis.
It’s bad PR for the Met Office; and it opens the door for depositions and discovery which would most likely result in further revelations that are bad PR for the Met.
Good stuff Steve. Now we’re all waiting on you to work your magic and tell us all what the heck The Met Office has been doing with these numbers.
The truth will out.
As the Water Rat said to the Mole:
“Believe me, my young friend, there is NOTHING–absolute nothing–half so much worth doing as simply messing about in R.”
Re: mpaul (#51),
I know nothing about things legal but as these data were passed on to Steve voluntarily by an unknown source I wouldn’t think that there was any problem. However the source may be batting on a sticky wicket so I agree that we should all leave him/her alone and not speculate to his/her identity. Let’s just thank him/her for his/her actions.
And how about once we’ve knocked the global warming nonsense on the head that we then take a shot at political correctness?
Michael Jennings– Which sensitive information has Steve released? Temps in Lund Sweden from a portion of the 1700s? Really?
Pete Webster was able to get this information. Card-carrying members of the Academic guild can apparently get it (as one FOI refusal stated). I can’t see that Steve has done anything irresponsible with this data yet, nor do I have any reason to assume he intends to do anything irresponsible with it.
Re: Geo (#63), (sigh) It was meant as sarcasm Geo. The absurdity of it should have been obvious but if not here is the winkie 😉
Are none of these defenders of secrecy ready to seek some blood vengence on the Mole? Every single poster who defended CRUs claim to secrecy now owes us their opinion on the mole. Hero? or Cheat? Anthony has his
picture. We should probably post it with one of those little black bars across the eyes.
Nigel & co.
there is no moral justification for keeping secret data that very significant, economy destroying decisions are being based upon. There are excuses and that is what the UK Team are making.
Their data and data operations may be totally on the up-and-up, but they fear the public laundry washing that Steve has done to GISS.
“Now, hand over the information … or your DATA gets it”
I don’t immediately see what commercial, political or military advantages would be compromised by releasing raw site data. But for the sake of argument, let’s suppose there is such an advantage. The difference between the raw and processed “gridded data” is as follows. By virtue of its construction by means of interpolations and adjustments, one cannot necessarily reconstruct specific site data from gridded data. The gridded data may be a function of the raw data but that function may not have a unique inverse. Think about the fact that if you had k moments of a sample of size N>k, you couldn’t necessarily uniquely reconstruct the sample from the moments. Thus for instance, a sample mean protects the confidentiality of the individual observations.
In the U.S., individual-level census data is very carefully protected by the government. You can get sample moments from census data down to a pretty fine level, but you cannot usuaully work with the raw data. If you are an academic, you can apply for a permit to go to a limited number of specific locations and work with the raw, individual data in a special facility, but you cannot carry the raw data out and you cannot publish any collection of statistics which would permit individual observations to be recovered. It is even difficult to tell what has really happened over time to the U.S. income distribution because of top-coding of the relatively few gigantic observations in the upper tail of the distribution. The top-coding is deliberate: It prevents individual observations from being potentially known, or guessed at and linked to specific individuals.
I just mention this to explain the difference between releasing the gridded data (which is composed from statistics) and the actual site data… Not to argue in favor of keeping the actual site data secret.
Don’t forget folks we still need to complete the FOI. It is vitally important that we get the data through the FOI process.
1. To protect the Mole if you feel so inclined
2. To demonstrate that the public has an interest in this data and a right to it.
3. Finish what you start.
See this post, and join in. we only need a few more volunteers
http://www.climateaudit.org/?p=6623#comment-350536
Forget the Info Commissioner as yet – I will keep that in reserve!
Did I hear right, there’s a mohel at the MET?
Oy!
Fordperfect,
Suppose that I authored a study showing that the past was warmer than the present. Suppose I claimed
that the data could not be shared because of an exclusive confidentiality agreement. What would you
say about my practice of using data that could not be shared? Would you be suspect of me and my motives?
would you say that I could produce science ( results that could be replicated) with that data? Let’s face it
If a “cooler” produced results that put AGW into question and if they used data that could not be shared,
all of us would be rightly suspect. Transparency is a core value in science. by accepting data with restrictions
Jones took his results out of the realm of science.
Re: steven mosher (#73),
I agree that transparency is a core value of science, but as with many methodological dicta this one isn’t always part of practice. People in my field tend to be very, very good about sharing data sets. Some of us have found, though, that neighboring disciplines aren’t so open. We have had great difficulty getting some neuroscientists (some of those doing social neuroscience…not all of them mind you) to cough up unprocessed raw data so that we can do our own statistical analysis on it. This is especially worrisome since we suspect them of “MacGyver Econometrics” (as one of my pals entertainingly put it…see http://pluto.ecom.unimelb.edu.au/seminars/app/UploadedDocs/Doc109.pdf).
This may seem unconnected but in fact climate science and social neuroscience share a lot. They build big ambitious models of very complex systems (climate, social brains) that are just beginning to be understood (maybe); they are small fields with dense co-authorship networks; and they generate lots of press and grant money. It is no surprise that they would be sociologically similar in certain ways, for instance in creating incentives for some to jealously guard data.
My main point is just that similar behavior crops up in other areas of science. It isn’t any prettier elsewhere, but it isn’t wholly anomalous either.
Re: steven mosher (#73),
I await Mr McIntyre to reply to this:
thefordprefect (#35)
or if you like you can give me your views
Re: thefordprefect (#92),
thefordperfect. First let me say that I appreciate the work you and your teammates did on clearclimatecode.org.
As an advocate of Open Source I applaud your efforts. It was also nice to see as I spent many months pestering Gavin to release the code. My stance on these matters is one of principle. Addressing #35? I will give you my views. And will you please FOI CRU to release the confidential agreements? Not the data, just the agreements. You cannot argue that we should not see these agreements. After all, if they don’t exist your whole position vanishes.
Re: thefordprefect (#91), you imagine the information has commercial value and that agreements were made; but neither Jones nor anyone else involved in this issue knows to whom such agreements were made or about what data any agreements were made. If it’s so important to respect the agreements, why is it that no records were kept of the agreements? It looks more like a grasping at straws to deny the request for information.
Re: thefordprefect (#91),
The hypothetical case you describe does not involve data that was collected and archived by a publicly-funded agency so it is not comparable.
So, what does a 1753 era thermometer look like and what is the margin of error of such a device? Just curious.
Re: Dogwood (#74),
not much different from a modern analog thermometer
http://en.wikipedia.org/wiki/Daniel_Gabriel_Fahrenheit
Re: Hans Erren (#101), Thanks for the link. I’m thinking a hand-made thermometer with a hand-painted temperature scale may have a different margin of error than a modern, machine-made thermometer. Just a guess. Would be fun to compare the two in a modern experiment, though.
Re: Dogwood (#119),
The margin of error of old hand made thermometers might be less than modern electronic ones. At least the old hand made ones didn’t suffer from drift.
Re: Phil (#138),
Considering that the old-fashioned thermometers were read by (often old) eye, and recorded on paper, I wonder if their error bars are actually much larger, tho.
Re: Dogwood (#77),
Might be a bit difficult to quantify the margins. That’s why they prefer tree-rings 😉
Re: tty (#186),
Remarkably similar to Jones90 version ( http://cdiac.ornl.gov/ftp/ndp020/ ) :
I’ll relate a little anecdote about the Official Secrets Act in the UK. Almost 40 years ago when I started working in the NHS (National Health Service) I had to sign the OSA because one of my duties as a very lowly official was to keep records of the Civil Defence stocks held at hospitals in our Region. This was sensitive stuff like the number of aspirin tablets/ crepe bandages etc.
Obviously I now can’t remember the individual details for each hospital but I believe, although I now longer work for the NHS, I am still covered by that signing and if I did provide any specific information I could still be open to prosecution under the Act!
The OSA is an open ended document , once you have signed it , you can’t get out of it. I signed it first in 1977 , closely followed by similar documents from the US, Saudi Arabia , Sultanate of Oman and sundry other countries. But, the courts CAN overrule it.
Nigel,
If Phil Jones had said from the start that the data was subject to confidentiality and commercial clauses you might have a small leg to stand on. But his initial reaction was that he didn’t want to share 30 years worth of work with people who might be looking for errors in that work. No mention of the previous reasons at all.
Re the sensitiveness of those old Swedish records. They might be more controversial than you think. I don’t know if the Uppsala series is in the data, but it seems likely. It starts in 1722 and is one of the longest continuous records in existence.
The warmest year in that series is 1723 and of the ten warmest years 5 are in the 18th century and 5 in the 20th.
Re: tty (#82),
Would you kindly provide a link for downloading the entire Uppsala series.
Proprietary data integrated into scientific results used globally to set trillions of dollars worth of public policy? Are you serious?
All this secrecy smacks much like the Medieval Catholic Church at the time when Tyndale translated the scriptures into English so the mob could work out for themselves what the Church heirarchy were on about.
Tyndale didn’t die a normal death either.
These are very interesting times.
‘If they decide not to provide data in future then CRU temperature data will have been compromised.’
Given the tender nerves about the impeding end of Mankind, I suspect that any state/organization that took its ball and went home would lose more value in public face than it would gain from selling temperature records.
The very fact that “Deep Cool” thought that it was important enough to take some personal risk to see that SM got a copy of the data is interesting in itself. It is far too early, however, to speculate on just what an analysis will reveal. It may not reveal what a lot of us are hoping for…. suppose Steve reports back that Dr. Jones has done a fine piece of work? Let him do the work and let the chips fall where they may.
“Deep Cool” HAS taken on a substantial amount of risk. Let’s not add to his exposure. There is also a lot of food for thought on ethics and responsibility here. Lord knows what I would have done in his shoes.
On the topic of the OSA, my understanding is that unlike US law, the UK allows prosecution of people who come to possess and distribute such information, regardless of whether they agreed to keep it secret.
In the US a leaker could be prosecuted, which the whole point of the Valerie Plame investigation, regardless of the results, but the reporters were immune from prosecution for passing it on. My understanding is that such is not the case in the UK.
re 78. jeez nails it.
By providing Open access to the CRU data, stevemc and others are ADDING VALUE. Through their efforts it might get transformed into science. And jones should thank them
IMO, already far too much information has been given about the source of this data. Do not underestimate the resources of governments to track internet communications.
All identifying files or records should be securely and strongly encrypted or thoroughly destroyed – the latter being a non-trivial exercise.
The argument in favor of confidentiality to protect commercial (or some other) interests if there were clear evidence of such agreements and interests.
Read the first quoted excerpt in this previous post:
http://www.climateaudit.org/?p=6618
Some of the data was provided to Jones under some sort of confidentiality agreement–but no one knows which data, because records were not kept. Does that sound like an important interest that is being protected by Jones or anyone else? Not to me–if it was important, he would know to whom he made such a promise as a condition of getting the data.
Oh well, never backspace and edit without careful proofreading–I’ve said that to myself often, but I won’t listen!
I meant to say: “The argument in favor of confidentiality to protect commercial (or some other) interests would be more persuasive if there were clear evidence….”
Can someone point to the language in the FOI denials that leads people to believe that the the Ministry of Defense is claiming that the data is a state secret? All I see them saying is that they are trying to protect confidential agreements and commercial interests that Phil Jones claim exist. I think this “state secret” notion got started in a tongue and cheek fashion, but seems to have morphed into a unsupported belief as people piled onto the comment.
If there is an upside in this for Prof Jones, he can now honestly claim that his data has been “peer reviewed”
Don’t know how he did it before.
The official secrets act was changed in 1989. It now only applies to intelligence, defence, international diplomacy, serious crime, and equivalent material obtained from other nations under high level information sharing agreements. Stuff like that. Not the weather.
If the British were forbidden to talk about the weather…, why, it would be unthinkable.
I never seen no data. Or anything else.
.
BTW here’s a picture of me on holiday romping around in Kenneth Grahame’s grave. Plenty of nice fat well-fed worms. Yum Yum YUMMY.
[deleted by request of photo copyright holder]
I think there is something strange about this whole affair. Could it be that CRU/Jones are trying to extricate themselves from an impossible situation?
It seems that “the Mole” has used an extremely and unnecessarily risky way to leak the data. Now if I wanted to leak something I would have copied the data to a CD or (preferably) an USB stick, put this into an envelope addressed to SM, and dropped the envelope in a letterbox. A little slower but essentially untraceable.
Re: tty (#103),
Not easy to do in MoD land*
CD writers and USB’s are/should be verbotten. Sometimes extra measures like epoxy filling USB ports are taken. Sometimes that creates problems plugging keyboards and mice in. Oops.
*Thats the theory, in practice folks in the UK know that plenty of sensitive data gets lost, left in pubs or on trains, misplaced by subcontractors etc etc and often with minimal or no protection. But given the amount of the MoD’s budget that’s been spent furnishing their new palace, there’s no money left to pay for drive or device encryption it seems.
The reluctance to officially publish the data requested here smacks to me of commercial interests rather than security. Once all the carbon regulations kick in fully, organisations like the UEA probably expected to spin off commercial ventures to sell product and services to the victims.
Thanks for the snippet of Lund data. Here’s what I’ve gleaned from it – not anything like a security threat I imagine, but just intriguing.
By removing the average for each of the months from the observations I obtain what I call a Monthly Difference (MDif). Examining this carefully reveals that the data appear to fall into four regimes which are from 1793 to the end of 1795, 1796 to the end of 1802, thence to mid 1812 and finally to the end of the data (end 1813). This classification provided the following statistics:-
Classification of MDif by Regime with 4 Classes and 252 Items in total
Regime Mean MDif Std Dev No. Percent Std error
Class in Class of total of mean
0-(1) -0.131647 1.70874 48 19.048 0.24663
1-(2) 0.306548 1.78096 72 28.571 0.209888
2-(3) -0.332653 1.64386 112 44.444 0.15533
3-(3) 1.07524 1.27483 20 7.937 0.285061
Sorry about the formatting. How does one ensure that spaces are present in the published version when required?
Within each regime there are no significant trends and each closely conforms to a simple linear model. The table shows that Regime 3 has a level about 1.4 degress above Reg 2 (and 0 and 1, for that matter). Would the citizenry have noticed this? I doubt it.
Here’s the ANOVA table:-
ANOVA for MDif (mean value 5.91316E-11) by Regime
SOURCE Sum of Sqs. DF Mean square
Between classes 43.1143 3 14.3714
Within classes 693.261 248 2.79541
Standard Deviation of MDif within Regime = 1.67195
F ratio (Between/Within classes) 5.141, Probability 1.823E-3
This would normally be regarded as fairly convincing evidence that there were differences amongst the regimes. I can’t post graphics – sorry – but these show convincingly that the regimes differ visually in their mean values.
Now it would be nice to look at the full data set!
Robin
Well done, Steve,
When scientists, however well-meaning, withhold data on issues that have major socio-economic implications, they are adopting a political stance. That is very clear.
When a scientist makes elusive data available to those who may have a different approach to processing, he is following the honest scientific method.
We have been treated like mushrooms for far too long in the climate debate. Thank you HADMOLE. Thank you Steve, Thank you Anthony, Thank you Ian (Plimer), etc., etc.
Jack Linard,
Quebec, Canada
I thoroughly enjoy what Mr. McIntyre does here, am pleased he has this data, think the excuses offered by the Met and CRU are probably bogus, so was delighted to help out with the mass FOI, and so on.
Having said that, here are some thoughts about the range of things we might mean when we say a scientist has some sort of “openness responsibility.”
First, there are two ways that “replication” gets used in science. One is appropriate to experimental inquiry and the other to observational inquiry. In experimental inquiry, experimenter Alan has satisfied replicability requirements NOT by sharing his data with you, but rather by telling you enough about his methods so that you can do his experiment yourself. If Alan additionally shares his data, he has done something more than is required for you to attempt to replicate his findings (in the experimenter’s sense of replication). As I said above, some experimental fields have social norms that strongly encourage sharing data (experimental economics) and I think that is best, but others do not have a strong norm about that (experimental psychology). What all experimental fields share is the demand that methods (experimental and statistical) be described in enough detail for an essentially identical NEW experiment to be done in someone else’s lab–NOT that the data be shared.
In an observational field, replication certainly could mean something very much like the experimenter’s meaning. If observational scientist Bob rounded up data from eleventy-zillion governments, and he tells you which governments and enough about what you need to ask for in order for you to do it yourself, then there is a sense in which Bob has satisfied the data-collection end of being open enough for you to replicate what he did. True, it would be grossly inefficient if Bob didn’t just share his data set. But he was certainly open enough for you to recreate it yourself. True, many granting authorities wouldn’t give Bob the money to do this unless he agreed to make the resulting data set public. And true, many journals would demand that Bob deposit his data publicly as a condition for publication. But not all granting authorities and journals would do that.
The real sin, if there is one, is if Bob doesn’t tell you enough to get from his data to his results. My understanding of the real crux of the issue with the CRU data is that this particular kind of openness is absent–no one really knows what Dr. Jones has done in going from the raw data to the gridded data. So even if you recreated Jones’ data set from scratch, you couldn’t reconstruct his gridded data or critique its construction very well, unless you were extremely lucky, and it would take a lot of pointless search.
I like Mr. McIntyre’s choice of the word “audit” to describe what he does because it doesn’t lend itself to confusion with the word “replication.” It also describes what he wants to do very well: Take the data and the alleged statistical procedures and see whether he can produce the same results; and/or critique the statistical procedures. But the critical thing is having alleged procedures in hand. Not getting the data is a pain in the butt, but not being told what Bob or Jones did with it is the really important sin, I think. And, Mr. McIntyre still doesn’t have that.
None of this is meant to excuse these particular people for not sharing this particular data. I agree with what many of you say, that given the social costs of type I and type II error here, society simply cannot afford secrecy here. But it is good to be clear about the strictly scientific general norms here, as opposed to the situation-specific matters. What I wrote above is just a stab at thinking about this and I would be interested what others think.
What’s Sandy Burger up to these days?
Hello. It’s me again. I’m getting a taste for this adventure thing now.
I think I’ll go over and dig up James Hansen’s garden, undermine his house’s foundations and drop my lunch down his chimney, after I’ve digested it.
Then I’m gonna stick a big worm in Gavin Schmidt-rhymes-with-….’s Sinclair ZX80. No more prancing-toady Al Gore Rythms for him.
Finally, I’m going retire to Al Bore-the-Toad’s golf club and spend the rest of my days implementing my grand green policy — digging his greens full of extra holes, tossing dirt all over them and clawing his balls every chance I get.
I’ll send a graph on a postcard next year to show my progress. I’ll do it in R.
If this information was indeed sensitive, it would already have been published in the New York Times.
Cru has their best agent as we speak hunting down the mole.
Just a guess: but I’m thinking after reading the comments here and at WUWT that someone in a position of authority determined that this data should be made available. And that exposure for releasing the data will not professionally endanger him, although he still wishes to remain anonymous.
As I said, just a guess.
I would also like to make note on a personal observation. I have read lots here about this data being “paid for on contract” and various other such permutations. Is it not a fact that the dollars that paid for said data, in fact, come from public sources (taxes, etc..)? ie: aren’t these data in fact then owned by the public? I usually get rather upset when I pay for something and don’t receive it. Should not this be the same case? In other words, the mole here has simply distributed to “the people” that which rightfully already belongs to “the people”. I also fail to see where this would have any harm, whatsoever, to the continued exchange of this data. If in fact, someone is making money selling this data to the MET, CRU, or whoever, do you honestly think they are suddenly going to say “oh, we don’t want to make money anymore, its no fun now that you have distributed the data”? Did accepting this information from the mole, suddenly open some broad competition for the gathering and distribution of such data? I think not, as there clearly are A.) few people gathering the data, and B.) few people consuming that data. Sounds like a rather closed system to me. I see absolutely no harm done to the MET or CRU from this action, however, I clearly seen the harm that has and is being done by NOT divulging this information thus far.
I for one, hope that A.) useful knowledge is gained by these actions (to either side of the AGW debate), and B.) that we NEVER learn the real identity of this mole and that he/she is protected of this action at all costs.
To quote from one of my favorite films (Sneakers).. “Setec Astronomy” = “Too Many Secrets”
Jack’s getting more miles from that “The Shinning” character.
I hope “The Mole” approved of Steve in making public his or her generosity.
It sure would have been nicer if Mr Jones had treated his data collection as one for all, and not all for one.
Re: pete m (#124),
The Mole must have known it would go public here in ClimateAudit. We all know full well what ClimateAudit is. Isn’t this what Steve McIntyre is pushing for all the time? The Mole may have wanted this.
Re: Gene Nemetz (#131), there are obviously ways of stating you have certain information without revealing where your source was located. That’s all. Not having a go at Steve – just interested in whether the source was concerned about their employer being revealed.
Jack’s getting more miles from that “The Shinning” character.Re: Steve McIntyre (#3),
I don’t think there’s a way to stop them from finding out who it is on their own. But maybe he doesn’t mind it. Maybe he wants the whole thing broken open.
I’m saying maybe.
We are down to the last 30 or so countries. So, we need a few more volunteers to write FOI to CRU.
This FOI is to request the confidentiality agreements that folks believe have been violated. Perhaps
if we get a look at these documents the data can remain safeguarded. For example, if thefordperfect
took his own arguments seriously he would FOI CRU for the agreements to bolster his argument
that the data is IN FACT covered by confidentiality agreements. But I suspect that ford doesnt take
his own argument seriously.
Re: steven mosher (#126),
I think without a law suit that the FOIs will get the same tired responses that other such requests have.
The Mole (and the Leaker) is definitely the way to go and for that success, Steve M and the Mole deserve my congratulations.
Get over it thefordprefect. You are begining to sound like the US government when its “secrets” are published in the NYT and WP – you know, the opposite of smart.
Re: steven mosher (#126), But I suspect that ford doesnt take
his own argument seriously.
My comments are what I think is true, from trying to get a few long temperature series from various countries in the past.
I do not wish to join your campaign
August 2009 issue, Scientific American :
This news about the Mole seems seems more weighty than “routine corrections”, doesn’t it?
It would actually be MI-6 who would knock on your door, Steve. MI-5 would ferret out the mole in the UK while MI-6 would operate covertly in Canada.
Or the CRU folks might just ask for an injunction in a Canadian court. You may want to give copies to Anthony and other trusted people outside of Canada to escape the jurisdiction of the Canadian courts.
Can someone help put this data into perspective for me? I understand that this data turns into gridded temperatures, which are combined to produce a “global average.” Is this “THE” data that is used by the IPCC when it says that temperatures are “unprecedented” and so on? Does the IPCC rely on any other data sets for global temperatures? In other words, if this data is relegated to the trash bin the way MM dispatched MBH98’s hockeystick (I know it keeps popping up), does the IPCC have another leg to stand on? “Well of course it has been known for some time that the CRU data wasn’t robust, which is why we haven’t been using that data for some time. That Canadian fellow who found some flaws, which we already knew about, was wasting his time. The URC data is where we have been putting our efforts. Pity that information is confidential.”
This information is extremely commercially sensitive. Here in NZ they are about to bring in Climate change policies that will cost each of us $3,000 a year. Wouldn’t want to lose that income would they?
We need to demand a full accounting of the “facts” surrounding Global warming, Release all data and make all journal articles free to everyone so that we may see for our selves the incestuous relationship that the major proponents have with each other. Most if no all research is supported by public dollars, the idea that the temperature data or any other data should be kept secret is utter nonsense. It is to important to leave it to a few select scientists and their lackeys. If we do not, the eco nuts will destroy our world as we know it… They are going to drive us in to a new dark ages Where only the elite have anything, because as we all know they are connected to their god Gaia
This whole process of reliably working out where the data came from and who sent it is fairly tricky. I could easily send an email with any address you care to name. As an outsider to an organisation you can trace to the gateway but not inside. In government organisations copies are kept of all electronic mail. I would say at the moment there might be an investigation in progress to see who sent it. My expectation would be they will find who sent Steve the email if in fact what appears too happened has. So in that event I am sorry for the party concerned because that does break the law and has consequences. Personally I would have got the data out some other way and sent it anonymously.
re 130. if you don’t want to join ford, I won’t bug you anymore. But what long series did you try to get? That might be an interesting case to investigate. I don’t have any issue backing or helping in an effort to get the data you wanted.
I love that stuff.
Looks like you’ve just made the front page of Slashdot. Hang on tight – it could be a bumpy ride.
Just for info:
one of the earliest data sets I found:
http://web.archive.org/web/20000411160929/www.cru.uea.ac.uk/cru/data/temperat.htm
Before anybody comes to any conclusions it would be much better to let SM or however analyze the data. Who knows it may correct or it may not be.
Dear Mr. McIntyre,
Please send me a copy of the CRU data.
Yours Sincerely,
Patrick M.
p.s. Don’t give me any lame excuses about not being able to release the data due to the confidential manner in which you obtained it. No really, I’m serious.
from the 3rd eca link above:
Click to access ECAD_report_2008.pdf
So it seems that half the european data was not available becuse of restrictions by the supplier.
Will you now be looking for a mole at KNMI?
Mike
PS Re: steven mosher (#123),
Thanks for that! However not recently, since I was banned by WUWT!!!!!
Re: thefordprefect (#148),
Ford that is a great document. I’ll make several points.
1. The data coverage, on first glance, appears to be more substantial than Jones.
2. Only 42% of the data is available to the public. the rest is used for Indices and gridding.
Based on the other things I read in the document i would say this. The data that is not posted
on the internet is most likely DAILY temps. Indices ( say monthly, or yearly or anomaly… ) You’ll find the same situation in the US. where for some stations you can get daily for pay and monthly for free.
It would be interesting ( if jones would share his code) to see just how important the closed data is. Also, since this european temperature series has been sited in several papers it would be interesting to match it up against
Jones and Hansen.
Since Slashdot was mentioned on this thread, I thought I’d post a snipped from the best post I’ve seen on the matter.
Quote from Richard Feynman (Nobel Prize in Physics 1965. If you don’t know who he is, read about him.)
“The point of all this is that real science is all about trying to prove your theory wrong. You do everything you can to prove it wrong, then have other people do what they can to prove it wrong. When all of you fail at doing that, when the theory has been refined such that it fits all the evidence and you can’t figure out how else to test it, then it is most likely the truth. THAT is what scientific rigor is about. It isn’t about coming up with a theory, ignoring data you don’t like, showing it to a few people who agree with you, and saying “Ok, we proved this true and nobody else can look at it.””
Re: Joeshill (#149), Surely you are joking…
Re: Jaye Bass (#153), I didn’t get the reference. (But you did.)
Upon researching the quote above, it is from Feynman’s commencement address to Caltech in 1974, where he discusses what he called Cargo Cult Science. It is reproduced in the book:
Surely You’re Joking, Mr. Feynman!: Adventures of a Curious Character, Richard Feynman, Ralph Leighton (contributor), Edward Hutchings (editor), 1985, W W Norton, ISBN 0-393-01921-7, 1997 paperback: ISBN 0-393-31604-1, 2002 Blackstone Audiobooks unabridged audio cassette: ISBN 0-7861-2218-8
Jaye, Thanks for pointing me to that.
Re: Joeshill (#156),
A fantastic book. Highly recommend it.
Paul the exRocket Scientist says
[SNIP – by RomanM – OT Inappropriate content for this blog]
Re: paul exRocket Scientist (#150), I guess we don’t need to wonder why you are an ex-rocket scientist.
I reckon that Steve is having a lot of fun with this one by sending everyone off on a wild goose chase.
I have read Climateaudit pretty much since it started and I’m sure that he is an honest, principled person who would not even hint that he’d received something via a mole let alone shout it to the world by titling a post “A Mole” and risking that person’s career.
In his post he does not refer to any individual, even obliquely.
“I’m now in possession of a CRU version…” and “…just because I’ve got a version of the data…” do not imply that he was sent the information by someone, per se.
Therefore, I’m going to put forward the theory that the Met Office has somehow managed to leave the data on a publicly accessible ftp data server (or equivalent), perhaps in the equivalent of Mann’s “censored” directory, and Steve has sourced it from there.
Re: Jack Lacton (#152),
Steve, out there with pick and shovel, prospectin’ them thar hills and dales.
========================================
[snip: tinfoil hat – ryanm]
This isn’t April 1st, is it?
[snip : way off topic – ryanm]
As an earth science data manager in the US, I know all too well the restrictions placed on data products that are gathered by private parties. If there is money to be made, the data are held tightly. We are fortunate that scientists can access these data at all. The more data that we have, the more reliable the results. The statements from the UK Met Office ring true. Their hands are tied. This issue needs to be addressed at a higher level.
Scientists, by nature, are very open in their research. It’s this openness that allows scientific understanding to thrive. (If only bio-medical research was more open.) We would prefer that all data are open so we can do the best science possible. Let’s promote “open source” data (and software, but that’s another issue) in general. We will all benefit.
Re: ZenGeek (#158),
Thanks Zen. You will find many advocates of Open source, open access, and open science here. Have you considered GPL licenses ( or some suitable copyleft license) for your project. One thing we saw in the Mann and Steig cases
was this. They took source that was in the public domain ( RegEM) modified it and then refused to release the
derived work. The same could happen to data. Where your raw data is put in the public domain. Somebody
takes that data, “adjusts” it. Creates an “index” and then puts that index behind paywalls. So, you might
want to have a look at Copyleft which would force them to copy back modifications. We are OT, here however.
I guess I will emphasize the need for open source software. A great deal of analysis goes into preparing data for scientific use. A key issue is calibrating the data from different instruments. The software used to compute the data products is a critical part of understanding the results. My project, BTW, will release all the raw data and all the software required to regenerate the results.
The raw data alone are not likely useful towards understanding the problem. However, these calibrated products, derived from proprietary data, are more useful and more likely to be openly available.
This is the kind of openness that you should seek. If you appear to be after only evidence for your theory, then you are no better than those that you accuse.
Re: ZenGeek (#159), You speak good words!
The problem with blogs is that that many read the posts.
This, wuwt, and many others criticise others (rightly sometimes) for mistakes. But instead of an interchange of critical ideas were mistakes can be quietly corrected with little loss of face, the world thinks Dr.Joe Bloggs got the concept screwed and is obviously an incompetant whose research is worthless. Bloggs cannot take this lying down and must defend his reputation against this public criticism.
Splashing you criticism across a blog does not make for good science.
But just like splashing We Have a Mole across the blog it does lead to more readers.
Re: thefordprefect (#160),
Apologies to Steve. I was speaking to the general “you,” inspired by a previous poster that clearly has a strong bias. That is exactly what the world needs – complete openness to promote progress and so the public can verify the results. More eyes means fewer bugs. (If we could just do something about Microsoft. Now, that is a completely different problem.)
Re: thefordprefect (#160),
As an honest questions:
What about the new practice of “Science by Press Release”? That is scicne in which the paper is full of caveats and notes that only tentative results are possible but in which the press rlease is full of hyperbole?
Why are sceintific papers accompanied by press releases?
What about peer reviewers who give only the most cursory reviews and do not check the data?
What about IPCC authors who use their chapters as a pale to advocate tehir own postions and dismiss peremtorilly revew comments?
Waht of the new practice of calling people “denialists”?
Re: thefordprefect (#160), I have seen many times that Steve begins with private, offline requests for data. If he finds problems with their work he approaches the authors offline for clarification. A number of cases have been cleared up with no public discussion, except maybe a note of commendation from Steve. The kind of genteel give-and-take you want to see only works when authors are forthcoming about their data and methods. The battles take place when authors try to stonewall or block replication. The high profile battle now unfolding for CRU data is the end result of a process of refusal that goes back many years. I was there at the NAS panel meetings in 2006 when von Storch publicly rebuked Jones (in absentia) for his refusal to release his data. Without the underlying data it’s impossible to replicate the gridded and global series, it’s impossible to check for errors, and most importantly, it’s impossible to examine how robust these products are to minor variations in methodological assumptions.
.
Data secrecy is not a problem unique to climatology. Bruce McCullough and I documented cases from across academia, chiefly from economics, in which authors refused to disclose data, often for studies that had important policy consequences. We can either wish the problem away or confront it.
Re: thefordprefect (#160),
I should add that, in engineering, patents can be used to protect the rights of the inventors/researchers. Are requirement of patenting is that the patent be ‘enabling.’ That is the patent must contain enough detail that a person of ‘ordinary skill in the art’ can reproduce it with only the patent as a guide. In addition to this, the patent must disclose the ‘best’ way of implementing the invention. If either of these is not done, the patent can be declared, by the courts to be ‘invalid’ and rendered non-enforceable.
The inventor cannot keep any data or methods confidential. If the inventor does the patent is invalid and worthless. The standard of “ordinary skill in the art’ is quite low and the patent is required to be very detailed and specific and not require any form of novel thinking on the part of the reader – no “proof is left for the reader”
An inventor is free to attempt to maintain the secrecy of their invention. However if someone else derives the same invention then they are free to practice it.
This is called the ‘patent deal.’ If an inventor agrees to publish their invention then and allow others to benefit from their research tehn they will be granted protection and the courts will recognize and enforce their rights to it. if they try to keep it secret then they are on their own. Perhaps this should be the same way in science. Someone is free to derive their own results by secret methods and data. However to be afforded recognition and publication, they must make their paper describing their findings “enabling” to someone of ordinary scientific skills. After all this is just the principle of replication.
Why should he give you the data when all you want to do is find something right with it?
Re: theduke (#161),
Steve has yet to respond to my request. If I have to I will send him a FOLI, (Freedom of Leaked Information), request.
Re: Patrick M. (#171),
From the fact that you see other people snipping posts you might surmise that steve is offline.
Re: steven mosher (#183),
I gathered that. 😉
Re: Patrick M. (#171),
But are you an academic? or a pandemic?
Re: Gerald Machnee (#188),
Well if Steve pulls that kind of shenanigans then I’ll put in a FOLI request for any agreements he has entered into upon receiving the data.
On the other hand, Steve could always leak the data to me through an anonymous party…
Re: theduke (#161),
Most amusing, you pair.
Geek:
Steve McIntyre has always made it clear the he does not have a theory. He wants to audit the data and techniques that are being used.
I believe (my words, not his) that he wants to validate (or invalidate, as the case may be) that there is adequate rigor in the literature, because there will be a significant effects as the literature drives decision making.
It’s astonishing how many people find this concept too difficult to comprehend.
“mistakes can be quietly corrected with little loss of face”
1. Who cares? Seriously. Everybody’s personal problems are not necessarily mine or anyone else’s concern.
2. A good way to conserve your face is to not make claims that may turn out to be wrong in the future. Your face is not at risk, then. An ounce of prevention is worth a pound of cure.
Andrew
Re: Andrew (#164),
if we scientists were all Vulcans you’d be right, but science is done by all-too-humans, so manners and opportunities for saving face and all that really do matter. Mr. McIntyre understands this. You start out by making nice, and only move on to less cooperative modes if necessary. See McKitrick’s post #169 too. You can catch more flies with honey than with vinegar, as the old saying goes.
We are down to the last 4 countries to FOI. Who wants to take them?
Someting that I haven’t seen expressed, is the premise that the Mole knows his job better than we do.
What I mean, is that he knows the data, knows what he signed, and knows what he can release. Maybe he leaked the data BECAUSE HE CAN, without restriction or repercussions from his job.
Maybe the only thing he has to fear is the wrath of Jones…
Re: canaries and Statistical Tests
This being a site with emphasis on data validity, it would be fun to apply statistical tests to the ‘mole’ data. A place to start would be the probability distribution of the leading digit. The Kolmogorov-Smirnov test could be applied to compare the actual distibution to Benford’s Law
Re: Bob Hamilton (#174),
that’s a neat idea but are we sure temperature measurements are one of those things that should follow Benford’s? Some r.v.s don’t (sum of normals would not; mixture of normals would). Is there a known finding here for Benford’s and temp?
Another complication, I think, would be the human tendency to round and/or record measurements on salient numbers (e.g. multiples of 2/10 of a degree, just as a suggestion). The heaping of observations on salient rounding points is common in historical data sets (heights for instance, which have their own problem of being roughly normal as well).
The FOI are completed. Thanks to everyone who helped.
Never can get appppology right!
Re: John S. (#177), If you do not get a link I can upload it somewhere (Cannot remember where I got it from it’s 1772 to 2008/12 monthly data:
Mike
Re 177:
There is a diagram here:
Click to access upps_www.pdf
And the complete temperature data are here:
http://www.smhi.se/content/1/c6/02/50/31/attatchments/uppsala_daily.zip
(Yes “attatchments” is correct).
I have had some correspondence off line in response to my petition.
I understand that should the information not be freely forthcoming before the House sits again, the matter will be raised in Parliament.
NW,
Aren’t “professional” people (and “scientists”?) supposed to ask and respond without the gamesmanship?
I know I’m asking a rhetorical question, but my posts usually get disappeared and I can’t help but cash in on this opportunity. 😉
Andrew
Re: Andrew (#193),
I tend to think about these matters as a pragmatic social scientist, or perhaps a descriptive sociologist of science, rather than as a prescriptive philosopher of science. I notice that many posters here have strong prescriptive notions about the practice of some kind of ideal science. From these notions emerge strong assertions about what scientists ought to do, what they are supposed to do and so forth.
Perhaps it would be good in some suitably-defined sense if everyone behaved that way. But think about this from the perspective of an individual scientist. If it isn’t true that everyone behaves according to the posited norms of ideal science, then it may not be individually rational for the individual scientist to do so either (even if he or she IS a Vulcan game theorist…such is the wacky world of game theory). I have to work with the professional and scientific worlds I actually live in, rather than some ideal one that doesn’t actually exist. That real world includes people with reputations, egos, emotions, desires for status, a taste for vengeance when they feel they have not been treated justly or politely, and so on.
I trust I’m saying something obvious to you. You and I are, I think, just having a semantic difficulty. You are talking about some sort of ideal science, and I (and McKitrick at #169) are talking about the professional worlds we actually perceive (our perceptions might be wrong, but that’s another issue). We may not think this world is ideal, but we play the human hand we’ve been dealt.
Re: NW (#195), I think I’m gonna barf.
Perhaps Steve may have made a strategic blunder in announcing his spectacular acquisition so early.
Firstly, it gives a heads up to those who cringe at the thought of Steve having his hands on this data. Expect them to prepare a formidable, sustained preemptive response.
Secondly, it does perhaps needlessly put Deep Cool in a real hot spot.
Thirdly, I would not reveal the names of any other parties who are privy to this dataset.
I think if Steve had just quietly acquired the data set, evaluated it, and then informed (or shocked?) us with the results a few months down the road, it probably would have delivered a bigger shock waves (assuming the data is dubious).
Could get awfully nasty, I fear.
I’ve waded deep into speculation, I know. I guess I have to trust that Steve knows how to play poker, and knows how to defend himself against mud slingers and dirty PR.
Then again, if we see a lot of panicked attempts to shut Steve up, then it could be a sure sign that there’s something totally rotten in Denmark.
The little stretch of 18th century Lund data that Steve Mc posted shows surprisingly little yearly correlation (R**2 = .28) with Phil Jones’ constructed CET series. In 20th century Europe, stations more than 1000km apart often show twice that much. It’s more than just the credibility of the HADCRUT global indices that is at stake here!
Re: sky (#193),
two observations about possible pure statistical reasons for that (as opposed to some sort of bad behavior).
1. One is the possibility that measures tended to be more accurate (that is, pure measurement error might have less variance) in the 20th century than the 18th. In that case, we’d expect even detrended correlations (and r-squares) between randomly selected stations in the 18th century to be lower than those in the 20th century. (Do we have any historians here who know about the relative measurement error across the age of instrumental measures?)
2. It doesn’t sound like you are working with detrended data. If common trends were stronger in the stretch of 20th century you are looking at (stronger than in that short snippet of 18th century history), AND fluctuations around trends were had essentially similar variance in both periods, then correlations (and r-squares) between raw measurements at randomly selected stations would be stronger in the 20th than the 18th century. (I’m sure someone here knows the rough facts about this and can clarify it for us.)
Re: NW (#211),
Neither of your observations really gets to the core of the low correlation in evidence between Lund and CET. Being a genuine measurement record, Lund correlates reasonably well (R**2 = ~.5) with DeBilt, whereas the constructed CET series correlates even more poorly (R**2 = .14) with DeBilt over the same decades, despite the fact that it is much closer.
I’m not necessarily suggesting “bad behavior.” All constructed series involve some manufacture in filling gaps and in equivalencing anomalies from different station records. That manufacture plays a major role the early centuries of the CET series is amply in evidence from certain internal checks of the data. The situation is even more grievous with the SST component of HADCRUT3, where many of the series in the little-traveled sectors of the Southern Ocean are the largely the product of manufacture instead of measurement.
Possesion of the complete data base used by Hadley runs the risk of exposing the tenous measurement basis of their touted “observational” product.
Re: sky (#237),
if I understood your other posts earlier, you are correlating yearly figures (created by averaging across months, but your basic unit of observation is a year average) across pairs of locations, for 21 years in all. Is this correct?
Re: sky (#237),
I should say getting R-squares from univariate regressions of one location on the other, but same diff (squared).
Re: sky (#237),
and do you speak the lingo of random coefficients and Monte Carlo simulation? Just checking so I don’t either insult your intelligence or use unfamiliar terms. We’ll gradually eliminate alternative possibilities together.
Re: NW (#240), et seq.
Yes, I’m regressing the 21 yearly averages (1753-1773) of one station upon another. This provides a crude, but effective, measure of zero-lag cross-correlation. Monte Carlo simulations of individual time series can be a useful game, if the complete auto-correlation function of the generating process is well-established. That’s not going to happen here with the paltry record for Lund or the uncertain record for Central England. Presumptions can be made, of course, ad infinitum, but that reveals little about reality. Having a day-job to attend to, I’ll leave that game to others.
Re: sky (#246),
Well, for what it’s worth, my intuition was that a difference in r-squares of 0.14 and 0.5 didn’t seem significant with 21 observations. Here’s the simplest possible null model I can think of with some reasonable random measurement error, perfect spatial autocorrelation of simple linear trends, but no temporal autocorrelation of errors. Dumb as a doorpost. (Expecting very witty riffing from the peanut gallery.)
Suppose you had an underlying true linear trend of +0.005 degrees per year at every city, for a two-decade change of +0.1 degrees. In other words it is a world with a perfect and identical linear trend in temp at every city. The ONLY source of variability in measured temps in this world is measurement error. At each monthly reading, temp is measured with a pure normally distributed error with a standard deviation of 0.1 degrees. In that case the twelve-month mean (year average) in every city would have measurement error with s.d. of about 0.028 (=0.1/root(12)).
So I drew 10,000 samples of 21 observations of temp for two cities, each with the identical per-year trend of 0.005 degrees, but with their own idiosyncratic normal measurement error in each year with s.d. of 0.028. The tenth and ninetieth centiles of the resulting distribution of r-squares (of regressing one city temp on the other) are 0.136 and 0.519, respectively. So 0.14 versus 0.5 is at best borderline interesting in a statistical sense, with this null model.
I admit there’s no distance effect here making closer city pairs better correlated than more distant city pairs (in this null model all city pairs have identical underlying trends and i.i.d. measurement error), but one could add in a bit of that and get a similar result. I think that adding in most reasonable kinds of stationary time series stuff within each city would simply make the 10th and 90th centiles of such a distribution even more extreme, but I don’t claim special expertise in continuous time series stuff so I might be wrong about that.
What I do think is that if this very dumb linear trend + measurement error null model can produce a similar amount of variability in r-squares of city pairs, it casts reasonable doubt as to whether that variability is very diagnostic of anything.
I have no doubt, though, that there will be some “manufacturing artefacts” in anyone’s indices, whatever their motives. At this level I certainly expect you are right.
Re: NW (#247),
Grossly unrealistic premises of your simulation (which eerily mimic the AGW story-line of “climate trend plus weather noise”) mislead you.
There simply is no universal trend evident regionally, let alone globally, in actual station records. On the contrary, LLS trends are highly variable and disparate even at neighboring stations. Moreover, variabilty of yearly averages is not the product of measurement error, but of natural meteorological processes. These processes totally swamp any error remaining after averaging some 730 (or 1460, depending on country) daily observations that go into the yearly average. Thus inter-station correlation is the not the result of a common trend corrupted by independent errors, but of weather systems commonly experienced. (All of northern Europe is usually traversed by cold fronts from the N. Atlantic). In fact, such commonality is the pro forma justification for the deplorable practice of equivalencing anomalies from different stations in constructing grid-point series. Thus the acf at regional stations shows considerable similarity. But it is never that of white noise, as you suppose, or even AR1, as IPCC would have it.
The centiles of your simulation simply have no bearing on the problem at hand. Anyone experienced with penetrating analyses of actual records will instantly recognize that R**2 = 0.14 between CET and DeBilt is abnormally low considering their proximity, especially since the same years of the 20th century produce a value of 0.83. I’ll refrain from waxing academic with confidence intervals from a semi-realistic statistical model.
Re: sky (#248),
well some of this I just don’t agree with. Adding trend variability across cities is just going to make the confidence interval much wider: I already did that one a few days ago. Adding cyclical variability will do the same thing only worse. Adding human measurement error, not “weather noise”, was meant to illustrate how even small amounts of measurement error can produce a lot of variability in results across even identical pairs of cities with identical underlying true values of temperature, not as an assertion about trends. The purely i.i.d. stochastic stuff, like pure human error in measurement, is underappreciated. I was trying to make a point about that.
Your point about the 20th century correlation between CET and DeBilt is well-taken. It could partly reflect less pure human measurement error in 20th than 18th century, but 0.83 versus 0.14 is starting to impress me… Point taken.
But think about pure measurement error, please! It matters.
Re: NW (#249),
Measurement errors and quantizing noise are well-known problems that easily quantified analytically. But you have to do the math right. It’s not a major issue in yearly average temperature series, where interannual variability with r.m.s. typically ~.6 degrees Celsius makes for a reasonable S/N ratio.
The basic problem with your simulations is that they lack any semblance of representing that variablity. You’re mistaken in thinking that adding cyclical components will only make “things worse.” For simplicity, take three sinusoids with periods given by the square root of 47, 31, and 23 and amplitudes of 1, .5, and .25 degrees Celsius. This takes you into right ballpark and prevents cyclical repetition in simulation. Add all three to one of your series, but only the first two to the other. You’ll find that the cross-correlation increasea and its variability decreases–dramatically. Beyond this suggestion, I can offer no furether help here.
RE sky #194,
Did you anomalize it first, i.e. subtract out monthly means? These look like raw Celsius readings. With surprisingly mild winters.
BTW, I don’t think the C scale existed before the French Revolution, so presumably someone converted it somewhere along the way. Unless this is some uniquely Swedish Fahrenheitsson scale…
Re: Hu McCulloch (#194),
My regression of the yearly averages was done without “anomalizing” the monthly data.
Re: Hu McCulloch (#194),
Hu, apparently Celsius (a Swede) developed the centigrade scale in 1742; the “Celsius” term was officially adopted internationally in 1948.
Steve,
I’m sure you’re having loads of fun with your new toys, and perhaps won’t even read this. But if you do, well few people have been talking about what to do with the data, and my plea is that you help us understand how HadCRUT3 in June 2009 can be so high when the satellite measurements are so low. It might be a rare statistical quirk, or a time lag, but it certainly seems like the sort of discrepancy which is worth an audit.
Cheers,
Rich.
NW,
I completely understand dealing with the reality. I just want to give some ink to the idea that it doesn’t have to be/shouldn’t be this way. People forget things that are not brought to their attention every once in awhile. If you work in such a culture where this kind of behavior is commonplace, there is obviously room for improvement, and perhaps in the future, people can agree on changing for the better.
Andrew
Persistence paid off. Thanks to whomever you are for risking the release of this data.
It takes brave people to do what’s right in the face of powerful governments. Your risk is much appreciated and oddly for a temperature metric, was a service to humanity. If the data is good we’ll know now, if not we’ll also know.
Either way, the public had the clear right to know. Just because the right is not written on a famous piece of paper doesn’t mean it isn’t true, we deserve to know the science behind the pending policy as surely as we deserve to pay taxes. Stating that the source data for a temperature series on which so many papers are based cannot be disclosed was asinine (stated politely).
Time to open the cookie jar and see what’s in…..
Maybe they just missed this telegram:
“Rheims May 8 1945
As you may have noticed there is no further need to classify weather data.
Thank’s for your cooperation!
IKE.”
I’m very worried by these words from Mr. McIntyre. It does make an impression that he doesn’t take this Mole seriously? If I were him, I would have kept this information secret until it has been fully investigated and implicated. Look at how Woodward and Bernstein worked with “Deep Throat” over a longer period of time. It might be wise for Mr.McIntyre to read this story and how to approach moles and how to actually benefit from this amazing turn of events.
By publicising it at this very early stage I fear this vey promising turn of events might have died at a very early age.
I’ve said it before, Mr.McIntyre really needs a good advisor how to explore his undoubtebly enormous potential of statistic knowledge into the real world and how to approach the outside world with this knowledge. Mr.McIntrye is sometimes just to linear in his thoughts.
Re: Hoi Polloi (#201),
Hoi Polloi, I’m betting post 152 is right on the money.
Re: EA (#209), In hindsight you may be right, my humble appologies, maybe I have to get used to McIntyres oddball humor. Just wondering if and how this knowledge will escape into the open world, or will it stay in this tightly knit elite website as so much of McIntyres rumnifications… That’s why I stay with my hint for a PR advisor.
Hoi PolloiRe: Hoi Polloi (#201),
I would advise you not to play chess with Steve McIntyre.
Re: jeez (#202),
If I remember well Chess is a game that allows you to win if you look ahead many moves and fathom your opponents strategics? Sometimes I do miss that with Mr.McIntrye.
Exactly, putting it all in the open is not the best strategic.
Re: jeez (#202),
If you do, keep the king at home.
You need to be careful when setting up leaks. I know about these things. My ‘annuity’ business used to depend on them. 🙂
Should the next step in the FOI requests be to request the confidentiality agreements? By knowing what parties are involved and what data is protected under these agreements new requests can be submitted that omit the data covered by the agreements. In that case they would have no reason not to release the rest of the data not covered by confidentiality agreements.
Hu McCulloch #195
I live in Lund and those are normal winters. We usually have January temperatures in the range from -5 to 5 degrees centigrade.
Well, FM. There’s a live one here speaking in pomo. ‘Linear‘. LOL
So I’m wondering if the sharp difference in June 2009 temperature data sets is why the Mole has acted? Was it finally more than he could stand?
We could use a few more moles…. say at NOAA, NCDC, GISS, etc.
Regarding CRU data:
A mole
Digging in a hole
Digging up my soul now
Going down, excavation
I and eye in the sky
You make me feel like I can fly
So high
Elevation
#20 Why are you always called Nigel?
#69 Public interest defence.
Re: Jack video :
It’s a rat, not mole. Has an auditor looked in to this? 😉
Fahrenheitsson…
Re: Hu McCulloch (#194),
– Ha! Chuckle. That is funny. Anders Celsius is a Swede.
http://en.wikipedia.org/wiki/Anders_Celsius
A Mole
In the words of Agent 86 in Get Smart…. Ahhhh. The ol’ Keep it secret in the name of openess trick.
Video of the mole
Maybe the supercomputers at CRU had finally had enough of this nonsense and decided to stir things up a bit?
“Ghost in the Machine” perhaps…
McIntyre really has to make the data set public. Keeping it to himself would not exactly be in the spirit of his work to date. Maybe I’ve missed a post of his, but has he said when we will get the data?
Re: DaveR (#226), No.
Jones really needs to make the data set public. He has no excuse for the secrecy he keeps his data shrouded in. Without verifiable data, any pronouncement Jones makes about the consequences of his analysis of it are mere hand waving.
What I am guessing that Steve has (from his description) is a previous iteration of the data set. Should Steve make it public, he would open himself up to criticism that he is using an outdated version, and claims that any audit he does is without value. He would also face the criticism of infringing IP by releasing what someone else claims ownership of for distribution.
What Steve has is a thread that may lead him to a rope that may lead somewhere fruitful. What he does with it and when he does it is up to him.
If you want the data, pester Jones.
Re: Joeshill (#228),
please read this entry thefordprefect (#148)
48% of the data the ECA uses has restrictions on passing it on. They have therefore made it unavailable as did CRU/MET Office.
Do you suggest that the ECA and CRU break these agreements (illegal?)?
How is this a problem of CRU. It is the suppliers of the data that need to be forced to give it freely. Perhaps the requests for names of institutions that are charging for data is the only good thing to come out of this mire. But it is not CRUs fault that it should not be made availabele.
Looking back over the threads McIntyre and others only started asking for restrictive suppliers in the last few weeks. All requests before 25th June(?) were simply demanding the data, and getting refusals.
Climatic Research Unit was established in 1972 when were these restive agreements made. It could be over 20 years ago. Are records still available? Perhaps they should be. But if not it is easy to say some data is restricted – therefore all data is restricted use.
The ECA being set up with the history of CRU as a foundation have noted the restricted suppliers and pulled their data leaving the other accessible.
Re: thefordprefect (#229),
I am far from persuaded that there are any “confidentiality” agreements that prevent CRU/Met Office from providing the data to me. The original CRU station data was published at CDIAC; a version from the mid-1990s was placed online at CRU. In my post CRU Then and Now, I showed that stations added since the 1994 public version have been from GHCN stations or green countries like Norway.
Re: thefordprefect (#229),
you need to be more precise ford. McIntyre is requesting temperature data from jones. That temperature dataset is
GLOBAL. In the case of jones’s data a large portion of it is taken from GHCN ( see the other threads), an undisclosed
portion is from non GHCN sources. GHCN, of course, is freely available which is why we can se Hansen’s data.
the ECA only covers EUROPE. As you note 48% of ALL there data for Europe is freely available. The portion of temperature data that is available is ( i recall) 42%. And freely available means free to redistribute in the form received. There are no
restrictions on using the data to create an “index” or for griding. Further, some of the restrictions may prevent people from redistributing DAILY data, but they can create monthly averages and distribute those. Or the data, typically supplied as a TMAX and TMIN, may be supplied as TAVE. It all depends on the agreements. So, we FOIed the contracts. With regards to your argument about restrictions and the contracts gone missing.
1. All contracts I have ever seen on these matters have terms which cover expiration. basically you can’t hold
somebody responsible to guard the data FOREVER. Typically it’s 3 years.
2. Your argument that “if it nots easy to say data is restricted, therefore all…” is a horrible incentive to keep bad
records. basically, you are rewarding people for bad document control. basically you are over classifying some
documents. over classifying documents ( promoting a non classified bit of data into the classified world) is
data mishandling.
3. Please write to the MET and demand that they investigate the mole. Please ask them to write Steve McIntyre and
request that he destroy the data.
4. Do you have any CRU data?
Somewhere amongst all the angst about ‘moles’ is the real meaning why this data and maybe this data in particular is being somehow ‘withheld’. Could it be that the real trend of the world’s temperature is not only down, but seriousely down. If down enough it would materially affect real estate values in a dramatic way in a short term. All other things being equal, such a set of facts would transcend in the Bibical sense any issues of ‘moles’, ‘fault’, canaries whether trapped or simply farted, or any other specious nonsense. If Europe is going to become like Siberia, and North America as well from Kentucky on north, then the world’s people have a right to know. Seems our supposed ‘mole’ and his titillating ‘partner’ are holding out on this data. The rude graph in the body of the replies could suggest that more data could be a ‘real canary in a real coal mine’. Question is web site holder, are we or are we not heading for an abrupt ice age? Yes or No? Please! Enough already of the ‘angels on heads of pins’ debate.
Nowhere have I encouraged readers to expect any smoking guns in this data set. Quite the opposite.
My own best guess as to why they are so obstructive about the data is the specific commercial interest of CRU. My guess is that they spend negligible time on quality control, but derive a lot of funding for a prestigious data set and use the funds for other purposes. They don’t want anyone to see how simplistic their analysis is and how negligible their quality control. Nothing more, nothing less. (But that’s just a guess. The real reason may be different again.)
The Lund record was almost certainly obtained from the GHCN dataset, here is my copy (with exactly the same values):
6450262700101753 -18 -13 37 76 115 146 167 159 134 99 32 -30
6450262700101754 -10 -14 -12 56 129 152 151 155 119 101 47 19
6450262700101755 -38 -53 8 79 120 178 182 154 121 84 37 20
6450262700101756 19 23 25 42 99 176 194 156 141 92 18 -1
6450262700101757 -28 7 14 82 107 182 214 176 136 52 60 13
6450262700101758 -39 -20 7 33 139 167 160 168 118 67 45 10
6450262700101759 24 23 34 62 102 174 201 181 131 91 21 -20
6450262700101760 -40 -10 7 61 118 192 182 171 153 85 40 25
6450262700101761 6 12 50 68 129 180 173 183 152 63 51 -6
6450262700101762 11 -7 -18 81 115 170 174 142 124 48 41 5
6450262700101763 -39 5 5 46 112 149 178 169 115 77 28 30
6450262700101764 -1 30 14 56 125 136 205 163 118 76 24 1
6450262700101765 -3 -23 29 69 102 153 159 169 119 90 45 -2
6450262700101766 -18 -27 21 81 119 173 188 172 138 87 58 -9
6450262700101767 -61 -6 21 27 98 139 164 173 150 89 64 4
6450262700101768 -55 -31 -24 51 107 163 179 171 125 82 49 21
6450262700101769 6 -5 23 57 113 156 176 159 136 52 26 32
6450262700101770 -23 0 -29 45 115 151 181 181 154 105 25 15
6450262700101771 -38 -38 -39 22 124 180 172 151 125 101 28 25
6450262700101772 -16 -21 -11 47 100 162 178 171 136 110 72 29
6450262700101773 10 -9 16 73 141 159 181 180 145 112 50 26
At least 95 percent of the temperature series used in CRUTEM and HadCRUT gridded products are publicly available. In some cases, the CRU has some additional data directly from weather services. Austria is a good example for that. As a Hungarian citizen, I have access to these “secret”/”sensitive” Austrian records via old Austrian yearbooks stored in our meteorological library…
Re: Adam Soereg (#234),
I agree with your comments. Like you, I believe that 95% of CRU is obtained from GHCN, with a very few non-GHCN sources, of which Austria is one (Norway, Sweden, Denmark are others.) Like you, I believe that they do relatively trivial manipulations of GHCN data. AS I’ve said elsewhere, that is my best guess as to the secret that they don’t want exposed and the only commercial interest that they are protecting.
Re: Steve McIntyre (#238),
Ding ding ding. The thing I cannot fathom is why such an important dataset is not handled by an organization that is committed to the highest standards of quality control and data transparency and method transparency. Jones, NOAA, and Hansen documentation ( short journal papers) hardly do justice to this vital dataset. Unlike others I don’t think there is some great antiAGW smoking gun in the data or methods. I agreed with gavin on this ages ago, but that belief doesn’t excuse the kind of pathological sloppiness we’ve all witnessed in their work.
Re: Steve McIntyre (#238),
What would this say about those commercial interest that would pay for something that has been collected by merely using, in effect, free and otherwise accessible data? I would guess that something other than commercial interests (I can see some interest in a recent data – maybe) is at the root of CRU’s hesitancy to reveal their exact sources.
If CRU mainly mimics other sources and applies little added value in adjusting the data and yet has scientists using the data over a long time period, perhaps they are simply not ready to reveal the minimal efforts applied to this data series with any commercial value of it notwithstanding.
The whole CRUTEM / HadCRUT gridded series can be easily reproduced for the most of the globe with the GHCN dataset. I’ve tried it for some gridcells and it worked.
The calculation done by CRU is very simple. They convert all data between 1850 and present to anomalies with respect to the 1961-90 base period, and then simply obtain the average values of these anomalies for each gridcells.
The anomalies are given with an accuracy of at least 3-4 digits in the CRUTEM file. After analyzing the final digits, you will know how many stations were used for the calculation of the current gridded data. If the final digits are similar to 0.025/0.050/0.075/, there were 4 individual stations, if 0.033/0.066/ it was calulated from 3 stations, etc.
Re: Adam Soereg (#235),
Seems to be quite straightforward. Using this data, I get quite close to current CRU NH:
But I’d like to find 1990 version of CRU NH to see how big the difference really is.
Re: UC (#250),
If you don’t mind an extra year, try this 😉
RE Adam Soereg #235, how do they convert a station that had no data before 1961 to a 1961-90 base period? Or one that has data for only part of this period, or only after?
Re: Hu McCulloch (#236),
According to Brohan et al, the station normal is taken from available values over the 61-90 normal period. And if there are less than 15 values, some special procedure is initiated. I used all available values in the previous figures, fixed it:
Makes a difference, better match now.
Re: Jean S (#252),
That set seems to include marine data, is there land-only average anywhere?
Re: UC (#254),
The effect of not having proper instrument shelters in the early 1800s sure stands out. I’m guessing on the basis of some cursory compilations that 1915 and 1926 were the warmest years of the 20th century, until the WWII urbanization intensified. Please confirm. That urbanization is what likely produced the trend seen in this indiscriminate CRU average.
Re: John S. (#255),
Another reason for the high volatility seen in the early stretch, of course, would be very limited geographic coverage of available data, likely confined to northern Europe. The variance of yearly average anomalies is by no means spatially homogeneous, but increases strongly with latitude and continentality of climate. This reveals yet another weakness of the “global anomaly” concept.
Re: John S. (#256),
That’s the main reason; for example for SH 1832-1840 only one station, Rio De Janeiro, contributing to the average ( any chance to find missing 1841-1850 data for that station ? The gap is annoying 😉 ) .
Too bad that the up-to-date data is confidential, it would be very educational to try different methods to generate global anomalies ( median , different normal periods, extension to -1850 , add reasonable uncertainties, etc. )
Re: UC (#257),
Can’t really help you with early Rio data; my archive of “oldies” is pretty much confined to Europe. I suspect that the Upsalla series, about which I had inquired earlier on this thread, has been incorporated into the constructed Tornedalen series. If not, I would sure appreciate the yearly series you posted here. Roman M has my e-mail address.
First differencing, as Jean S. proposes, obviates the need for an arbitrary “norm” in the anomaly approach. It also high-passes the interannual signal, which will have some statisticians shaking their heads, despite the fact that this determinstic effect is analytically well-known. The AGWers will be screaming, of course, because their beloved trend has been reduced to a (highly variable) constant. Instead of taking the median–which is a bastard measure analytically–I simply normalize station data wrt the 20th century mean and work in terms of standard deviations in compiling areal averages. It makes some difference, but since I exclude stations that show strong UHI in my compilations, that difference is not dramatic. The drama comes from using urban stations like Rio, with their unrelenting trends.
Re: John S. (#258),
Go to http://cdiac.ornl.gov/ftp/ndp020/ , open jonesnh.dat and search for Uppsala.
Hmm, I don’t have the Peterson 98 at hand, but
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/grid/README_GRID_TEMP.txt
..and I don’t have time to do the math, but I guess ‘mean of first differences’ method is equal to some anomaly method. With median things might change.
In the normal case mean is better, but the distribution can be different, especially if we have moving stations.
Re #209 and #152 I agree. The mole is likely inanimate, with electricity in its veins.
I have an old book titled “World Weather Records, 1941-1950”. It was put out by the U.S. Dept. of Commerce Weather Bureau in 1959. The previous three volumes were “published by the Smithsonian Institution” in 1927, 1934, and 1947, and “have been widely used in meteorological and climatological research all over the world.”
I think that Mr. Soereg is correct that many of the records are available in hard copies in libraries (if they all haven’t been sold off to used book stores). The entry for Budapest Hungary actually goes back to 1780 and the provenance of the stations used is explained in the station notes as well as a brief description of how adjustments were made.
There’s more difference in SH versions, but ndp020r1.txt tells that
and NH is already corrected for inhomogeneities.
r2(CRU,UC) for 1961-1990 is 0.96, so I’m not completely lost with my gridding m-files. It would be interesting to try this with up-to-date station data, eh? 😉
Expanding on my comment 245 above, the “World Weather Records” volumes were digitized into the Global Change Master Directory at NASA as (DSI-9644). They appear to be available through that website. I believe Mr. McIntyre said Jones used these for some of his papers. Jones published some papers in the Monthly Weather Review back in Vols. 110 and 112 which explain the methods he uses in determining Variations in Surface Air Temperatures with some of his data. Looking through his publishing record at the CRU website, it looks like he has put an enormous amount of his life into dealing with these data. But looking through the provenance of the data in my hard copy book, I doubt if it’s even possible to reproduce what has been done. It’s an absolute rabbit’s warren of private individuals, colonial gazettes, private companies, previously published papers and books, college professors, various monarchs, and diaries as well as mostly governmental met. offices.
Wasn’t CRU originally set up by Hubert Lamb at UEA in the early 70’s? I’ve never read his Climate Past Present and Future, but maybe Jones has just built on his (Lamb’s) foundation and over the last 30 or 40 years his data set has just sort of grown with no real established provenance?
In 1907 Gilbert Grosvenor said of the US Weather Bureau, “It is fitting then, that meteorology, the science of the weather, should be a distinctly American product, and that the people of the US should have the best weather service in the world. The US government spends $1,500,000 a year on its Weather Bureau, which is more money than all the governments of Europe combined spend for similar service. It has a staff of many hundred skilled experts and trained observers who in all parts of the country are constantly on the watch to see what the heavens will bring forth.” I think I would be very suspicious of the accuracy of world data unless it’s very recent. I think anything prior to the 1950s or so should probably be considered a proxy.
Thanks for the directions to CRU’s known ftp site. I was really hoping to get the complete Uppsala yearly series as shown by Bergstrom in your #222, instead of the gapped version. Oh well.
Obviously, when the entire sequence of first differences is known precisely, any time series can be recontructed from the starting point by elementary accumulation of those differences. But this is also quite pointless, since the advantage of working without a “norm” is lost thereby. In effect, the starting point becomes the norm. When, however, that sequence needs to be estimated from variously deficient data, the estimation error also accumulates. Frankly, I seen no advantage to NOAA’s method, but only the disadvantage that an egregious error is propagated forward. Amidst the choppy signal produced by first-differencing (whose amplitude response is given by 2sin(pi*om/oms), where om is radian signal frequency and oms is the sampling frerquency), bad outliers can readily hide.
At the risk of belaboring a point, standardizing largely (say >85%) intact records wrt to their 20th century means allows each available data point to be included in areal averages. Inasmuch as the form of the spatial distribution of temperatures can scarcely be considered to be well-known, I see no advantage to using their median. Outliers are easily spotted in standardized form and can be eliminated entirely. Furthermore, when geographic coverage deteriorates, the volatility of the global result is nowhere near as great as in the anomaly approach. To be sure, the results obtained by standardization can no longer be expressed rigorously in physical units, but only in terms of statistical ones. Since dry-bulb temperatures by no means fully represent the total thermal energy fluctuations, this purely statistical treatment is all that seems warranted.
Re: John S. (#261),
It’s not CRU’s ftp site 😉 #222 series was just down-sampled to annual from the data linked in http://www.climateaudit.org/?p=6634#comment-350759
We’ll see how robustly different methods behave once the raw data is available.
It is essential that there be public access to data sets of great importance collected using public money. Restrictions should be should be anathema to any right thinking person. Is there a lawyer who can comment on the fact that the Lund data set was collected by many people who deposited it a public form for later retrieval? In other words, copyright does not apply (not claimed by the author of the creative work and too old), nor does any normal confidentiality agreement as you cannot claim confidentiality for any information already in the public domain. Nor does any obligation on a “servant” apply as any release is not a breach!
What is at issue is what analysts are doing with the data- and this should be completely open to pulbic scrutiny as is normally the case in science.
That is very strange that the cru is so protective of the data that is supposed to be for public use. I wonder what they are hiding did something strange happen in their data?
19 Trackbacks
[…] some WUWT readers may have learned from reading Climate Audit, an anonymous source deep within Hadley CRU has provided Steve McIntyre a copy of a data file he […]
[…] A Mole […]
[…] offentliga data för temperaturstationer (som han nu ändå fick ut via en mullvad). Steve McIntyre sammanfattar en rad märkliga svar från Met […]
[…] is niet voor het eerst dat CRU nogal amateuristisch met gegevens omgaat. Een paar maanden geleden ‘lekten’ de temperatuurmetingen uit die Jones al jaren weigert vrij te geven. Al snel bleek dat het niet om […]
[…] that CRU had left versions of the confidential station data on their FTP site for many years (see CA, July 2009, A Mole?) NERC recognises that some data holdings supplied by UKMO under the arrangements are commercially […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed,” he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] of the beam data. He publicised some, from metropolis in Sverige between 1753 and 1773 – “sensitive aggregation indeed,” he noted on his Climate Audit blog. The mass period he claimed on the book that the mol had […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed,” he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed,” he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed,” he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed“, he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed“, he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – “sensitive information indeed“, he noted on his Climate Audit blog. The following day he claimed on the blog that the mole […]
[…] every other FOI had been illegally blocked but what else can you do? 25 July 2009: The next day McIntyre announced that he had got a mass of CRU data, essentially all that sought in the harassment… from “a mole”. Note that this may be true or may be misdirection to protect external hackers. […]
[…] ikke lang tid siden, skete der faktisk et leak af data fra en insider på CRU. I dette tilfælde, fik Steve McIntyre fat i dataen, noget han i øvrigt havde bedt om aktindsigt […]
[…] re-visited this issue in the Mole post last summer. I observed in a comment (along the lines of my 2008 post): Nowhere have I encouraged […]
[…] a full set of the station data. He published some, from Lund in Sweden between 1753 and 1773 – "sensitive information indeed", he noted on his Climate Audit blog. The following day he claimed on the blog that the mole had […]
[…] interest was added to the protest by the “Mole Incident” (see here) and CRU’s subsequent deletion of numerous files from their FTP site, including two different […]
[…] Update: OK, been reading a lot over at Climate Audit and was pointed to a CRU Mole: A […]