The Guardian reviews the various theories on FOIA2009.zip, closing with what must be the absolute worst nightmare scenario for UEA. After observing that “so far, the police investigation has got nowhere.”, they state:
It is not even clear whether the crime of computer data interception has actually occurred.
They describe a scenario in which no crime occurred, observing:
If this hypothesis turns out to be true, UEA may end up looking foolish. For there will be no one to arrest.
Climate Audit 
105 Comments
Does this mean that the contents were already internally collated by someone at CRU, and not by an external “hacker?” If so, then this is the first admission of this that I’ve seen.
Consider this possibility: The files were the residual emails of a batch which had already been *sanitized* from the CRU systems, in order to illegally prepare an incomplete response for a future (likely successful) FOIA request.
Re: jallen (Feb 5 23:00),
…and it only took a couple of half-hour sessions with Phil to convince them that was the way to go. (just extending a mere possibility)
Never ascribe to malice that which stupidity adequately explains.
I’ll subscribe to CTM’s theory. I still wonder how the package was created in the first place.
The title of the file by itself makes it very likely that the file was created internally. This was definitely the ‘gang that couldn’t shoot straight’. The lack of a truly independent inquiry is disappointing.
I posted a comment about that in the “The Mosher Timeline” thread earlier today.
Here
There is a real good possibility that the UTC time offset of 5 hours on all of the zipped files that have the “2009 Jan 1 00:00:00” time stamp and a 4 hour offset for those files that retain their original time stamp show that the unix computer that zipped the files was in the Eastern time zone (4 hour EDT – 5 hour EST) and some file were zipped before and after Nov 1 (end of DST). Of course, the time zone could be changed to hide the actual location, then again, maybe not.
Re: ErnieK (Feb 5 16:52),
There are some CRUTeam players in NYC. Back to the old merry-go-round: Did whistleblower assemble the files or did CRUTeam assemble the files in order to dump them? CRU could have scraped the CA keywords, done a search on their servers, grabbed all of the incriminating emails and got their purse snatched before they could delete.
ErnieK – I have always thought that the very first email of the bunch was some sort of clue in that regard.
Ernie,
It isn’t usual for a computer to have the wrong timezone set on it. Several OS’es default to a USA timezone when they are installed.
Although they are calling it a backup server its probably just some network storage that the staff can copy files to for safe keeping. The timezone of the server wouldn’t be that important as long as it was consistent.
Re: TerryS (Feb 5 19:07), It would be a real clue to the investigators though, if the “backup server” in question is a unix machine, is running version 2.3 of zip, and does in fact, have it’s time zone set with a 5 hour GMT offset (UEA should be a zero offset I believe). If it does not, then the FOI file was probably not zipped on that machine.
For the record, living in the EST zone, it seems to me that every new computer is actually preset to PST.
Re: ChrisV (Feb 5 22:51),
If it has windows it’s set to PST with the geolocator pointed at Seatlle 🙂
Re: Jimchip (Feb 6 08:58), Both the epoch date nameing of the mail files and the header information of the zip files indicate that the mail and data were compiled on a unix based machine which was probably NOT a Macintosh (AFAIK the latest version of zip on a Mac is version 2.1 and 2.3 was used to zip the FOI files.)
Re: ErnieK (Feb 5 16:52),
Unix Epoch Time is set to GMT and is ALWAYS time zone independent. Doesn’t matter what time zone the server is located in. There are a number of ways it can be manipulated with a few lines of CLI code, but a mail server would, by necessity, have to be set to GMT. Otherwise, the UTC timestamp would be showing mismatched local times in the email headers.
Re: Duke C. (Feb 6 13:25), Well yes and no. Epoch time is the number of seconds since the Epoch (00:00:00 UTC, January 1, 1970. Converting a given moment it time anywhere in the world should result in the same epoch number – and it does, if the local time is first converted to GMT/UTC and then converted to epoch, but that requires that the unix computer doing the conversion knows what time zone that it is in.
For example: the last email in the FOI2009.zip file, 1258053464.txt converts to “Thu, 12 Nov 2009 19:17:44 GMT”, but the email header date is “Thu, 12 Nov 2009 14:17:44” which is a 5 hour difference. Now, if the computer converting the email header date to a epoch time file name was in the EST zone (or had it’s time zone set to EST) then 1258053464 will convert to “Thu, 12 Nov 2009 14:17:44”.
Re: ErnieK (Feb 6 16:32),
ErnieK:
Not disputing what you posted. In fact, what you discovered running “zipinfo-v” is new to me. But it doesn’t prove the machine was located in the eastern time zone.
IT guys write some pretty convoluted shell scripts containing “timezone aware” objects for a variety of reasons. One example- They may want to shift late night housekeeping functions by “fooling” an archive routine into thinking it is midnight when in actuality, it’s 4AM. And that comment in the Guardian about the 5 hour time difference isn’t anything new. The investigators could have picked up on it by reading CA. We were discussing the time zone offset here a few days after the .ZIP file was released.
The 10th email from the bottom has an offset of 4 hours.
Re: Greg F (Feb 7 12:38), A lot of the emails have an offset of 4 hours. My theory is the e-mail epoch file names were created on a computer that had it’s time zone set to the Eastern time zone (which as Jimchip and Duke C. point out, might have nothing to do with the actual location of the computer*), and some files were processed during daylight savings time (EDT) which would be a 4 hour offset, and others were processed after the switch to standard time (EST) which would be a 5 hour offset.
As Duke C. says, the time can be manipulated by a savvy IT guy, but internal header UTC timstamps within zip files and epoch time conversions depending on the timezone are easy to overlook and miss even by the best IT guy, if he didn’t think about it at the time.
* I don’t subscribe to the theory that the time zone was simply EST/EDT because the computer came that way. Anyone knowledgeable enough to pull this off probably had his/her computer set to the correct time and zone. I believe that the time “clues” left behind were either a simple oversight or intentionally set that way to hide the actual time zone.
Re: ErnieK (Feb 8 13:18),
“intentionally set that way to hide the actual time zone” or intentionally left that way. I was tempted to do that and see but I wanted the clock set correctly. meh. It would have been so easy to accept bogus defaults thinking “that will help to obfuscate things more”.
Re: ErnieK (Feb 6 16:32),
Ernie, I found out that a fresh Suse linux install defaults to NYC (eastern) timezone. Just like windows, only different. Modoki.
A crack pot IT investigation could well have within 48 hours determined if it was by hacking or not. Only apologists for warming are claiming theft of files. I am sure Jones can join O.J. Simpson and continue looking for the real criminal.
Lol, I assume you mean “crack”, as in extremely good at what they do, rather than “crack pot”, which means something completely different.
“so far, the police investigation has got nowhere.”
The British police baffled by a case involving computers? Quelle surprise.
Why am I thinking of s stuck needle? The same observation repeated over and over and over again.
Give it up. Steve’s blog is his diary. His public appreciates his perhaps historic efforts to shed light upon the darkening of scientific processes, and its rebirth. I joke not.
Having the name of FOIA2009.zip it sounds like a collection of emails someone wanted to keep hidden from the powers that could request them. I wonder if the emails still exist on the server or if the exact ones in the zip file were deleted around the same time the file was created. In that case, I would suspect a climate scientist of ‘hiding the’ emails and then being careless with the zip file. It could have been Prf P J!
Lol. I don’t think so. Why would you name something so it would be easy to find as such? I would have named it “Phil’s ski pictures” or something totally innocuous. Naming it “FOIA2009” is like saying “DON’T LOOK AT ME!! I’M NOT A COLLECTION OF FOIA-SENSITIVE FILES!!”
It is a bit silly to claim that the name of the file indicates who created it or the motives of the person that created the file.
“Freedom of Information” would have been the purpose behind the hack or a whistleblower. It also would be the name of the act that allowed McIntyre to request that information.
So, if I were the techie that was asked by the school to collect the data from the archives, my choice of file names would probably have been FOIA2009.zip. If I am a hacker or whistleblower that thinks the information needs to be given to the public, I would also call the file FOIA2009.zip.
You do not know by the name of the file if the creator of it was doing his job, was trying to hide the information, or trying to distribute it to the world.
The one constant in every apologist article on this topic is that “stolen emails” always finds its way to the first paragraph. Just to let you know who was the “victim.”
A guy’s wife finds emails to his girlfriend and divorces him. He tells his friends “We broke up because of some stolen emails.”
In a sense, yeah.
Oh noes!!! No one to arrest. They will then resort to the typical lynch mob mentality and demand arrest of “the usual suspects”.
FOIA2009.zip Doesn’t the ‘FOIA’ part of the filename indicate that it was put together (or at least named) by an American or someone working in the US. As I understand it, ‘FOI’ is what the Brits call their act while we call it ‘FOIA”.
Maybe a Brit who created the file named it FOIA based upon “Freedom of Information Archive”, a descriptive name. Read nothing into a name when there are logical alternatives that also meet the criteria.
When I was in the British civil service we always called it FOIA, it’s easier to say.
Re: Rhoda R (Feb 5 19:10), The leaker actually called it both. The first file that was uploaded to RC was named “FOIA.zip”. The post that was placed on RC called it “FOI2009.zip” (download now: HYPERLINK “http://ftp.tomcity.ru/incoming/free/FOI2009.zip”)
Re: ErnieK (Feb 5 20:56),
Perhaps he used both FOIA and FOI to make it difficult to determine whether his nationality was British or American. If he’d used only one, people could have guessed his nationality. Of course, he could have used this to confuse people by doing it the way that was not indicative of his nationality.
I was federal civil service (US) for 30 years and we never called it FOI. It was FOIA, whether written or spoken.
I have wondered if the file was assembled by a Brit, then the A in the file name might be a version designator as I use an alphabetical designator for version control on my computer. The obvious speculation being that this would imply a second (at least) version exists or was contemplated.
So, thank you Rhoda R for raising this point
Your time frame is off. Nobody was saying anything until the files were verified. It coulda been a hoax. There’s nothing wrong with a little discretion while the verification is going on.
“You’d think there’d be discussion on the blogs of something like that.” It started when Mosher started posting the emails…Verification.
“But McIntyre was meanwhile guarded with his source in Norwich. He emailed him back: “I haven’t seen such a website. You’d think there’d be discussion on the blogs of something like that. I’ll definitely stay tuned!!” Only after the bloggers had launched their great scoop did he inform Dennis”
First you need to be in command of all of the facts. The only things steve knew were what Charles and I told him. And we only told him what we knew.
1. That a comment had been placed in the que at WUWT. Charles had downloaded the file and passed me a CD. We believed we had the only copy of the file. We didnt think it was at any other site. Steve asked for the link and charles said no. I read steve mails over the phone to verify the authentcity of the mails in the stack which were sent by him. So Steve had not seen it on any website. Neither had I.
2. One concern that Anthony had, being in Brussels, was that he was being set up. So nobody was supposed to speak of the existence of these files to anyone.
Charles knew, I knew, Anthony knew and Steve knew. The question about whether to make it public or not was real. Anthony, as the owner of the blog wanted to get back to the US and consult legal authorities before making the decision.
So there wasnt any intent to deceive. Steve is very precise with his words.
But why did you think you had the only copy? Why did you (Stev Mo, CTM and Steve Mc) not look at any other popular skeptic sites to see if it was there too? It was posted at WUWT, at Jeff Id and at climate skeptic at about the same time, right?
Also, what exactly was the “posting direct from the secret leaker” the Guardian says you recieved?
Re: PaulM (Feb 6 17:39),
Briefly, Steve Mc missed the “miracle” link at CA, Jeff Id let the comment go through and then took it down later when he got home, CTM took down the WUWT link. I don’t know about climate skeptic.It’s reasonable that WUWT had a link given the popularity of the weblog. I assume Mosher, et al were busy worrying about the copy of the files that they did have and not running around wondering if anyone else had links to the files.
Lastly, “posting from the secret leaker”. I’ll leave that to someone else.
PaulM,
Perhaps you should be asking The Guardian?
What about the fact that Paul Hudson was forwarded at least some of the archive in October? (i.e. before the ‘official’ release of the FOIA2009.zip archive).
Steve: the Paul Hudson thing is a red herring. Please do not discuss – see old posts.
I think that ZT’s point about Hudson potentially having a valid (or useful) email address may be a good one.
Hudson says that someone forwarded to him the “chain” of emails that dealt with his “happened to global warming” article.
He did not say from whom he received this chain of emails (and he certainly wasn’t included in the ones that are in the collection). Now, it might have been done by one of the individuals involved in the exchange (and not in any way related to the later release of documents): one of those “warnings” that the Team occasionally meted out to journalists who were becoming unreliable. Alternatively, it could have been someone involved in the release of the materials.
“A programme called TOR, for example, can be downloaded which will automatically switch between a random variety of servers. Digital forensic examination of the archive of emails and documents suggests that it was first created around 30 September, and subsequently added to during October and finally in November – when one of Osborn’s sets of program code was added – just ahead of the full-blown leak.
Significantly, that analysis suggests that the archive was created on a machine running five hours behind GMT, which would put it on the east coast of North America.”
Ah ha. Hadn’t seen that bit of sleuthing before. I wonder how solid it is?
Re: Rich (Feb 5 19:44),
Here and Here are some mole topics.
Thanks, but I couldn’t find the phrase “I downloaded [it] from the public CRU ftp site” or “No hacking was involved” on either page. I was just curious to see the context of the original quote, as I’m sure there must be more to it.
This is very speculative and perhaps in error, but I also suspect the leaker was an American. Something about the language in the accompanying text. Perhaps some Brits can comment on this, but is the phrase “limited time offer” commonly used in the UK? How about “kept under wraps?” Also, and this might be a stretch, but in the russian server address the word “incoming” is used, and this was a term commonly used by American soldiers in Vietnam to warn of incoming mortar rounds . The use of the term “Hopefully,” to begin a sentence is also improper and I’d be interested if Brits ever use the term the way it was used in that sentence.
It just sounds like an American to me. But I could be wrong.
The use of the word hopefully to begin a sentence may be controversial, but it isn’t incorrect. According to Merriam Webster’s Collegiate Dictionary, Tenth Edition:
Re: MJW (Feb 5 21:21),
Here’s the context: “Hopefully it will give some insight into the science and the people behind it.”
We are off topic here, but my expository writing teacher in college would have said there were several better ways of saying the same thing. He would have suggested, “I am hopeful that . . ” or “I hope that . . .” or even “One hopes that . . . ”
In it’s defense it seems to be a short cut for saying, “I am hopefully saying that it will give . . .” etc. But that is verbose.
It may be considered standard usage now, but it really is clumsy.
Hopefully, you now understand why I think it is improper. 🙂
This is a quote from around that time:
Strunk, William Jr. and E. B. White. The Elements of Style, 3rd Edition. New York: McMillan Publishing Co., Inc., 1979 (p. 48).
Re: Rich (Feb 5 22:47),
Thanks, Rich. I vaguely remembered something like that was in The Elements of Style and went looking for my copy. Because we’ve moved twice in the past five years and have packed and re-packed once too often, I couldn’t find it.
“Silly” sums it up.
This is the last I’ll say on this matter, as this isn’t a language blog, but I’m not alone in considering “The Elements of Style” an unreliable reference. Linguist Geoffrey Pullum says,
Re: MJW (Feb 5 21:21),
Just for fun, I ran a quick and dirty grep search on FOIA/mail for all sentences beginning with “Hopefully”.
There were nine individuals who started a sentence using “hopefully” at least once. 5 Brits, and 4 Americans.
Interestingly, Phil Jones topped the list, with 6 unique hits.
That’s a good example of applying the scientific method to the issue. According to Merriam-Webster’s Dictionary of English Usage the earliest uses of hopefully as a sentence modifier are American, but this usage crossed over to the British Isles by the 1970s, where in addition to the usual criticisms, it was sometimes denounced as an Americanism.
(Yes, I did promise to say no more on the “hopefully” controversy. But Duke C.’s comment dealt not with the proper or improper use of the word, but with whether its use was a clue to the nationality of the leaker.)
I’m a fairly literate Englishman, I don’t find any of those words or phrases unusual. They don’t in any way suggest a non-British author.
Re: Faustino (Feb 6 01:41),
Thanks, Faustino. I’ll take your word for it. There are common idioms. I wasn’t sure if the ones I mentioned were among them.
Re: Rich (Feb 5 19:44),
Sorry Rich, I misread your links and ended up duping them. I don’t know where the particular quote came from.
First, let me be very clear that I don’t think its possible to bring a theft prosecution because all of the information released was public information. If it was a hacker, then they can bring charges for unauthorized access to a computer, which is a serious charge in the UK — but its not theft. So the use of the word ‘stolen’ is wrong in my mind.
However, the CRU keeps insisting that the information was private — and therefore the information was ‘stolen’. If that’s the case, then UK PII laws require UEA to notify, within 30 days, all individuals whose personal information was breached.
Steve, I’m wondering if you received such a notice?
Failure to provide notice carries stiff civil penalties. And, if it turns out that UAE failed to adequately safeguard PII (by, say, leaving the file on a public FTP server), well then they are in for a really rough ride.
They really should be more careful about the words they use to describe this thing.
About a week or two after the release I put my bet on an unguarded server. We don’t know yet of course, but I’m still leaning that way.
For all of you, who have not seen it, there is a new publication on the Science and Public Policy Institute called “ClimateGate Analysis”. On 149 pages, John Costella presents a thorough analysis of the most interesting email correspondence. It’s quite an insightful read … Here is the link
Click to access climategate_analysis.pdf
Before you lot get too excited about Costella’s “analysis” this is the original source
http://www.assassinationscience.com/climategate/
When reading his infinitesimally nitpicky style it reminded me of the sort of “thinking” that proves that man didn’t go to the moon; that 9/11 was an inside job; Iluminati etc.
Then I looked at the homepage of the website he joint edits:
http://www.assassinationscience.com/
He presents “proof” that the most famous film of the Kennedy assassination was faked frame by frame!!! Then I looked at the rest of the links. Jeez! Now, just because he appears to be a semi-paranoid conspiracy theorist doesn’t necessarily mean that he is wrong but if you read his analysis and start to get swept up with it, just remember the type of mind that created it and the other type of stuff that minds like that also create.
As far as the emails being grabbed by hackers or not goes, wasn’t RealClimate hacked at the same time which seems like a mighty big coincidence…
Steve, Don’t publish this, please – you have very high standards and rightly. And you’ve tried this last 10 years to get others to meet them. And they failed – again and again. But please don’t get bitter – I’ve watched all this time and seen what a gentlemen you have always been. Your worst fears have been realised – but let us hope people might be better from now on. Anyway, I hope your well, I just sense a certian lack of breath on your part. Be well
If I posted this before, forgive me – but I think Darrel Inces (5/2/2010) article in the Guardian is the most apt and knowlagable – it is about the mess, ‘Harry’ told us, that is the code.
I believe that data was accidentally released as many of the climate organisation have a less than professional attitude to data handling generally. I became a climate “denier” after having been sent collection of data by the simple if rather silly mechanism of reply to all instead of reply to sender because the request for the data was tacked onto the bottom of a social get together message.
If there was anything sinister in the release it would have been possible to track it by now. It is probably only because the actual release was done by accident by someone with a total right to it that it is so hard to trace.
As to the police still trying to find out how to turn the computer on that is unfair. They are actually in shock at finding out it is not just for playing computer games.
Someone at UEA should be heading for a 6-month stretch:
http://en.wikipedia.org/wiki/Wasting_police_time
On the wording: young Brits speak a subCalifornian dialect routinely.
Re: dearieme (Feb 6 08:12),
OK, OT, but “Valley Brits”? Hmmm, interesting. Fits my age window for W/B also <–on topic
Andy Revkin is still of the opinion that an “illegal” act hs been committed:
http://dotearth.blogs.nytimes.com/2010/02/05/signs-of-damage-to-public-trust-in-climate-findings/
Steve: This is still very much a possibility.
Yes and I think at this stage it would be more accurate to report the level of uncertainty rather than to present it as a given. People unfamiliar with the situation could be given the wrong impression.
“dearieme
Posted Feb 6, 2010 at 8:12 AM | Permalink | Reply
On the wording: young Brits speak a subCalifornian dialect routinely.”
English is the international language of science; but its not English English or American English.
Scientists often talk mid-Atlantic, which makes life easier on non-English native speakers. Written Scientific English differs from both American/English English and people talk in the language they write.
As a native Englishman, living in America for 5 years I can state that there are no language proxies that lead one to assume an American over an English writer.
Even spell checkers are typically set for American spelling when writing for Journals, you can write color, sulfur and specialize for British journals.
An accidental release or unguarded server may in fact be the best outcome for UAE. If a leaker/whistleblower is identified, then he/she would no doubt have lots to say, and I doubt that much of it would be pleasant listening for UEA, CRU or any of the Hockey Team. Their best bet is to be shown to be sloppy.
UEA statement quoted at Bishop Hill: http://www.parliament.uk/documents/upload/091210_UEA_Vice-Chancellor_Letter.pdf
This presents a possibility that none of us has considered (I think). Gavin said that there were 4 downloads from RC. What if the copy at the Russian FTP site is a “descendant” of the version at RC. In other words, the poster RC was one of the 4 downloaders who then placed it over in Russia?
We’ve all tended to assume that the person who uploaded the dossier to RC was the same person that uploaded it to the Russian server – but this might not be the case. Of course it’s also possible that Acton’s language is uninformative on this nuance – intentionally or otherwise.
Wouldn’t it be ironic if RealClimate had played a pivotal role in the distribution of this material?
Re: Steve McIntyre (Feb 7 23:18), “Schmidt says the hacker “disabled access from the legitimate users, and uploaded a file FOIA.zip to our server. They then created a draft post”.
It read as follows: “We feel that climate science is, in the current situation, too important to be kept under wraps. We hereby release a random selection of correspondence, code, and documents. Hopefully it will give some insight into the science and the people behind it. This is a limited time offer, download now: HYPERLINK “http://ftp.tomcity.ru/incoming/free/FOI2009.zip” \t “_blank” http://ftp.tomcity.ru/incoming/free/FOI2009.zip.””
I know Gavin has given the time when the FOIA.zip was uploaded to RC but has he said when the “draft post” was created? And was the “draft post” created by the same “hack” method into RC that the upload was done? If at the same time (or within minutes) of the upload, then the file must have already been on the tomcity server. If hours later, then two different people are a possibility which fits in to the difference in naming conventions of the FOIA vs. FOI files.
Re: Steve McIntyre (Feb 7 23:18), Maybe I am reading too much into the wording Edward Acton used in this memo, but “…downloaded in whole, or possibly in part…” has bothered me for some time.
It makes me think that the “backup server” was a dedicated server used for storing backups of e-mail servers and data files from other computers within the UEA computer network. Someone then downloaded either all or part of those backup files (they really don’t know which). Nothing indicates that the leaked files were already compiled into a single (or several) files on that server and that file was downloaded.
The implication is that someone might have the entire contents of that backup server stored somewhere and has only chosen to release a small portion of those backup files. (A backup server could have backups of every desktop workstation on the UEA network).
Re: ErnieK (Feb 8 01:52),
Maybe I can clear that up a bit. Mosher has mentioned that each one of the released emails has a “nugget”. I still think (and I don’t know) that, regardless of the whole set of files that someone might have, there was a serious filtering operation that occurred outside of any identity bleaching. GregF has a plausible scenario. I have a 4GB thumb drive (and a few more as spares). The zipped file is ~62MB. One question was did they zip on the server or take it home and zip? Who knows? But, one can store a lot of files in 4GB. No one wants to read all the typical university email stuff like “Late seminar today, about an hour late for dinner”, unless, maybe, it ends with “Cheers, Phil”, and I haven’t seen any of those late for dinners from Phil either.
Re: Steve McIntyre (Feb 7 23:18),
It’s curious than no one (to my knowledge) has taken a closer look at the chain of provenance WRT FOIA.ZIP and FOI2009.zip. I just did.
Don’t have time to take a closer look right now, but here is what pops out:
FOIA.ZIP:
Bleached timestamps are preserved (01-01-2009 00:00)
Unix pkzip version- 2.1??
Extended unix utc byte values are different-
FOIA.ZIP- 7B 25 06 4B 10 86 5C 49
FOI2009.ZIP- 03 D0 4D 5C 49
There are other indications that FOIA.ZIP (the version uploaded to Realclimate and downloaded 4 times)differs from FOI2009.ZIP (the viral version)
FOI2009.ZIP could very well be a descendant. Maybe it was the leaker, maybe it was an anonymous soul not seeking credit… Who Knows?
ErnieK and Jimchip-
Heres a link:
FOIA.ZIP
Re: Ian Fordyce (Feb 8 13:43), Will the real first version of the FOI(A) file please step forward.
First glance at this file shows that it was zipped on a Macintosh computer possibly at 2009 Nov 20 05:13:31 UTC (2009 Nov 19 20:13:32 local time) which is a 9 hour offset. The mail files also seem to have a 9 hour offset (Alaska time zone?). The timestamps on Central directory entry #4:, the “__MACOSX/” directory, seem to be actual times which would place it days after the first “leak”. So it appears that this file may be a “after the fact” upload to 4shared.com. But again, the dates and times must be taken with a grain of salt.
Is sure seems that one of the archives was download, un-zipped and then maybe re-zipped again??? — but they sure are not the same. The FOI2009.zip contained 4662 entries and the FOIA.zip has 9422 entries. This may be because the way a Mac stores its files and file attributes.
Archive: /Download_tmp/FOIA.zip 65731344 bytes 9422 files
End-of-central-directory record:
——————————-
Actual offset of end-of-central-dir record: 65731322 (03EAFAFAh)
Expected offset of end-of-central-dir record: 65731322 (03EAFAFAh)
(based on the length of the central directory and its expected offset)
This zipfile constitutes the sole disk of a single-part archive; its
central directory contains 9422 entries. The central directory is 1058315
(0010260Bh) bytes long, and its (expected) offset in bytes from the
beginning of the zipfile is 64673007 (03DAD4EFh).
There is no zipfile comment.
Central directory entry #1:
—————————
FOIA/
offset of local header from start of archive: 0 (00000000h) bytes
file system or operating system of origin: Unix
version of encoding software: 2.1
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 1.0
compression method: none (stored)
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2009 Jan 1 00:00:00
file last modified on (UT extra field modtime): 2009 Jan 1 01:00:00 local
file last modified on (UT extra field modtime): 2009 Jan 1 09:00:00 UTC
32-bit CRC value (hex): 00000000
compressed size: 0 bytes
uncompressed size: 0 bytes
length of filename: 5 characters
length of extra field: 12 bytes
length of file comment: 0 characters
disk number on which file begins: disk 1
apparent file type: binary
Unix file attributes (040755 octal): drwxr-xr-x
MS-DOS file attributes (00 hex): none
The central-directory extra field contains:
– A subfield with ID 0x5855 (old Info-ZIP Unix/OS2/NT) and 8 data bytes:
73 20 06 4b 10 86 5c 49.
Central directory entry #4:
—————————
There are an extra 16 bytes preceding this file.
__MACOSX/
offset of local header from start of archive: 503383 (0007AE57h) bytes
file system or operating system of origin: Unix
version of encoding software: 2.1
minimum file system compatibility required: MS-DOS, OS/2 or NT FAT
minimum software version required to extract: 1.0
compression method: none (stored)
file security status: not encrypted
extended local header: no
file last modified on (DOS date/time): 2009 Nov 19 20:13:32
file last modified on (UT extra field modtime): 2009 Nov 19 21:13:31 local
file last modified on (UT extra field modtime): 2009 Nov 20 05:13:31 UTC
Re: ErnieK (Feb 8 15:11), The link Here is an explanation of the “__MACOSX/” directory and related files that the Macintosh “Archive” includes in zip files. It is just extra metadata that could be of some use in a Mac to Mac file transfer, but is meaningless for a PC or other unix machine. Since these __MACOSX files were created by the Mac at the time of zipping, the creation time stamp should be the actual time of the Mac’s clock. Assuming the clock was correctly set, these files were probably zipped 2009 Nov 20 05:13:31 UTC. I doubt that this is the original RC FOIA.zip file.
Re: ErnieK (Feb 8 15:11),
I agree with your observations. It turns out that a young fellow from Anchorage, Alaska was the 4shared.com source.
However, this would not account for the different unix pkzip version.
Re: Ian Fordyce (Feb 8 20:15), The Macintosh version of zip is 2.1. That was the first thing that alerted me that the file you linked to was zipped from a Mac. Seeing the __MACOSX files confirmed it.
Don’t get hung up on the name of the zip archive though. It could be renamed to anything and that would have no effect on the contents if the zip file. The main directory that was zipped file is named “FOIA” (with sub-directories mail and documents). If that FOIA directory was zipped without specifying a specific output file name, then the default file name of the zipped file would be “FOIA.zip”, which of course was the name that first appeared on RC and the Anchorage person uploaded to 4shared.com – they simply used the default name. Whoever put the file on the Russian server probably simply renamed the file.
My guess is the Anchorage person downloaded and un-zipped the file, which could have been FOI2009.zip then decided to zip it again on his Mac which resulted in the default FOIA.zip name, and placed it on the 4shared.com.
The only person that probably knows if the first file uploaded to RC and the Russian server file are identical is Gavin (who probably retained a copy of the RC file) and he isn’t telling.
Re: Ian Fordyce (Feb 8 13:43),
Your Filestube link definitely has a different FOIA.zip than the one I’ve been working with since late Nov. I’ll agree your link is a MacOSx zipped file. So now I know of three:
FOIA.zip (original?) bleached to mostly Jan.1, 2009
FOIA2009 that I haven’t looked at.
Mac FOIA.zip
I compared the documents folders and did a random look inside the files: The two FOIA.zips look identical except for the Unix zipping vs. the Mac zipping. I didn’t comapre the emails yet.
Re: Jimchip (Feb 8 22:17),
ErnieK, I’m missing something here.
FOIA.ZIP central directory abstract from the Alaskan connection:
part number in which file begins (0000): 1
relative offset of local header: 0 (0x00000000) bytes
version made by operating system (03): Unix
version made by zip software (21): 2.1
If the dossier had been re-archived with a Mac then the version upper byte would have been re-set to (07), the pkzip Macintosh identifier.
Here’s a reference: PKZIP Doc.
Also, when FOIA.ZIP (Alaska) is extracted the individual file properties retain the washed modified/create date (1/1/09 00:00)
When FOI2009.ZIP is extracted the file properties last modified date is changed according to the local time on the downloader’s computer. this would give some credibility to the parent-descendant theory.
These were my results, yours may vary. 🙂
Re: Duke C. (Feb 9 01:36),
Duke, I just did the download again from Ian’s link to make sure but the extracted files have all the .MacOSX metadata in one folder and then the ‘normal files’.
I extracted on a WinXP SP3 system. So, I think those files touched a mac at some point and I didn’t see that before Ian’s link. The ones I’ve been working with extracted into two folders not four, the four being ~2x the two in terms of number of files.
I did look at FOIA2009.zip and I agree with your last modified date statements.
Re: Duke C. (Feb 9 01:36), Yes, it is confusing if you are not familiar with Macintosh. The original Mac which was a single user system, based on an operating system designed by Apple would have used the ID of 7. The “new” Mac, OSX (Operating System 10) is based on The University of California (Berkeley) version of the unix operating system for which Apple has designed a elaborate user interface (the average user has no idea that it is a unix machine.) So an ID of 3 is correct for unix based Macintosh.
The version of the zip can also be confusing on a Mac because there are two ways to zip files. One is to use the /usr/bin/zip unix command. The version of that zip can vary with the version of the operating system (OS 10.3 had version 2.1 of zip and OS 10.5 has version 2.3). The other, and more common way to zip files, is to use the Apple provided user interface where one only needs to select the files(s) to be zipped and use the “Compress/Archive” option which will create a zip file whose name will be the same as the main directory (i.e. FOIA will become FOIA.zip) and the version of the zip will show as version 2.1.
Now this zip has been modified by Apple to save the extra extended file extended attributes that Apple uses. That is why you will find extra files like “__MACOSX/…” in the zip archive. The Mac zip creates those files as place holders for those attributes even of the files contain none. In my opinion, that is dumb, messes up the zip format, and created a larger zip file, for no reason, if there are no extended attributes to store.
The second part of your comment deals with the preservation of timestamps being different between the two files. I believe this is again due to the difference is the way the files were zipped and the different subfields created be each (the different subfields might contain the date to recreate the exact timestamps or not). I haven’t had time to say for sure and will post it if I do find out the precise reason for this.
Re: ErnieK (Feb 9 14:20),
So what we have here is a kid in Alaska who downloaded FOIA.ZIP on his Mac around 11/19, took a look at it with a Ziplight type Mac utility and as an afterthought, uploaded it to a file sharing website. The best that I can tell, Ziplight (Spotlight) injects a bunch of Mac meta-data which allows fast file scanning without any extraction occurring. I think his source was the original FOIA.ZIP, the same as Jimchip’s.
What’s more interesting, however, is the 9 hour offset that matches the local time of the source. This is sort of independent verification that the unix extended UTC attributes have a “timezone aware” function, and makes it more likely that the leaker may have used a computer set to eastern time.
Also, there is no unique PKZIP OS identifier for Linux. My hunch is that a Linux created zipfile using zip utility (can’t remember the name) version 2.51 would show up in the header as Unix Ver. 2.3.
A Linux machine set to EST (perhaps by default during installation, per Jimchip) is starting to make more sense.
Re: Duke C. (Feb 9 20:02), Yea, I think that is about it. Actually all the guy in Alaska had to do was down load the zip archive and double click on it and the Mac will automatically un-archive(unzip) it. He then decided to create his own archive using the Mac’s Finder utility “Compress/Archive” and it built a new zip file containing the __MACOSX info and added the UTC time stamps (when done this way the user never actually uses unix commands at all, only the user interface, which uses the unix /usr/bin/ditto command, not zip, to produce the zipped file. It is ditto that produces the extra Mac files.)
Does anyone know if the file we have been discussing, the FOI2009.zip with the 5 hour offset, is the same file that came from the tomecity.ru server that CTM first downloaded? There seem to be several versions floating around.
I just found another FOIA.zip on the “megaupload” server Here
This one has fewer clues. No UTC time stamps at all. Zipped used version 2.0 on a unix machine. The only clue that I could find is the FOIA/documents directory has a modify time of 2009 Nov 22 09:35:22 and the FOIA/documents/yamal directory has a modify time of 2009 Nov 22 09:33:00 – as if someone “touched” those directories without changing anything.
If those times are correct clock times, then this file probably just another after the fact clone.
I got the FOI2009.zip off the Russian server at about 6:00 PM EST on the 19th of November.
Re: Greg F (Feb 9 23:13),
Greg F,
Do you have a way of verifying the MD5 checksum on your russian server copy?
It SHOULD match this:
da2e1d6c453e0643e05e90c681eb1df4 FOI2009.zip
I think there is some historical significance in establishing an online source wrt the genealogy/provenance of the original data source(s),which are rapidly disappearing amongst an ocean of clone files. IMO, CA is the proper venue for doing this.
Partially verified Genealogy thus far-
11/17/09 6:20AM EST-
The Gavin Schmidt original FOIA.ZIP is born (Jimchip has a copy?)
November 17, 2009 at 9:57 pm-
An FOIA.ZIP offspring appears at the Air Vent:
http://ftp.tomcity.ru/incoming/free/FOI2009.zip (you have a copy)
November 20, 2009 at 12:59 am-
Link to another offspring source posted at the Air Vent:
http://www.megaupload.com/?d=XD050VKY
If the MD5 checksums match, it’s an identical twin to the Russian server version
Any versions appearing after 11/20/2009 are irrelevant.
If we catch Mosher in a generous mood, perhaps he’ll loosen his grip on that WUWT CD-rom copy.
How about it, Mosher? Just puttin’ it out there… 🙂
(Apologies in advance for the duplicate post. had to remove links to bypass moderation)
Re: Greg F (Feb 9 23:13),
Greg F,
Do you have a way of verifying the MD5 checksum on your russian server copy?
It SHOULD match this:
da2e1d6c453e0643e05e90c681eb1df4 FOI2009.zip
I think there is some historical significance in establishing an online source wrt the genealogy/provenance of the original data source(s),which are rapidly disappearing amongst an ocean of clone files. IMO, CA is the proper venue for doing this.
Partially verified Genealogy thus far-
11/17/09 6:20AM EST-
The Gavin Schmidt original FOIA.ZIP is born (Jimchip has a copy?)
November 17, 2009 at 9:57 pm-
An FOIA.ZIP offspring appears at the Air Vent:
tomcity.ru FOI2009.zip (you have a copy)
November 20, 2009 at 12:59 am-
Link to another offspring source posted at the Air Vent:
megaupload=XD050VKY
If the MD5 checksums match, it’s an identical twin to the Russian server version
Any versions appearing after 11/20/2009 are irrelevant.
If we catch Mosher in a generous mood, perhaps he’ll loosen his grip on that WUWT CD-rom copy.
How about it, Mosher? Just puttin’ it out there… 🙂
Duke,
The checksum from my copy matches.
Re: Duke C. (Feb 10 13:40), Thanks Duke, the file I have been using matches your MD5 checksum. I agree it would be nice if the file Mosher has on the CD also matches. The $64 question it does the file that Gavin has, the original upload to RC, also match – how about it Gavin, want to share?
Re: ErnieK (Feb 9 21:57),
“the FOIA/documents directory has a modify time of 2009 Nov 22 09:35:22 and the FOIA/documents/yamal directory has a modify time of 2009 Nov 22 09:33:00”
Do the files inside have the bleached Jan. 01 08 dates?
Re: Jimchip (Feb 9 23:48), Jan 1 09 you meant. Yes, as far as I can tell – I didn’t check one to one but it looks the same as the FOI2009.zip except for what I noted. All the mail and many of the docs have that date.
Re: ErnieK (Feb 10 00:45),
All things begin with Yamal.
-FOIA/documents/yamal/-
Have you discovered the clue?
Re: ErnieK (Feb 9 21:57),
Yup. A clone of unknown origin. Looks like it was extracted with a Unix/Linux anti-virus product using the old .ARC format. The files are in alpha-numeric order as opposed to the pkzip arbitrary format. And the UTC/UID/GID fields are wiped clean which changed the archive size. File size before compression matches foi2009.zip (157.06 megs)
Duke C
Some time back I suggested something to the effect:
1. loading the files on a flash and take home.
2. Load up old OS on a spare drive.
3. Bleach files, rinse and repeat.
4. Wipe drive.
5. Upload files from public WiFi with plug in wireless adapter. (Avoids getting laptop identified by MAC address).
Ian, that’s very interesting. I wasn’t aware that there were differences in versions in circulation. Can you interpret further what the version differences indicate – if anything? Do you assume that FOIA.zip is the RealClimate version by the name or is there any other evidence linking it to RealClimate?
Re: Steve McIntyre (Feb 8 14:46),
The only evidence is the frequent reference to a file named FOIA.ZIP (along with a few files available here and there on the ‘net bearing the same name), which can, in most cases, be traced back to Gavin Schmidt. I have no reason to believe that he misstated the file-name back in November when he announced the RC Hack.
Greg and ErnieK, thanks.
foi2009.zip at Megaupload.com is identical to the original FTP Russian server version found at tomcity.ru which is no longer available.
From any other source, verify the checksum:
da2e1d6c453e0643e05e90c681eb1df4 FOI2009.zip
Now, it’s time to draft an FOIA request letter to NASA and get the Gavin Schimdt version released.
Whom should I address it to? Hansen? 🙂
Interesting post from Gavin in reply to the definition of “hacked”. Evidentally it’s still hacking if you get the info from an “office mates” computer:
212 # 15
[Response: Someone who thinks that GW is a hoax is a crank. And stealing people’s emails from a central server and publishing them is a hack regardless of where you do it from….]
Gavin, are you sure you are on solid ground here? Are you sure about UK laws, public and private servers, privacy laws? I don’t know much but you should check carefully first before making such a sweeping pronouncement about email and data ownership, hacking and leaking.
[Response: You think it is legal in any western democracy to hack into your officemate’s computer and put their emails on a public server? No way Jose. Unauthorized access to a computer and unauthorised use of any data you find there is a crime almost everywhere. – gavin]
Re: Martin (Feb 17 17:19),
Gavin and Mann… Jailhouse Lawyers. Maybe they had legal coaching but GS should stick to modeling or whatever he does for NASA and leave the legal opinions to someone who knows. Or, start a separate blog:
Gavin Schmidt’s Official NASA Supported Blog Guide to Climate Legality.
And, regarding his other comment re: “Whistleblowing only works if you are revealing illegal activity”…Uh, Gavin, they found illegal activity but it was past the deadline to prosecute.
“This is now OT. – gavin]” Hahrumph!
More here if interested:
215.Also remember that in the UK there is legislation that protects whistleblowers. Would you call someone in the UK who leaks information from a file in an office (whether private or public) a thief – if it is in the public interest? You are on shaky ground Gavin and should stick to the science and keep away from UK laws and interpretations thereof.
[Response: Sorry, but I will continue to call a theft a theft, and you can continue to cling to the comforting thought that the theft was by a modern day Robin Hood. Whistleblowing only works if you are revealing illegal activity. Arguments about paleo-reconstructions don’t count. This is now OT. – gavin]
http://www.realclimate.org/?comments_popup=2806
Re: Martin (Feb 17 17:46),
Too bad he declared it off topic before you pointed out that failing to live up to ones responsibilities under FOI (or FOIA) is illegal activity.
3 Trackbacks
[…] an inexact science, As the climate debate heats up the populace realise they’ve been conned, Police still trying to work out how to turn the computer on, India to form new climate change […]
[…] Guardian: “Not even clear that a crime actually occurred” The Guardian reviews the various theories on FOIA2009.zip, closing with what must be the absolute worst nightmare […] […]
[…] an inexact science, As the climate debate heats up the populace realise they’ve been conned, Police still trying to work out how to turn the computer on, India to form new climate change […]