CRU Erases Data

As readers know, both myself and various Climate Audit readers have requested CRU station data from both the Met Office and CRU. While my initial request to CRU was refused, I asked for a reconsideration and the matter is still outstanding, as are all the other CRU requests. Under U.K. Freedom of Information Act, once FOI requests have been made for information, public authorities are not permitted to “alter, deface, block, erase, destroy or conceal any record held by the public authority, with the intention of preventing the disclosure by that authority of all, or any part, of the information to the communication of which the applicant would have been entitled.” As fordprefect observed the other day, the Computer Misuse Act also prescribes various offences, one of which is the unauthorized modification of the contents of a computer with the intent “to prevent or hinder access to any program or data held in any computer”.

I was therefore more than a little surprised that on July 27, at about 2:42 p.m. (UK), CRU deleted three files from their data directory ftp://ftp.cru.uea.ac.uk/data/ entitled newcrustnsall.dat.Z, newcruextusall.dat.Z and newcrustnsall.hdr. I have before and after screenshots of the contents of this directory showing the deletion, screenshots that I will show below. I also have a screenshot showing the most recent change of the directory and that no other files were changed. [As readers observed, it is possible and perhaps even likely that the files were moved to a concealed location, rather than totally erased, which would present similar sorts of issues for FOI compliance.]

As noted by CA reader Jean S, on July 28 at about 2:01 p.m. (UK), one or more files was deleted from the directory ftp://ftp.cru.uea.ac.uk/people/philjones/. In this case, we do not have a before screenshot, only evidence of the deletion of one or more files from the directory. [Again, it is possible that the file was moved to a concealed location, rather than totally erased.]

First, here is a screenshot of ftp://ftp.cru.uea.ac.uk/data/ taken on Saturday, July 25. You;ll notice the three files named above, two of which are dated 2/24/2003.

Second, here is a screenshot taken late yesterday showing that the three files named above have been deleted.

Third, here is a screenshot of the host directory showing that ftp://ftp.cru.uea.ac.uk/data was changed on July 27 at 2:42 pm:

Fourth, here is another printscreen of the contents of ftp://ftp.cru.uea.ac.uk/data in time order, showing that nothing was added on July 27 2:42 pm. The July 27, 2009 change was the deletion of the three files named above from the public directory.

As Jean S observed, the FTP directory also shows a change in Phil Jones’ directory on July 28, 2009, a screenshot showing the time stamp is below:

Here is a screenshot of the Phil Jones’ directory showing that nothing was added to it on July 28, leaving only one conclusion: that something was deleted from the public directory. In this case, I do not have a before- screenshot.

One obvious course of action right now will be to make a FOI request to CRU, seeking information about the contents of the various files that were deleted from public directories and whether they were in any way relevant to any of the outstanding inquiries, and, if they were, whether the deletion of these files from the public directories required authorization by the university’s FOI officer and whether such authorization was obtained.

172 Comments

  1. Nylo
    Posted Jul 29, 2009 at 6:24 AM | Permalink

    Something tells me that they will get away with it, I mean, with the deletion, without any heads rolling. However I think that in the end, they won’t be able to prevent Steve from analysing the CRU data in a completely legal way.

  2. benpal
    Posted Jul 29, 2009 at 6:29 AM | Permalink

    If they think they can get away with this kind of manipulation, they must be amateurs, really.

  3. VG
    Posted Jul 29, 2009 at 6:38 AM | Permalink

    I think now you are in a position to take them to the cleaners go for it!

  4. Posted Jul 29, 2009 at 6:39 AM | Permalink

    Good sleuthing, Steve!

    However, they presumably haven’t deleted all copies of the data, just the ones that were publicly accessible, so I don’t see that there would be any FOI violation for destroying requested data.

    Of course, the fact that the data was publicly available all along as they postured about national interest sensitivity, destroys the case for not making a certified copy publicly available now.

    Even though you and evidently others have downloaded this data, there is still the problem that there is no certification by CRU that this is the data they actually used. They may come back and claim that this was just a compilation of GHCN data that they were using to compare to the proprietary top secret data set that is actually used to construct the CRUTAR series.

    So it’s still important to move forward with the academic FOI requests for the data, as well as the requests for the purported confidentiality agreements.

    I put in my request for the data from my OSU computer last Friday. From the one-month response time to your last request, it looks like they have exactly one month to reply, and so will drag their feet for the full month on these requests as well.

    I don’t quite see the point of erasing all the copies of the downloaded data, however. Erasing your copy protects you from action by INTERPOL for having picked up state secrets that CRU left lying about on the street, but what’s to keep you from getting a copy from some else who did the same thing? The data is not certified yet, but we could all have a lot of fun scrutinizing it in the meanwhile.

    • thefordprefect
      Posted Jul 29, 2009 at 7:05 AM | Permalink

      Re: Hu McCulloch (#4),

      Erasing your copy protects you from action by INTERPOL for having picked up state secrets that CRU left lying about on the street, but what’s to keep you from getting a copy from some else who did the same thing?

      There is NO STATE SECRET – it is commercial data with a value attached. The 2003 data probably has little value but if anyone comes after you it will be lawyers from the various bodies who normally sell their data, requesting non disclosure and a few hundred £.

      • steven mosher
        Posted Jul 29, 2009 at 10:39 AM | Permalink

        Re: thefordprefect (#7),

        Ford, the 2003 data, on your account has little value. That is a nice admission.

        1. Did you contact the commercial interests who own this data?
        2. Will 2009 data become similiarly worthless in 2014.
        3. The harm CRU and MET claimed was a harm to international relations between countries if CRU broke
        confidentiality.
        4. Deleting the data, on its face, indicates that CRU believes one of 2 things, maybe both:
        1. The data still has commercial value
        2. International relations will be harmed if it remains public.
        5. CRUs and METs contention that relations will be harmed is now testable. The data was made available. This
        Month the commercial interests have a decision to make. Do they continue to supply data to CRU or Not.
        If they stop, then CRU has a leg to stand on. If they don’t then CRU and MET were wrong about the damage.

        • Steve McIntyre
          Posted Jul 29, 2009 at 11:01 AM | Permalink

          Re: steven mosher (#31),

          Let’s stop hypothesizing “commercial interests” until those interests can be identified. Once the data is contributed to GHCN, there is no commercial interest left, if there ever had been.

          The only remote candidates are the countries where Jones uses non-GHCN versions (which seem to be public though): Norway, Sweden, Denmark,.. IT should be easy for Jones to locate his confidentiality agreement for Denmark or to ask his Danish counterpart for a copy if he’s misplaced the CRU copy.

          For that matter, maybe it’s time for David Palmer to get more actively involved in this file. If Jones can’t produce the confidentiality agreements to Palmer – that were said by Palmer to apply – maybe Palmer should be carrying out some of his own due diligence and assessing for himself what actually exists, contacting some of the supposed suppliers to see if they’re aware of the Jones confidentiality agreements.

          For that matter, maybe CA readers in the counterpart countries should start sending FOI requests to the various meteorological authorities requesting copies of any confidentiality agreements between the national authority and CRU.

        • thefordprefect
          Posted Jul 29, 2009 at 11:49 AM | Permalink

          Re: steven mosher (#31), You talk absolute twaddle, you do!
          By following this course you have possibly put data transfer at risk – FOR WHAT. Will this affect the other users of commercial dat for research – who knows. I suppose if this happens you will have accomplished your main desire of destroying any proof the GW is occuring.

          If the commercial interests do not take action then all is ok. BUT YOU DO NOT KNOW THE WILL NOT.

          It is a risk that the CRU obviously would rather not make.

          Deleting the data is just to add a limit to dissemination from the source. If you pass it arround then that is your problem.
          I have no interest in this data although I know of other sources – the safe bit:

          10010 709 87 10 Jan Mayen NORWAY 19212002 341921
          10050 780 -142 9 ISFJORD RADIO NORWAY 19121979 101912
          10080 783 -155 28 Svalbard Lufthavn NORWAY 19112002 341911
          10100 693 -162 -999 ANDENES 18681955 101868
          10250 697 -189 10 TROMSO/LANGNES NORWAY 19492002 101949
          10260 697 -189 100 Tromsoe NORWAY 18902002 341890
          10280 745 -190 16 Bjoernoeya NORWAY 19202002 341920
          10490 700 -232 -999 ALTA 18711939 101871
          10530 707 -237 -999 HAMMERFEST NORWAY 18481863 101848
          10620 765 -251 -999 HOPEN 19471959 101947
          10650 695 -255 129 Karasjok NORWAY 18762002 341876
          10920 707 -301 -999 MAKKAUR NORWAY 19511960 101951
          10980 704 -311 14 Vardoe NORWAY 18292002 341829
          11000 660 -20 0 Ship “M” NORWAY 19492002 341949
          11120 656 -123 -999 BRONNOYSUND NORWAY 18701955 101870
          11520 673 -144 13 BODO VI NORWAY 18682002 101868
          11530 668 -140 39 Glomfjord NORWAY 18902002 341890
          12120 629 -65 13 Ona NORWAY 18682002 341868
          12350 621 -91 626 Kjoeremsgrendi NORWAY 18902002 341890
          12410 637 -96 7 ORLAND III NORWAY 19512002 101951
          12420 626 -114 628 ROROS NORWAY 18711955 101871
          12710 635 -129 12 Trondheim/Vaernes NORWAY 17622002 341870
          13130 608 -47 20 HELLISOY FYR NORWAY 18671955 101867
          13170 604 -53 12 Bergen-Florida NORWAY 18162002 341816
          13550 611 -75 36 Laerdal NORWAY 18902002 341890
          13720 606 -91 167 Nesbyen NORWAY 18972002 341897
          14030 593 -49 55 Utsira Fyr NORWAY 18672002 341867
          14150 589 -56 9 STAVANGER/SOLA NORWAY 19512002 101951
          14480 581 -81 9 Oksoey Fyr NORWAY 18902002 341890
          14720 584 -88 12 TORUNGEN FYR NORWAY 18671955 101867
          14820 590 -105 6 Ferder Fyr NORWAY 18852002 341885
          14920 600 -107 94 Oslo-Blindern NORWAY 18162002 341816
          20220 684 -188 388 Abisko SWEDEN 19132002 341913
          20800 684 -225 327 Karesuando SWEDEN 18792002 341879
          21040 658 -151 475 Taernaby/Hemavan SWEDEN 19012002 341901
          21200 670 -177 337 Kvikkjokk SWEDEN 18892002 341889
          21270 651 -172 325 Stensele SWEDEN 18612002 341861
          21420 666 -196 260 Jokkmokk SWEDEN 18602002 341860
          21960 658 -241 5 Haparanda SWEDEN 18592002 341859
          22260 632 -145 376 Oestersund SWEDEN 18602002 341860
          22860 638 -203 12 UMEA SWEDEN 17961804 101796
          22880 636 -208 6 Holmoegadd SWEDEN 18542002 341854
          23240 620 -144 360 Sveg SWEDEN 18752002 341875
          23610 626 -179 8 Haernoesand SWEDEN 18592002 341859
          24180 594 -135 46 Karlstad SWEDEN 18592002 341859
          24330 606 -156 160 Falun SWEDEN 18602002 341860
          24581 599 -176 13 UPPSALA SWEDEN 17222002 351739
          24851 593 -181 44 STOCKHOLM SWEDEN 17562002 351756
          24960 594 -195 12 Svenska Hoegarna SWEDEN 18802002 341880
          25120 578 -119 20 Goeteborg SWEDEN 18602002 341860
          25160 576 -116 19 Vinga SWEDEN 18802002 341880
          25500 578 -141 226 JONKOPING FLYGPLATS SWEDEN 19512002 101951
          25760 578 -166 9 VASTERVIK SWEDEN 18591907 101859
          25820 587 -179 13 Landsort SWEDEN 18802002 341880
          25840 584 -192 12 Gotska Sandoen SWEDEN 18802002 341880
          25900 577 -183 42 Visby SWEDEN 18592002 341859
          25920 574 -171 4 Oelands Norra Udde SWEDEN 18542002 341854
          26040 567 -129 25 Halmstad SWEDEN 18592002 341859
          26160 554 -128 5 Falsterbo SWEDEN 18792002 341879
          26180 557 -132 73 LUND SWEDEN 17531773 101753

          The data should be made available but it is not in the hands of CRU or MO it is those with their grubby mitts on the purse strings. They are just the recipients of your childish and dangerous pranks.

        • Steve McIntyre
          Posted Jul 29, 2009 at 12:00 PM | Permalink

          Re: thefordprefect (#39),

          ford, so far, CRU has not provided the slightest evidence of any commercial confidential agreements. We’ve asked them to do so. If they have the form of agreements that you surmise, then CRU will be able to provide evidence. Until they do, I remain doubtful of the existence of relevant commercial agreements.

        • thefordprefect
          Posted Jul 29, 2009 at 12:24 PM | Permalink

          Re: Steve McIntyre (#42), look The other group: ECA have stated point blank in many places that some data is unavailable because it is restricted. Why should it be any different for CRU.
          Why do you not beleive them?
          I do not know any more than you. BUT you are posing a threat to data transfer by your actions.!!!!!

          Are ECA lying too?

          Click to access ECAD_report_2008.pdf

          Climate variables
          At present the data set includes series of
          nine climate variables: daily …
          About 52% of the series is publicly available
          from the ECA&D website. The other
          48% comes with restrictions: these series
          are for ECA&D indices calculations and
          gridding purposes only.

        • steven mosher
          Posted Jul 29, 2009 at 12:47 PM | Permalink

          Re: Steve McIntyre (#42),

          Well, mr palmer is in receipt of my request. We shall see.

        • steven mosher
          Posted Jul 29, 2009 at 12:34 PM | Permalink

          Re: thefordprefect (#40),

          Re: steven mosher (#31), You talk absolute twaddle, you do!

          you saying so makes it so, I guess. But let’s test your accuracy. how well do you know things?

          By following this course you have possibly put data transfer at risk – FOR WHAT.

          I didn’t put the data transfer at risk. it was never at risk to begin with. CRU have CLAIMED that
          disclosure will put it at risk. by posting the file, CRU is testing its own hypothesis. They should
          thank me for pointing out their adherence to the scientific method.

          Will this affect the other users of commercial dat for research – who knows. I suppose if this happens you will have accomplished your main desire of destroying any proof the GW is occuring.

          Now, ford. you will recall that i asked you not to motive hunt. As Anthony and Steve and Lucia and others will attest
          I am a lukewarmer. I’ve posted countless times that I believe in AGW. I do so for the record again. The addition of GHG
          to the atmosphere by humans is a practice that will raise the temperature of the planet. I used effin modtran hitran and lotran in my former employment. I kinda understand the basic science. http://en.wikipedia.org/wiki/MODTRAN. As I have said many times I believe that AGW theory is the best explanation we have for the warming we observe in the historical record. There is no proof only evidence. proof happens in math and logic. In science we have confirmation and disconfirmation. I am interested in the phenomena of LTP, but LTP is no “explanation.” Now, why am I a luke warmer, as opposed to a “warmer.” I’m a luke warmer because I think that the methods for accounting for UHI or rather discounting UHI are flawed. Warmers point to parkers paper and petersons paper, and hansens work. I’ve read
          most if not all of what they wrote and have made my focus getting the data and methods behind those papers PUBLIC.
          I read Ross’s work on UHI and I se something different. Different conclusions and a different approach. An Open approach. open data and open methods. Until the warmers produce results, like ross, that are open to audit I must
          side with Ross and conclude that a good portion of the warming we see may be UHI. Not all, after all GHGs warm
          the planet. How much? which gets to the issue of sensitivity ( climate models, my other hobby horse)

          Just look at when I post. I post almost exclusively on temperature series posts. on posts about open access. There is a reason why I asked you not to motive hunt. that reason was to protect you from making just the kind of mistake you made. Now, go and sin no more.

  5. thefordprefect
    Posted Jul 29, 2009 at 6:55 AM | Permalink

    Blimey gov do you have this wrong – check with a lawyer who knows the subject.

    “alter, deface, block, erase, destroy or conceal any record held by the public authority, with the intention of preventing the disclosure by that authority of all, or any part, of the information to the communication of which the applicant would have been entitled.”

    This means he has to retain what you have asked for. I.E. the files you have not seen.

    “to prevent or hinder access to any program or data held in any computer”

    This means that you must not block HIS access to HIS files!!!
    There is no reciprocity here.
    CRU have every right to continue their daily work – adding deleting updating files provided any outstanding files in FOI requests are still available (hidden/password protected/or archived)

    Do you have permission to trawl through his directory as you have admitted above? Is this legal!!!!!! I’m not sure, are you?
    mike

    • Ryan O
      Posted Jul 29, 2009 at 7:00 AM | Permalink

      Re: thefordprefect (#5), Seriously, man, what planet do you live on? The CRU site directs you to the ftp directories. See the bottom of this page, for example:
      .
      http://www.cru.uea.ac.uk/cru/data/lwt.htm

      • Posted Jul 29, 2009 at 7:09 AM | Permalink

        Re: Ryan O (#6),
        Well, he comes from a small planet somewhere in the vicinity of Betelgeuse, so one has to make allowances. Maybe on his planet everyone who downloads data from a publicly accessible ftp site without first asking permission is thrown in jail.

        I wonder who has the most egg on their faces – CRU for refusing to release data they had already released, or the auditors for not noticing that the data they were asking for was sitting there all the time?

        • Steve McIntyre
          Posted Jul 29, 2009 at 7:45 AM | Permalink

          Re: PaulM (#8),

          People may surmise that the three missing files contain relevant data, but please note that I have not specifically said so. (People are entitled to speculate:) )

          We don’t actually “know” that the data had been sitting in that location all this time. It may have been. But data can be moved from directory to directory in FTP sites without the date-time stamp changing. It’s possible that the data was in some other directory and later moved into a public directory. This happened with Mann’s data so I have some experience. But equally it might have been there.

          Willis made the original FOI requests for station data in 2006. [Warwick Hughes had asked for the data in 2004 and again in 2005]. When CRU refused to provide the data, Willis reasonably assumed that they were responding in accordance with their obligations under the FOI ACt. There are many files on the CRU directory. Willis had no obligation to parse through every CRU file to see if it was an undisclosed responsive file. I note that there are no links from any webpage to the specific files in question.

          During Willis’ attempts to get the data, I was working on other matters and following his progress from a distance. (I was involved with FOI to CRU on another matter – urban stations in Jones et al 1990, a matter that caused considerable subsequent attention.) As an observer of Willis’ efforts, I had no reason to suspect that CRU would not disclose the existence of publicly available data that was responsive to Willis’ request, nor was the double checking of CRU’s response a priority of mine.

          I did not personally request CRU station data until May/June 2009 when I noticed a representation on the Met Office webpage and later learned that they had sent a copy to PEter Webster. I think that I’ve moved matters along quite nicely once I got involved. Again, I do not feel that I had any obligation to parse CRU files one by one to test whether CRU had misspoken in their refusal letter.

          It was a bit serendipitous that I even noticed these particular files and thought to test them. However, even my worst critics will probably concede that I have a bit of an eye for such things. If it took 30 days or so for me to spot the file, I don’t think that I need to apologize to anyone for taking “so long”.

          Neither Willis nor I have anything to apologize for.

          On the other hand, CRU and Met OFfice have legal obligations. If the requested data is already publicly available, then they are obliged to provide the location of the data. We are not obliged to play guessing games with them.

        • steven mosher
          Posted Jul 29, 2009 at 12:10 PM | Permalink

          Re: Steve McIntyre (#17),

          I wonder if Willis should issue another FOI relative to the posting of the data.

      • thefordprefect
        Posted Jul 29, 2009 at 7:19 AM | Permalink

        Re: Ryan O (#6), while I’m here the directory in question is not that linked to in the data link – up 1 directory and into a private area (foolishly unprotected).

        Steve:

        As fordprefect observed the other day, the Computer Misuse Act also prescribes various offences, one of which is the unauthorized modification of the contents of a computer with the intent “to prevent or hinder access to any program or data held in any computer”.

        Me in one thread back:
        For the purposes of subsection (1)(b) above the requisite intent is an intent to cause a modification of the contents of any computer and by so doing—
        (a) to impair the operation of any computer;
        (b) to prevent or hinder access to any program or data held in any computer; or

        This is to be read with my following post. Steve had highlighted 2b as if it gave him right to access data – rather than the opposite
        Mike

        Steve: I made no such interpretation. My observation pertained to people deleting data without authorization – an issue that may or may not arise as we go along in this matter. I firmly believe that the mole’s actions were authorized or that he believed that he was authorized.

        So another hmmmmm for McIntrye.

        Steve: We were talking about the “mole”. When the “mole” placed the data online, I believe that he was authorized to do so or that he believed that he was authorized to do so. And doubtless did so as a routine activity.

        The subsequent deletion of the data is a different issue. This was not a “mole” activity as we were talking about the mole – it was deleting data rather than disclosing data. It was an effort to eradicate evidence of the “mole’s” activity. It was not routine or ordinary course of business. The state of mind of the person would be entirely different.

        TRy to think about these things a little more.

        • Ryan O
          Posted Jul 29, 2009 at 8:47 AM | Permalink

          Re: thefordprefect (#11), Ah. I see. So you must ask for permission every time you follow a link. After all, there might not be an official page specifically directing you to the link in question. Man, at that rate, you’ll never get to the end of the internet!

        • Posted Jul 29, 2009 at 9:43 AM | Permalink

          Re: Ryan O (#19),

          I’ve seen the end of the internet, it looks a lot like the middle.

          If these are the HadCrut files we’re discussing, I bet there’s some PhD over there beating themselves like Dobby the house elf.

        • Posted Jul 29, 2009 at 9:57 AM | Permalink

          Re: thefordprefect (#11),

          while I’m here the directory in question is not that linked to in the data link – up 1 directory and into a private area (foolishly unprotected).

          Then it wasn’t private, was it.

    • henry
      Posted Jul 29, 2009 at 7:24 AM | Permalink

      Re: thefordprefect (#5),

      Read again:

      “alter, deface, block, erase, destroy or conceal any record held by the public authority, with the intention of preventing the disclosure by that authority of all, or any part, of the information to the communication of which the applicant would have been entitled.

      With the FOI still pending, it has yet to be decided if Steve (or any other “non-academic”) is entitled to the info.

      Until that’s decided, what right does Jones have to “conceal any record held by the public authority, with the intention of preventing the disclosure” to anyone.

      If the data was freely available on Jones’ own FTP site, without a restriction of the academic/non-academic clause, how can he delete ANY files, or even say that the restriction was valid.

      I guess we’ll just have to wait and see how the FOI from the academics are handled, and see what new hoop one has to jump through to get the data.

    • Geckko
      Posted Jul 29, 2009 at 7:32 AM | Permalink

      Re: thefordprefect (#5),

      would have been entitled

      Past imperfect.

  6. Pierre Gosselin
    Posted Jul 29, 2009 at 7:14 AM | Permalink

    Thanks to fordprefect
    for bringing the Computer Misuse Act to our attention. That rotten smell coming from Denmark just keeps getting.

  7. Pierre Gosselin
    Posted Jul 29, 2009 at 7:14 AM | Permalink

    …stronger!

  8. Posted Jul 29, 2009 at 7:26 AM | Permalink

    No folder named “censored” this time? 😮

  9. EddieO
    Posted Jul 29, 2009 at 7:27 AM | Permalink

    FoRe: Ryan O (#6), FordP
    Why do you object to this data being audited by Steve and others, who contrary to the propaganda put out by climate alarmists have little vested interests in the results? The international community is on the verge of setting out on a hugely expensive exercise which could be based on some very bad science and/or statistics. It is imperative that the data be made available to the public and to independent analysts if the whole exercise is to gain credibility.

    Steve has shown in the past what happens when published authors don’t release their data, or their methodology, in the field of climate change. Finding flaws in a)the data, b) the analysis or c) any post processing is in everybody’s interest.
    Similarly finding that the data confirms that climate change is as bad as the public are being lead to believe surely can not be a bad thing as it would bring sceptics into line with the IPPC’s procrastinations?

  10. DG
    Posted Jul 29, 2009 at 7:57 AM | Permalink

    This works quite well but you must have it forwarded to an email address to monitor daily otherwise changes could be missed.
    H/T Lucia

    http://www.changedetection.com/

  11. Posted Jul 29, 2009 at 7:59 AM | Permalink

    Anyone checked Alinksy’s Rules for Radicals lately?

    RULE 4: “Make the enemy live up to its own book of rules.”

    RULE 5: “Ridicule is man’s most potent weapon.”

    RULE 6: “A good tactic is one your people enjoy.”

    What a bunch of raging lefties there must be in the climate-skeptic community! But don’t forget:

    RULE 7: “A tactic that drags on too long becomes a drag.”

  12. Jimmy Haigh
    Posted Jul 29, 2009 at 8:52 AM | Permalink

    Pierre Gosselin:
    July 29th, 2009 at 7:14 am

    “That rotten smell coming from Denmark just keeps getting… stronger!”

    That will be Esbjerg…

  13. George Tobin
    Posted Jul 29, 2009 at 9:01 AM | Permalink

    This may be breaking some new ground in the UK. My impression is that there is not wide use of FOI requests there. The law is fairly recent (2000) and judging from the clumsy response in this case, it is a new experience for some agencies. A silly kneejerk reference to national security, moving or deleting files … somebody may need to learn that it’s not the crime, it’s the coverup.

    What could possibly motivate this? Mere puerile hostility to non-believers or has there been a discernible Hansenesque massage of historical data?

  14. Scott
    Posted Jul 29, 2009 at 9:11 AM | Permalink

    Not to throw a mundane reason into the mix, but it is quite possible that since the data in question is over 5 years old, that it simply passed the date of their data retention policies and was removed during the course of regular business. That it wasn’t removed closer to the 5 yr mark (if that’s their retention policy) could be chalked up to someone doing a poor job of data clean-up.

    • Steve McIntyre
      Posted Jul 29, 2009 at 10:20 AM | Permalink

      Re: Scott (#22),

      it simply passed the date of their data retention policies and was removed during the course of regular business.

      Not a chance. There’s no such data retention policy: many files that were considerably older were left untouched.

      Why these files? Why now?

      • Scott
        Posted Jul 29, 2009 at 10:31 AM | Permalink

        Re: Steve McIntyre (#27),
        Hey, I was just trying to give them some credible excuse… not my fault if they botched it all up. 🙂

  15. Jeremy
    Posted Jul 29, 2009 at 9:43 AM | Permalink

    @TheFordPrefect

    Why is this “commercial data with a value attached” if this is information to which the FOI is applicable? That sounds mutually exclusive. How can the FOI be applied to information that has commercial value? If the FOI is applicable, shouldn’t it essentially be nearly public domain at that point and therefore ‘value-less’?

  16. Craig Loehle
    Posted Jul 29, 2009 at 10:12 AM | Permalink

    The data could have commercial value for weather forecasting companies, but ONLY for the most recent few days. It boggles the mind that they have ever sold any data more than a week old. Who would buy it? Surely it only has academic value.

  17. Kenneth Fritsch
    Posted Jul 29, 2009 at 10:24 AM | Permalink

    This episode gets better and funnier with every turn – the records of confidentiality agreements get lost, the silly responses to FOI requests and now this. The removal of CRU files would indicate to me that the CRU people are quite sloppy with their work and that the listing of these files now removed was inadvertent. An organization being run by an absent-minded professor(s) comes to mind with a lead role for Peter Sellers.

    I would be remiss if I did not also acknowledge the efforts of thefordprefect for playing straight man for some rather humorous responses. His fuzzy thinking and wording play well into the bureaucratic maze of confusion being exposed here.

    • bernie
      Posted Jul 29, 2009 at 10:44 AM | Permalink

      Re: Kenneth Fritsch (#28),
      No, I beg to differ. Mr Bean is clearly in charge. Look what happened to Whistler’s Mother. “Be afraid, be very afraid…”

  18. Jimmy Haigh
    Posted Jul 29, 2009 at 10:29 AM | Permalink

    Ford Prefect wants the answer to be 42.

  19. Robinson
    Posted Jul 29, 2009 at 10:58 AM | Permalink

    This is rump-smackingly insane. What has Phil Jones got to hide? Perhaps some major incompetence or fraud? It’s always possible that this is simply Kafkaesque bureaucratic stupidity, but really, I’m having trouble seeing what the motive behind this kind of behaviour would be. Apart from incompetence or fraud (a bit harsh, I suppose), the only other motivation I can think of is that CRU gets advantage from being the only institution with this data, specifically in terms of leverage for those working on it. They get to publish, you see and that being the case, are the only ones able to offer advice on the relative merits or otherwise of keeping it going and investing in it (research grants). Whatever the case, I see this is only going to end one way: CRU and Phil Jones embarrassment.

    • Ryan O
      Posted Jul 29, 2009 at 11:17 AM | Permalink

      Re: Robinson (#33),

      What has Phil Jones got to hide?

      .
      The sad thing is . . . probably nothing. I can’t imagine anything CRU does to the data would be more arcane than GISTEMP, and GISTEMP certainly didn’t get pummeled when Hansen finally released it. That’s what makes this all the more laughable.

    • Dave Andrews
      Posted Jul 29, 2009 at 2:31 PM | Permalink

      Re: Robinson (#34),

      Surely the worry here for the Met Office and CRu is the pivotal role they play in the IPCC process. If they are shown to be wrong in any respect they perhaps fear the whole edifice will start to crack.

  20. Posted Jul 29, 2009 at 11:14 AM | Permalink

    So if they don’t answer what is basically your request to delete the data for them, will you be planning to then publicly host the data for them so all scientists can get a copy? After all, I’ve seen no claims of copyright.

    Perhaps you would consider it as a free service to HadCrut. If they don’t reply at all, then I assume they want you to host it.

  21. Gerald Browning
    Posted Jul 29, 2009 at 11:15 AM | Permalink

    Steve McIntyre,

    I point out my comment (#150) from the second mole thread. Also see my response to NW’s comment #153
    on that same thread.

    Jerry

  22. Posted Jul 29, 2009 at 11:25 AM | Permalink

    Files for downloading are in “people” “projects” and “data”.

    ( ftp://ftp.cru.uea.ac.uk/welcome.msg )

    But ‘people’ is private area, don’t go there 😉

  23. Posted Jul 29, 2009 at 11:57 AM | Permalink

    Met office losing data? I thought I ought to check what is happening over here with the averaged and adjusted Central England Temperature. The data is disappearing for this year. It seems they are having to average it from one station only, and getting a very different result from the raw data.
    Rather than intrude on your grief here I have explained more over at my Castle….

    The data is linked from Maximum Central England Temperature, 2009

    The pale dotted line in the middle is this year (click to enlarge) – Phew what a scorcher! Look at that late June peak, way above average!

    Where was it? We know this record is a blend of records from apparently three stations, and their records are also graphed on the same page:

    Can you spot the problem? We’ve lost Pershore and Stonyhurst is disappearing. This leaves Rothamsted which is cool on the whole idea of a June heatwave…

  24. Pierre Gosselin
    Posted Jul 29, 2009 at 12:20 PM | Permalink

    Gekko 15
    “would have been entitled”
    Past imperfect? Are you sure about that?
    I’d say it’s the Conditional II in the passive voice.
    I’m not aware of any past imperfect tense in the English language. There’s an imperfect, also known as the past continuous. Ex: Steven was searching for some data when the files disappeared.
    I’m sure I would have heard of it by now if there had ever been such a thing.

  25. Posted Jul 29, 2009 at 12:20 PM | Permalink

    RE The Englishman, #41, As quoted on your webpage, your data source clearly states,

    Please note that the Met Office data sets are available for bona fide academic research only..

    Are you legally authorized to examine it or discuss it?? This sounds like another security breach involving the Met Office!

  26. Bob McDonald
    Posted Jul 29, 2009 at 12:37 PM | Permalink

    Pssst! Hey….I’ve got a copy of yesterdays weather report…..how much will you give me for it?

  27. steven mosher
    Posted Jul 29, 2009 at 12:44 PM | Permalink

    Ford just to be clear so you dont try to confuse things. I believe in AGW ( GHGs warm the planet). The warming exhibited in the record is a result, on my view, of UHI and GHG. If you are a warmer, you typically believe that UHI has all been accounted for by “adjustments” ( see peterson,parker,hansen, jones). Lukewarmers, like me, believe that UHI still infects the record. We would take Ross’s studies as our primary source. We would tend to argue for lower climate sensitivities.
    hence, our interests are the state of the “observations” ( temp records, proxy studies) and the use or misuse of computer models. Got it?

  28. Posted Jul 29, 2009 at 12:50 PM | Permalink

    I just got a letter from David Palmer indicating that he will consider my request for the CRU data under the Environmental Information Regulations 2004 (EIR), and that FOIA therefore does not apply.

    The statutory timescale is 20 working days.

    • Jonathan
      Posted Jul 29, 2009 at 2:18 PM | Permalink

      Re: Hu McCulloch (#50), I’ve had no reply other than an out of office message on Friday. I’m feeling left out 😦

  29. Gerald Machnee
    Posted Jul 29, 2009 at 12:53 PM | Permalink

    I suspect that the files are more likely hidden than completely deleted. I had requested some reports through FOI and was told that they were lost due to a computer malfunction. I then filed an FOI to get details of the computer malfunction. I never got the malfunction report, but they managed to locate a hard copy of my first request.

  30. Pierre Gosselin
    Posted Jul 29, 2009 at 1:00 PM | Permalink

    SM 42
    Can’t get any clearer than that.

    Jeff Id
    You’re right.
    No copyright + no confidentiality = no unauthorised use.

    ford
    “childish and dangerous pranks”
    Hyperbole. There’s nothing wrong with checking over public data that is uncopyrighted and not regulated by confidentiality agreements. If genuine agreements materialised, Steve would destroy the data. There’s really nothing to be so uptight about.
    (You’ll note I used a Case II if-then clause, implying an improbable condition).

  31. henry
    Posted Jul 29, 2009 at 1:02 PM | Permalink

    At present the data set includes series of nine climate variables: daily minimum (TN), maximum (TX) and mean temperature (TG), precipitation amount (RR), sea level air pressure (PP), snow depth (SD), relative humidity (RH), cloud cover (CC) and sunshine duration (SS).

    About 52% of the series is publicly available from the ECA&D website. The other 48% comes with restrictions: these series
    are for ECA&D indices calculations and gridding purposes only.

    Read this again – to me, it appears the that the data restriction applies to the USAGE of the data, not the DISTRIBUTION of the data.

    Doesn’t say that you can’t have it, just that its primary use is for their internal calculations.

    • steven mosher
      Posted Jul 29, 2009 at 2:35 PM | Permalink

      Re: henry (#52),

      Yes henry. people should also recognize that TMIN and TMAX are covered and TG ( mean temp) but not, it appears, Tave. In the climate papers I am used to reading and the data sources I’m used to looking at you see these things.

      For stations, like ASOS, where data is taken hourly you get TMIN,TMAX and TG, where TG is calculated from integrating over time. these figures are reported daily. For other stations using min max thermometers you get Tmin and tmax daily. From this data, Tave is calculated. (tmin+tmax)/2.

      Then you have monthly data, where Tave is averaged over the month.

      So, it is possible for a commercial entity to give me TMIN and TMAX and TG on a daily basis and restrict me from publishing this data. they may also let me calculate TAVE on a monthly basis and redistribute that calculated figure.

      If and when we get a look at the agreements it will allow us to refine our FOI accordingly. Obviously, we would love the daily data, but daily data does have value to the agriculture market ( calculating chill days, growing days, etc etc). anmd you want daily data to see how they do their QC and infilling for missing days.. and for correlation studies, but
      in a pinch monthly Tave will do.

  32. TAG
    Posted Jul 29, 2009 at 1:08 PM | Permalink

    Re:thefordprefect (#40),

    Will this affect the other users of commercial dat for research – who knows. I suppose if this happens you will have accomplished your main desire of destroying any proof the GW is occuring.

    Let contemplate the possibilities

    A weather service is put put because of this. It refuses to release its data and stubbornly sits on it. AGW research is badly handicapped Other governments plead for the release of the data but the service is adamant. Our data is at risk and so we will not release it.

    Someone sends the $20 to buy the data

    The problem is solved.

    • steven mosher
      Posted Jul 29, 2009 at 2:48 PM | Permalink

      Re: TAG (#53),

      Haha. Ford is so clueless I don’t know where to begin. But lets play along. there are two widely used global temperature records. GISS and CRU. On ford’s analysis I am destroying CRUs ability to get data and create their record.
      I am doing so (according to him) because I want to remove proof of GW. But what ford doesnt get (because he does not pay attention to details — calls them twaddle and waffle) is this.

      1. CRU and GISS have a very large overlap in data sources. If CRU goes away we are left with GISS.
      2. GISS tends to higher averages for global temps than CRU because of the way they handle estimate
      of data at the poles.

      So, yes, I have in the past argued that CRU should not even be looked at as a credible piece of work because it has closed data and closed source files. end of story. If that means we are left with GISStemp as the record, so be it.
      Does Gisstemp indicate a warmer planet? yup. so much for me removing proof of GW. quite the opposite.

  33. bob
    Posted Jul 29, 2009 at 1:09 PM | Permalink

    lol lol lol

    this is just getting more and more ridiculous.

    Do you seriously think the computer misuse act covers them deleting (or perhaps moving) their own files from a public ftp server?

  34. Steve McIntyre
    Posted Jul 29, 2009 at 1:59 PM | Permalink

    Given the possibility (even probability) that CRU have moved the data into private directories (or even offline), I will qualify some sentences in the above post to reflect this possibility.

    • Basil
      Posted Jul 30, 2009 at 6:31 AM | Permalink

      Re: Steve McIntyre (#58),

      Given the possibility (even probability) that CRU have moved the data into private directories (or even offline), I will qualify some sentences in the above post to reflect this possibility.

      The situation may not rise to requiring this level of caution, but just because a datestamp is preserved when files are moved doesn’t assure that they were not changed. Forensically, datestamps are essentially worthless. If a hash of the the moved data wasn’t performed first, then we really have no assurance that the “moved” files are the same as the originals.

  35. Reid
    Posted Jul 29, 2009 at 2:10 PM | Permalink

    What we have here is a perfect example of flummery.
    For those unfamiliar with the British term, flummery is defined as – meaningless or deceptive language; humbug.
    It’s time to take off the gloves and accuse CRU and Jones of engaging in flummery.

    • theduke
      Posted Jul 29, 2009 at 2:41 PM | Permalink

      Re: Reid (#57),

      We call it “flim-flammery” over on this side of the pond. Same meaning.

    • Gene Nemetz
      Posted Jul 29, 2009 at 4:04 PM | Permalink

      Re: Reid (#57),

      Ok. But what took you so oong?

  36. Your Mom
    Posted Jul 29, 2009 at 2:13 PM | Permalink

    In spite of the missing data, we certainly know what kind of people they are now.

  37. Steve McIntyre
    Posted Jul 29, 2009 at 3:14 PM | Permalink

    It’s interesting to re-examine the responses to Willis’ requests here:

    Click to access cru.correspondence.pdf

  38. Neil Hyde
    Posted Jul 29, 2009 at 2:21 PM | Permalink

    Steve,

    Slightly O/T , but should you decide to appeal to the Information Commissioner, I live about three miles from the office, and pass by there every morning/evening on my way to and from work. I am more than willing to hand deliver, and take a receipt for anything you would wish to lodge with them.

  39. Hoi Polloi
    Posted Jul 29, 2009 at 2:47 PM | Permalink

    So now we know we have data and next thing we have not, and there’s a LOT of buzzing around.

    BUT, do we know if it really matters, the data, do we? OR, is it all a lot of wind flowing around from one hole to another.

    As far as I see it, there’s a lot of messages about a “Mole” whch apparantly is a website with data for everybody available. I did some googling (hadley+cru+data) and came past the same website in a few minutes: (ftp.cru.uea.ac.uk/data/)

    SO, can we finally get to the point if it (the DATA) really matters, or are we to continue the Deep Cool Mole Show, fercrissakes.
    Too many red herrings swimming round, alresady.

  40. Mike B
    Posted Jul 29, 2009 at 2:52 PM | Permalink

    Steve,

    Thanks very much for a most enjoyable diversion for the past two days. As luck would have it, a particularly distasteful project at work had left me much in need of a chuckle or two. And your “mystery” has been a welcome relief.

    And also a special “shout out” to “thefordprefect”, whose pitch-perfect combination of righteous indignation and over-the-top dramaqueenery has provided numerous laugh-out-loud moments.

    As always, Steve, you’ve been able to make an important, if somewhat mundane, topic much more interesting with your wit. Well done all around.

  41. Paz
    Posted Jul 29, 2009 at 3:03 PM | Permalink

    I don’t get it. Once a FOI request is made, they cannot change what they put on their FPT servers? This a very weird interpretation. As long as the data is *somewhere* and still accessible, I presume they have all the right to administer they FTP servers as they see fit.

    • steven mosher
      Posted Jul 29, 2009 at 3:25 PM | Permalink

      Re: Paz (#67),

      Paz, the point is that it is not advisable to make changes to any documents, especially their storage, that could fall under the request of an FOI. CRU was put in a tough spot.

      1. an FOI request was made that this data be supplied to Willis ( in 2003) and others more recently.
      2. those requests were denied. At the time the requests were made the office in charge of the request
      should have demanded all the records from Jones. Who he got the data from, contracts associated with it,
      who he had supplied it to, where he kept it. etc. For example, when I have been disposed on IP related
      cases I supplied the legal department with all my papers, notes, files, floppies, backed up disks.. everything.
      If CRU had done a proper investigation they would have found these files posted on a public FTP. And, they would
      have to inform the requesting parties of this and informed the people who supplied the data.

      3. Once they found out that the data was public they had these choices.

      A. Move the files and destroy evidence that they had been public. Remember, they had no knowledge that
      we had done screen caps of the ftp site.
      B. Leave the documents there, preserving the evidence but endangering their commercial partners.
      C. Add a notice to the site or password protect the files from being downloaded. This would preserve
      the rights of the data owners and the evidence.

    • steven mosher
      Posted Jul 29, 2009 at 3:38 PM | Permalink

      Re: Paz (#67),

      “I didn’t see any files on any ftp site. somebody claiming to be connelly mailed me the data. what ftp site? what data? ”
      once Jones removed the data, there was the option of playing this game. That would have been mean and dishonest.

  42. Keith
    Posted Jul 29, 2009 at 3:06 PM | Permalink

    Wow, I was just exploring the CRU ftp site. Phil Jones keeps a PDF copy of Santer et al. (2001) in his personal folder on the site.

  43. Keith
    Posted Jul 29, 2009 at 3:10 PM | Permalink

    It also looks like they delete some files from the projects directory at ftp://ftp.cru.uea.ac.uk/projects/advance10k/ . The projects directory shows the file changed on 7/29/2009, but they is no file with a that date in the directory. Looks like the “mole” has lead to a lot of record transference.

  44. bluegrue
    Posted Jul 29, 2009 at 3:15 PM | Permalink

    Let’s take a step back. Fact is, the files in question have been available on an unprotected FTP server. Does that mean they were meant for public access? Not necessarily. There are at least four possible causes:
    1) They were meant for being publicly available. Unlikely IMHO, given the MetOffice refusal.
    2) Error on behalf of a computer admin, who failed to protect the directories within the /people directory. Remember, the public links go to /data, which /people is not part of.
    3) Dr Jones mistakenly thought the directory was not publicly accessible.
    4) The directory used to be private and was made public later on, without Dr. Jones being aware of either the change or the data he put there.

    Ask yourself, if you forget to lock your flat, are strangers entitled to enter your flat, look through your belongings and take photos and leave again, without taking anything but pictures? It’s not a fair analogy, but it may get you thinking.

    So what really happened? Let me speculate.

    MetOffice is telling the truth, they are not allowed to publish the data. Unlike many posters here I do believe that data is given for restricted usage on word of honor between scientists who know each other, across institutions, without a paper trail. After all it works here, as Steven Mosher professed on the last thread, why do you expect it not to happen elsewhere? Next Steve finds the FTP access and starts to write blog posts, echoed by Anthony. Dr. Jones becomes aware of the gaffe, that data given into his safekeeping are publicly available. He must move the data away from public access. This explains all of the changes observed on the FTP server. Whether this leak will result in permanent damage to the MetOffice’s access to data remains to be seen, it will evolve over the next months. If anyone was expecting special effects or instant consequences, that person has obviously seen too many movies.

    Now I have a question to Steve:
    Why did you decide to turn the affair into a mystery show for your readers. IMHO, you have squandered the chance of your lifetime to demonstrate in action that you are not simply out to publicly embarrass the MetOffice. You could have done the good deed, taken all the evidence you want in silence and contacted Dr. Jones and/or the MetOffice notifying them that the very data you sent a FOI request for is publicly available, suggesting they might want to take it down before too many people notice. This would have greatly improved your standing with the MetOffice. You would also have given them a reason to trust you. If they had decided to “play foul”, you still could have outed them. So why did you turn it into a juicy story for your readers instead?

    I’m aware, that I not simply asked a question, but also included my negative view on how you handled this. Feel free to snip the question part and/or refuse to answer, I will not complain.

    • Greg F
      Posted Jul 29, 2009 at 4:07 PM | Permalink

      Re: bluegrue (#71),

      Ask yourself, if you forget to lock your flat, are strangers entitled to enter your flat, look through your belongings and take photos and leave again, without taking anything but pictures? It’s not a fair analogy, but it may get you thinking.

      The presumption on somebody’s residence is that you don’t enter without permission. The presumption on the Internet is if the site is accessible to the public it is public. The analogy is not only not fair, it isn’t really an analogy.

      • bluegrue
        Posted Jul 29, 2009 at 4:23 PM | Permalink

        Re: Greg F (#77),
        how many FTP sites with data intended for the public have you seen, that are organized by /people/name1, /people/name2, … in addition to a /data or /pub section or a /lib section exposed to the public? From my limited experience, this is an organizational structure for internal data. Looks like the admin simply put the root of a unix/linux system up for public FTP access, be it indifference, negligence or incompetence. Judging by the screen shots it could well be, that the supposedly private user accounts have been exposed.

        • Greg F
          Posted Jul 29, 2009 at 7:19 PM | Permalink

          Re: bluegrue (#82),

          It is an FTP server. The purpose of an FTP server is to share files with people on the internet. There is no reason at all to use FTP internally to store or share files. None. Nobody in their right mind would keep their personal folders on a FTP server. Also, when you upload or download to FTP it simply creates a copy.

          “Personal folders” on a FTP server is not unusual, I see it quite often. The FTP server should reside in the DMZ and should not be part of the internal network. Just because you can see the folders does not mean they are “exposed to the public”. As far as being the root, it may be. Although, there are clearly folders that would be in the root that are not present. The ones that are there may very well be there for remote administrative reasons. Accessible to those who have the appropriate permissions.

          Judging by the screen shots it could well be, that the supposedly private user accounts have been exposed.

          Judging by the volume of files in the personal folders it is unlikely those are their “private user accounts”. In fact one account (craigwallace) has a grand total of one document in his folder. Does that sound like a “private user account” to you?

    • Gerald Machnee
      Posted Jul 29, 2009 at 4:20 PM | Permalink

      Re: bluegrue (#71),

      This would have greatly improved your standing with the MetOffice. You would also have given them a reason to trust you. If they had decided to “play foul”, you still could have outed them. So why did you turn it into a juicy story for your readers instead?

      It does not work that way. Just read Steve’s post in # 70.
      Jones has NO intention of cooperating.

    • pete m
      Posted Jul 29, 2009 at 4:28 PM | Permalink

      Re: bluegrue (#71), with respect, Steve has a good track record of notifying people / organisations of errors. Who are you to say this was not intentionally left to open access? Further, re your analogy, perhaps a better one would be if I gather 100 notes from various musicians and put it into a song, then post it on my front door as allowed material to view and inspect. Can I then complain if someone with a good memory writes it down later and inspects their version, rather than rely on my version? Perhaps even if that person writes a critique of my song structure?

      At the end of the day, the real issues are:

      Why is not the most crucial, urgent scientific debate taking place in the world today not required to also be the most open? Particularly when so much of our economic well being is to be put at risk because of it.

      Why is the most basic mundane data going back centuries not made public?

      Why are studies published and relied upon when they cannot be verified?

      Why are governments allowing public funds to be used to gather this data and then not be made available to the public?

    • Kenneth Fritsch
      Posted Jul 29, 2009 at 4:40 PM | Permalink

      Re: bluegrue (#71),

      Ask yourself, if you forget to lock your flat, are strangers entitled to enter your flat, look through your belongings and take photos and leave again, without taking anything but pictures? It’s not a fair analogy, but it may get you thinking.

      I think we have perhaps here a better straight man than prefectford. Actually the above comment and others in this post are funny unto themselves with no replies required. Bluegrue, if you have more material keep it coming.

      Why did you decide to turn the affair into a mystery show for your readers. IMHO, you have squandered the chance of your lifetime to demonstrate in action that you are not simply out to publicly embarrass the MetOffice.

      Did you ever consider that the CRU people have embarassed themselves and Steve M has only provided opportunity and a public forum for doing so. Why are you so interested in keeping all this hush hush? Do not we need to know what kind of stuff are public servants are made of and is not the internet an excellenmt tool for doing just that? Long live the internet and the users like Steve M who put it to its best use.

    • TerryS
      Posted Jul 29, 2009 at 5:20 PM | Permalink

      Re: bluegrue (#71),

      Let’s take a step back. Fact is, the files in question have been available on an unprotected FTP server. Does that mean they were meant for public access? Not necessarily. There are at least four possible causes:

      [snip]

      Ask yourself, if you forget to lock your flat, are strangers entitled to enter your flat, look through your belongings and take photos and leave again, without taking anything but pictures? It’s not a fair analogy, but it may get you thinking.

      Your example should start: “Ask yourself, if you forgot to lock your flat and left a sign on the door saying free public access…”

      It is irrelevant what CRU’s or Phil Jone’s intentions were. The simple fact is they put the data on a public site in a public folder in a publically readable file. That means Steve’s hands are clean. Any consequences are due to CRU/Phil Jone’s incompetence and not Steve’s actions.

      If I recall correctly SCO accused IBM of illegally accessing their anonymous ftp site and downloading files in their recent court case. They didn’t get very far with the accusations.

      All this does not necessarily mean that Steve can redistribute the file since there may be copyright restrictions for example. It does mean that any third party that put restrictions on the data being redistributed can only hold CRU/Phil Jones liable for making it public and not Steve for downloading it. Furthermore, unless CRU/Phil Jones avail themselves of Steve’s offer they might also be held liable for Steve redistributing it.

    • Steve McIntyre
      Posted Jul 29, 2009 at 5:32 PM | Permalink

      Re: bluegrue (#73),

      bluegure, if CRU believes that I am in “wrongful” possession of data that they placed in the first tier of their public data and provide the notice to me described in my prior post, then for a limited time (contrary to the wishes of many readers) I’ve said that I would destroy the data and return to square one, relying entirely on FOI procedures.

      Thus far, they haven’t asked.

      While a blog thread would not constitute legal notice, in any practical sense, there is overwhelming evidence that they are following blog events here and cannot pretend that they were unaware of this offer.

    • Ryan O
      Posted Jul 30, 2009 at 7:41 AM | Permalink

      Re: bluegrue (#73),

      Ask yourself, if you forget to lock your flat, are strangers entitled to enter your flat, look through your belongings and take photos and leave again, without taking anything but pictures? It’s not a fair analogy, but it may get you thinking.

      Let us think of this in terms of a fair analogy, then.
      .
      Putting something on an anonymous FTP server is like putting an old couch on the side of the road with a sign on it that says “FREE!”
      .
      Anonymous FTP servers have one purpose: dissemination.

  45. Mark_T
    Posted Jul 29, 2009 at 3:26 PM | Permalink

    This is frigging BS and only makes me more suspicious of their work and motivations and I’m sure I’m not the only one that feels this way.

  46. RomanM
    Posted Jul 29, 2009 at 3:34 PM | Permalink

    I was milling around the CRU ftp website looking for the mole when I ran into a “censored” directory.

    The two files were not accessible, and I don’t think that they are necessarily related to the events of this thread, but I found their names amusing.

    • Raven
      Posted Jul 29, 2009 at 8:05 PM | Permalink

      Re: RomanM (#76),
      The choice of the ‘censored’ folder name makes it highly probable that it is intended as a dig at Steve Mc.

      The date stamp means nothing since any sysadmin can set the time to anything that they want.

      • theduke
        Posted Jul 29, 2009 at 8:17 PM | Permalink

        Re: Raven (#104),

        Yes, exactly. A file named “youstupidbastard” isn’t exactly one you’d expect to find at a distinguished institution like CRU.

        Or is it?

  47. AndyL
    Posted Jul 29, 2009 at 4:09 PM | Permalink

    I don’t think everyone on here has considered the impact of this story in UK.
    .
    Following various losses of data, the UK public are becoming very concerned about how securely public bodies keep sensitive data. The government are trying to create a massive database containing all sorts of private data, including individuals’ bio-metrics, and opposition is growing as people become no longer trust the government to keep their data secure.
    .
    If it turns out that the Met Office claimed that making this data available would damage international relations, and then left the data on a publicly accessible server, it would create huge embarrassment. A lot of journalists would be interested in the story
    .
    Perhaps Steve should consider putting out a press release?

  48. Kenneth Fritsch
    Posted Jul 29, 2009 at 4:20 PM | Permalink

    From a business perspective I could understand a commercial interest paying for reanalysis climate data and climate forecasts providing there were not reasonably reliable public sources of the same data. Those areas of exploitation would make much more business sense than for historical temperature and some other types of climate data.

    ECA&D reports doing forecasts but I saw nothing on reanalyses. ECA&D did note that 6 papers used their data exclusively. I think that the use of CRU temperature data might be considered in the same vein by CRU, i.e. a matter, not of financial reward, but of academic pride. Of course without more details from these organizations, this is all conjecture like other offerings here on this thread.

    One thing remains and that is that CRU did not use commercial exploitation as a reason for not fulfilling the Steve M FOI request nor has anyone shown evidence that they have ever used that reason in such denials. I know that some scientists can be very opposed to commercial exploitation in any area, but not using what would appear to be a rather legitimate reason for denying requests (as opposed to the silliness that was used) would seem to carrying that silence or embarrassment a bit far.

  49. Jean S
    Posted Jul 29, 2009 at 4:22 PM | Permalink

    If Willis needs to prove that those newcrustn* files were publicly available during his FOI, here it is: WayBackMachine index from Dec2005. Unfortunately, only the .hdr file is in archieve.

  50. Steve McIntyre
    Posted Jul 29, 2009 at 4:47 PM | Permalink

    I know a little about the Team moving data in and out of public directories.

    People may recall the history with Mann’s FTP site. I originally asked Mann for the location of his proxy data. He said that he’d forgotten but it was around somewhere and Scott Rutherford would locate it. I nagged Rutherford and he eventually provided me a URL for the data. We noticed problems with the data (especially principal components) and sent the entire dataset to Mann asking him to confirm that this was the data used in MBH. He said that he was too busy to reply and don’t call me, I’ll call you.

    After we published MM2003, Mann deleted the data set to which we’d been directed, a new data set materialized in his public directory (the only such data set of all his publications) – he said that we’d used the “wrong” data set and, had we not been so stupid, we would have used the “right” data set, now available in his public directory. He pretended that the data that suddenly materialized had been public all along – pointing to the date (which proved nothing about when the files were put on the public directory).

    The “Community” thought that this was a magnificent putdown of the interlopers.

    Mann then untruthfully said that we had asked for an Excel spreadsheet and errors had been made in preparing the spreadsheet for us. I had marked down the date of the files to which we had been directed just before Mann deleted them and they were long prior to our request. Unfortunately I didn’t take a screenshot. I had retained my email requests, which clearly did not ask for an Excel spreadsheet – as CA readers know, that is entirely inconsistent with my methods anyway. But the Community took Mann’s word for it.

  51. steven mosher
    Posted Jul 29, 2009 at 4:58 PM | Permalink

    Click to access 1.2750!uea_manual_draft_04b.pdf

    here is a good read

    • David Holland
      Posted Jul 29, 2009 at 5:16 PM | Permalink

      Re: steven mosher (#90),

      I think you mean http://www.uea.ac.uk/polopoly_fs/1.2750!uea_manual_draft_04b.pdf

      Its interesting but it does not mention the Environmental Information Regulations.

      Does anyone here think CRU has anything approaching an easily accessible systematic means of disseminating the environmental information they hold? By law since February 2005 they should have been progressively making THE information available, not just what they fancy.

    • David Holland
      Posted Jul 29, 2009 at 5:19 PM | Permalink

      Re: steven mosher (#90),

      My apologies. The editor mishandled the link for me also!

      • steven mosher
        Posted Jul 29, 2009 at 8:26 PM | Permalink

        Re: David Holland (#94),

        No worries. I haven’t had time to read the procedures, but it would appear looking at the responses to Willis and others that the rationale for not releasing the data has been shifting. One wishes that CRU and Jones had merely followed the dictates of science and made data and methods available. It’s tedious to hold them to the dictates of law, but if that is what it takes, then that is what it takes.

    • Geoff Sherrington
      Posted Jul 31, 2009 at 7:33 AM | Permalink

      Re: steven mosher (#89),

      Try that URL now and you’ll be asked to delete all cookies including “cru” before the site will open for you. Is this a recent change too? At least that’s what it does on my system.

      Click to access 1.2750!uea_manual_draft_04b.pdf

  52. David Holland
    Posted Jul 29, 2009 at 4:59 PM | Permalink

    It is very unlikely that any information has been permanently deleted. I assume the data was information on the states of the elements of the environment. In which case Regulation 4 is a legal requirement.

    Environmental Information Regulations 2004
    Dissemination of environmental information Regulation 4. –
    (1) Subject to paragraph (3), a public authority shall in respect of environmental information that it holds –
    (a) progressively make the information available to the public by electronic means which are easily accessible; and
    (b) take reasonable steps to organize the information relevant to its functions with a view to the active and systematic dissemination to the public of the information.

    It isn’t spelt out in the Regs but I think that, once a public authority has put up environmental information it can’t lawfully just take it down other than to update it or to maintain it or its indexing or if the public authority considers it obsolete. I think best practice in these circumstances would be to put up a temporary page explaining the reason and giving an email address for interested parties.

    However if it was environmental information, what has happened to it is also. So ask.

  53. verm
    Posted Jul 29, 2009 at 5:18 PM | Permalink

    bluegrue,

    Ask yourself, if you forget to lock your flat, are strangers entitled to enter your flat, look through your belongings and take photos and leave again, without taking anything but pictures? It’s not a fair analogy, but it may get you thinking.

    You don’t seem to have read this thread very well, in particular UC’s comment #39. The welcome file for the anonymous ftp site specifically refers to the “people” directory being for the downloading of files. thefordprefect seems to also owe an apology.

    • bluegrue
      Posted Jul 30, 2009 at 1:35 AM | Permalink

      Re: verm (#92),
      Yes, I indeed missed #39. My apologies to Steve.

      At least I seem to have been good amusement to some others.

  54. RW
    Posted Jul 29, 2009 at 5:31 PM | Permalink

    Under the FOI act, requests can be refused if deemed vexatious or repetitive. Do you feel you might be in danger of being rejected on those grounds, before too long?

    • John M
      Posted Jul 29, 2009 at 6:39 PM | Permalink

      Re: RW (#96),

      Not that I want to speak for Steve Mc, but I believe that might fall under the category of “Go ahead, make my day.”

  55. Posted Jul 29, 2009 at 6:01 PM | Permalink

    Steve

    Have you written to them directly with your offer?

  56. Patrick M.
    Posted Jul 29, 2009 at 6:53 PM | Permalink

    There is also the possibility that the data was left public specifically so that if any FOI requests were actually honored, the Met Office and Dr. Jones could just say, “The data has been publicly available for a long time now. I don’t understand why we keep getting these FOI requests.” It would be opportunity for them to claim Steve was just too stupid to find the data that has been public all along. Unfortunately for them Steve finds the data on his own before they spring their trap and blogs about it.

    It’s another possible scenario to be considered.

    Just my $0.02

    • theduke
      Posted Jul 29, 2009 at 8:07 PM | Permalink

      Re: Patrick M. (#100),

      Given the history of attempts to obtain data by those associated with this site (see Steve’s post #72) the scenario you describe is plausible. It might be that they even got a little reckless so that they might be able to make in-house jokes at CA’s expense, which could, to speculate further, explain the hurried bit housecleaning that occurred yesterday.

  57. icman
    Posted Jul 29, 2009 at 8:04 PM | Permalink

    Greg F.

    Finally someone that makes sense. Nobody uses an FTP server for internal personal folders in the 5 large corps I have worked for. From Novell, Unix, Windows, Linux environments.

  58. steven mosher
    Posted Jul 29, 2009 at 8:34 PM | Permalink

    Oh crap.

    Well I’m starting down the path of seeing what CRUs policies are and records management manuals. In the doc
    I refered to above the second paragraph says:

    2. Purpose of Code. This Code of Practice provides guidance on the handling of
    requests for information submitted under the Freedom of Information Act 2000
    (‘the Act’). It supplements the UEA’s parallel Guidelines on the Management of
    Records under the Freedom of Information Act 2000, which provides guidance on
    the implementation of records management in the UEA.

    And when you click on the link to the guidelines….. you get a password protected intranet.

    Dang, these jokers can password protect their policies about managing records, while they let
    precious, internationally sensitive data sets wander about FTP sites.

    I like irony.

  59. tetris
    Posted Jul 29, 2009 at 8:46 PM | Permalink

    There never is smoke without some sort of fire…
    However, for a true conflagration to take place, fuel and oxidizer have to mix in the right proportions. Let’s hope that we will all soon find out whether the “oxygen” contained in SteveM’s coup causes full combustion of the [presumed] fuel in the CRU files.

    Steve: I’ve never suggested that I expect anything remarkable in the CRU files. I’ve discouraged such speculation repeatedly. Please stop making such speculations here.

  60. steven mosher
    Posted Jul 29, 2009 at 8:50 PM | Permalink

    Ok.. I said this would be interesting reading. In my FOI to CRU I asked them for policies and procedures with respect to public employees entering into agreements either written or verbal and their responsibilities for record keeping. Why? because of these things: 1. CRU has stated that 98% of its data ( See prior correspondence) was already open.
    2. because there are open sources ( see hansen) that perform the same job. 3.) because they couldn’t tell Willis which Met services they had contracts with, 4. in case they claim they lost the contracts. In light of that I give you this:

    V Freedom of Information and confidentiality obligations

    31.

    Public authorities should bear clearly in mind their obligations under the Freedom of Information Act when preparing to enter into contracts which may contain terms relating to the disclosure of information by them.
    32.

    When entering into contracts with non-public authority contractors, public authorities may be asked to accept confidentiality clauses, for example to the effect that information relating to the terms of the contract, its value and performance will not be disclosed. Public authorities should carefully consider the compatibility of such terms with their obligations under the Act. It is important that both the public authority and the contractor are aware of the limits placed by the Act on the enforceability of such confidentiality clauses.
    33.

    The Act does, however, recognise that there will be circumstances and respects in which the preservation of confidentiality between public authority and contractor is appropriate, and must be maintained, in the public interest.
    34.

    Where there is good reason, as recognised by the terms of the exemption provisions of the Act, to include non-disclosure provisions in a contract, public authorities should consider the desirability where possible of making express provision in the contract identifying the information which should not be disclosed and the reasons for confidentiality. Consideration may also be given to including provision in contracts as to when consultation with third parties will be necessary or appropriate before the information is disclosed.
    35.

    Similar considerations will apply to the offering or acceptance of confidentiality obligations by public authorities in non-contractual circumstances. There will be circumstances in which such obligations will be an appropriate part of the acquisition of information from third parties and will be protected by the terms of the exemption provisions of the Act. But again, it will be important that both the public authority and the third party are aware of the limits placed by the Act on the enforceability of expectations of confidentiality, and for authorities to ensure that such expectations are created only where to do so is consistent with their obligations under the Act.

    I would think going forward that CRU should consider the following.

    1. Dropping data that is confidential OR
    2. Clearly marking and controlling it.

    Recall that when Willis asked for the MET agencies to contact, he could not get a straight answer. You would think that after being unable to answer his request that they would have got their act together. hmm. I wonder if they do an audit of their compliance with requests? hmm.

  61. thefordprefect
    Posted Jul 29, 2009 at 9:55 PM | Permalink

    Thank you for publishing the FOI comunications

    Click to access cru.correspondence.pdf

    Now I see the “try and find something wrong with it” comment in context it seems obvious it is a rather poor joke:

    Warwick,
    Hans Teunisson will reply. He’ll tell you which other people should reply. Hans is
    “Hans Teunissen”
    I should warn you that some data we have we are not supposed top pass on to others.
    We can pass on the gridded data – which we do. Even if WMO agrees, I will still not
    pass on the data. We have 25 or so years invested in the work. Why should I make
    the data available to you, when your aim is to try and find something wrong with it.
    There is IPR to consider.
    You can get similar data from GHCN at NCDC. Australia isn’t restricted there.
    Several European countries are. Basically because, for example, France doesn’t want the
    French picking up data on France from Asheville. Meteo France wants to supply data to
    the French on France. Same story in most of the others.
    Cheers
    Phil

    You do not finish such a devastating comment as:

    We have 25 or so years invested in the work. Why should I make
    the data available to you, when your aim is to try and find something wrong with it.

    With

    Cheers Phil

    Perhaps Phil does not do smilies :o)
    “Phil” then goes on to explain that IPR is the problem and a way to get round this problem. The Meteo France comment is a bit of humourous (antanaclasis?) dig at the French.

    France doesn’t want the French picking up data on France from Asheville. Meteo France wants to supply data to the French on France.

    Note again no smilies.

    I also note that you were frequently told that it was an IPR problem very early on. But you kept pushing for data that should not be provided because it is not CRU IPR but the data providers IPR.

    Perhaps Phil put the data ouside the secure zone because he did not agree with witholding it. He certainly should not have told you it was there – that would be admitting it was not an accident. To remove it now is theonly logical action.

    I know you consider this to be having a little bit of fun at Jones’ cost. But have you considered how moch a few hundred FOI requests are going to cost a small university (who are probably struggling financially). Its probably equivalent to tutoring a couple of undergraduates for 3 years.

    Mike

    • steven mosher
      Posted Jul 29, 2009 at 10:27 PM | Permalink

      Re: thefordprefect (#110),

      Do you think that Jones informed the French of his obligations under FOIA when he entered into this contract?
      Do you think he contacted the French, the interested third party, when the FOIA request involved their data?

      • Posted Jul 29, 2009 at 11:50 PM | Permalink

        Re: steven mosher (#115),

        Do you think that Jones informed the French of his obligations under FOIA when he entered into this contract?
        Do you think he contacted the French, the interested third party, when the FOIA request involved their data?

        The agreements (if they actually exist) would have predated the introduction of the Freedom of Information Act. I wonder if there is any guidance on what to do about pre-existing confidentiality agreements. It would seem reasonable for CRU to inform the French or the Syrians or whoever that the data was now subject to FoI.

    • Steve McIntyre
      Posted Jul 29, 2009 at 10:50 PM | Permalink

      Re: thefordprefect (#110),

      The “25 years invested” comment was no joke. Von Storch was shocked at this statement, disbelieving that any scientist could say such a thing. He asked Jones about it and Jones stood by the statement – as Ross reported in a comment recently.

    • Mark T
      Posted Jul 29, 2009 at 11:03 PM | Permalink

      Re: thefordprefect (#110),

      But have you considered how moch a few hundred FOI requests are going to cost a small university (who are probably struggling financially). Its probably equivalent to tutoring a couple of undergraduates for 3 years.

      Then maybe they should step up and teach those undergraduates how science is really done, release the data, admit they were wrong, and get over this nonsense?

      Mark

    • jeez
      Posted Jul 29, 2009 at 11:07 PM | Permalink

      Re: thefordprefect (#110),

      I’ve seen people finish scathing and negative emails with “cheers” many time, in fact, I’ve seen it used as an automated signature and I had to point out to a partner once how weird it was to end a negative rant with:

      “Cheers Gordon”

      He changed his email signature after that.

      Also, there is no context possible in which to read Phil Jones’s comment that is not antiscience.

      Even if WMO agrees, I will still not
      pass on the data.

      Puleeze.

      His reference to IPR can be loosely translated in context as:

      IT’S MINE! MINE! MINE! MINE!……MINE!

    • Posted Jul 30, 2009 at 2:16 AM | Permalink

      Re: thefordprefect (#110),

      I know you consider this to be having a little bit of fun at Jones’ cost. But have you considered how moch a few hundred FOI requests are going to cost a small university (who are probably struggling financially).

      This is a gem, even by your standards. In 2008 poor impoverished Phil Jones only had about 8 research grants, worth over a million pounds of UK and EU taxpayers money.

    • David Holland
      Posted Jul 30, 2009 at 3:21 AM | Permalink

      Re: thefordprefect (#110),

      If CRU, the Met Office, Defra and all other public authorities were bothered about the cost of the Environmental Information Regulations 2004 they would have started to put investment into conforming with regulation 4 even before it became a legal requirement. The Convention from which EIR descended was signed in 1998, so they had plenty of time.

      In the ICO Guidance Note issued Jan 2005 it stated:

      Regulation 4 (4) (a) refers to information from Article 7 (2) of the Directive. In effect
      this means that as a minimum each authority subject to the Regulations will, (where it holds such information), be obliged to include the following:
      (a) texts of international treaties, conventions or agreements, and of Community, national, regional or local legislation, on the environment or relating to it;
      (b) policies, plans and programmes relating to the environment;
      (c) progress reports on the implementation of the items referred to in (a) and (b) when prepared or held in electronic form by public authorities;
      (d) the reports on the state of the environment referred to in paragraph 3;
      (e) data or summaries of data derived from the monitoring of activities affecting, or likely to affect, the environment;
      (f) authorisations with a significant impact on the environment and environmental agreements or a reference to the place where such information can be requested or found in the framework of Article 3;
      (g) environmental impact studies and risk assessments concerning the environmental elements referred to in Article 2(1)(a) or a reference to the place where the information can be requested or found in the framework of Article 3.
      (h) facts and analyses of facts which the public authority considers relevant and important in framing major environmental policy proposals.

      Looking at the above, almost everything they have should be proactively diseminated. The least cost approach would be to unlock all the files and start building a decent index.

      FWIW My beef with the ICO is

      (where it holds such information)

      Under the Aarhus Convention the European Governments all agreed to ensure that

      Public authorities possess and update environmental information
      which is relevant to their functions

      That in my view means that the can not refuse to hold and disseminate information that is easily available to them and relevant to their functions – like Review Editors’ reports and Expert Reviewers’ comments.

    • TerryS
      Posted Jul 30, 2009 at 3:34 AM | Permalink

      Re: thefordprefect (#110),

      By IPR I assume you mean Intellectual Property Rights.

      Since you can not copyright or patent numbers I assume that the IPR Phil Jones is talking about is either Database Rights or “Sweat of the brow”.

      The UK defines a database (for the purposes of Database Rights) as:

      ” Databases
      3A. – (1) In this Part “database” means a collection of independent works, data or other materials which –

      (a) are arranged in a systematic or methodical way, and

      (b) are individually accessible by electronic or other means.

      (2) For the purposes of this Part a literary work consisting of a database is original if, and only if, by reason of the selection or arrangement of the contents of the database the database constitutes the author’s own intellectual creation.”.

      Since these are plain text files they might have difficulty exercising Database Rights on them due to (b). “Sweat of the brow” however is another matter. In either case CRU/Phil Jones need to actively protect them in order to maintain any rights they hold. If they are aware of somebody having the data and that person’s intention to redistribute, then by not making any attempt to prevent the distribution they are giving their implied consent.

      Just like climate, the Law is open to interpretation so if I was Steve I would get legal advice before distributing any data.

    • Kenneth Fritsch
      Posted Jul 30, 2009 at 11:17 AM | Permalink

      Re: thefordprefect (#110),

      I know you consider this to be having a little bit of fun at Jones’ cost. But have you considered how moch a few hundred FOI requests are going to cost a small university (who are probably struggling financially). Its probably equivalent to tutoring a couple of undergraduates for 3 years.

      Fordperfect, you need to pass this one onto the CRU people. “Due to the cost of maintaining our meager budgeting for tutoring graduate students and a fund for lost kittens, we are forthwith suspending replies for FOI requests”.

      Ford you are a perfect candidate for the bureaucracy, if you have not already qualified.

      • steven mosher
        Posted Jul 30, 2009 at 11:35 AM | Permalink

        Re: Kenneth Fritsch (#141),

        Ford would be well advised to stop digging. It’s funny the lengths people on my side ( believers in AGW) will go to defend the shoddy work that people sometimes put out and the difficulties they impose WRT replication. It is exactly the kind of behavior that prompted me to make my infamous piltdown mann comment. There too, solid science, was almost taken down by people too eager to shut the other side up. While Jones’ behavior is not fraud, my point remains. When the proponents of AGW ( like me) let our side get away with shoddy behavior the focus shifts to that behavior and the personalities involved. Free the data, free the code, free the debate.

        • Ryan O
          Posted Jul 30, 2009 at 11:50 AM | Permalink

          Re: steven mosher (#142),

          Free the data, free the code, free the debate.

          .
          And thusly was the Lukewarmer Credo coined.
          .
          Lukewarmers, unite!

        • steven mosher
          Posted Jul 30, 2009 at 12:35 PM | Permalink

          Re: Ryan O (#143),

          Yup. that’s our credo. somebody tell Lucia and the other lukewarmer bloggers.

        • John F. Pittman
          Posted Jul 30, 2009 at 1:13 PM | Permalink

          Re: steven mosher (#142), “”Free the data, free the code, free the debate.
          That’s our credo. somebody tell Lucia and the other lukewarmer bloggers.””

          I believe we are here already.

        • thefordprefect
          Posted Jul 30, 2009 at 1:15 PM | Permalink

          Re: steven mosher (#142), You have me wrong

          The data SHOULD BE FREE All I have said is CRU may be hamstrung by agreements. For McIntyre to make try to entrap jones is childish. I would not defend a anti=AGW person in similar circumstance – I would leave it up to others (unless it became too serious)

          I am truly concerned at all the FOI requests you are firing off to a Small university whose funds are already under pressure. How much does it cost to service an FOI?

          Read the paper trail of this situation. To me it looks like initial poor communications leading to outright hostility.

          Perhaps McIntyre should have a beer and a quiet chat with Jones.

        • stephen richards
          Posted Jul 30, 2009 at 1:21 PM | Permalink

          Re: thefordprefect (#146),

          The FOI would cost de nada, rien, aucun, nothing if they all followed their own policies of achiving and making available data and code.

        • JimR
          Posted Jul 30, 2009 at 1:28 PM | Permalink

          Re: thefordprefect (#146), Steve McIntyre seems to have made every effort to obtain this data before resorting to the use of FOI. If this whole incident has put a strain on the funds and time of Jones and the university the blame rests solidly on Phil Jones shoulders. The initial requests to Jones as well as the MET were reasonable requests and the responses were less than cooperative. That is why FOI were created and it could have/should have been avoided with a little academic cooperation.

        • Willis Eschenbach
          Posted Jul 30, 2009 at 1:47 PM | Permalink

          Re: thefordprefect (#146), you ask us to “Read the paper trail of this situation” … dude, we wrote half of the paper trail and we have most definitely read the other half, their responses to what we wrote. You’re coming late to this party, and now you want us to re-read what happened before you got here? Get real. This is not the result of a misunderstanding. It is the result of Phil Jones not wanting to have anyone examine or attempt to replicate his work. As Jones stated, his concern is that we might have the unspeakable effrontery to actually try to find errors in his work … and you are defending that? Really?

          The whole thing could have been avoided if Phil Jones had either:

          1. Simply released all of the data, or

          2. Released all of the data that is not covered by any non-disclosure agreements, and released the agreements themselves to show why the rest could not legally be released.

          Instead, there has been a continual obfuscation and coverup that has led to the current contretemps. You complain about the “poor university”, but the data is paid for by taxpayers, not the university. The cost of complying with the FOI would have been trivial. The cost of childishly fighting it could be large … and that’s our fault how? Say what? I just asked for the taxpayer-paid information.

          Overall, though, you seem to be studiously ignoring the main issue. This is that science depends on transparency. If it can’t be replicated, it’s not science. Your defense of Jones hiding the data is anti-science. You are free to be as anti-scientific as you wish … but trying to persuade us that hiding data is a reasonable scientific point of view is a sad reflection on the current state of climate “science”. As we used to say on the cattle ranch where I grew up, “You can piss on my boots, but you can’t convince me it’s raining”.

          w.

        • reid simpson
          Posted Jul 30, 2009 at 5:31 PM | Permalink

          Re: thefordprefect (#146),

          Problem is, what kind of beer would they drink? All kinds of innuendos involved in the choice. And should it be served warm or cold? etc.

  62. Steve McIntyre
    Posted Jul 29, 2009 at 9:24 PM | Permalink

    In a letter to Doug Keenan in correspondence here on May 31, 2007, Jones relied as follows on confidentiality agreements:

    May 31, 2007 Jones Reply to Keenan
    Dear Doug,
    I have done some searching in files – all from the period 1990-1998.

    This is the time when we were in contact with a number of NMSs. We have also got datasets from fellow scientists and other institutes around the world. All supplied data (eventually and sometimes at cost), but we were asked not to pass on the raw data to third parties, but we could use the data to develop products (our datasets) and use the data in scientific papers.

    It is likely that some of the NMSs and Institutes have changed their policies now – and that the people we were corresponding with (all by regular mail or fax) are no longer there or are in different sections.

    The lists below don’t refer to all the stations within these countries, nor to all periods, but to some of the data for some of the time.

    The NMSs
    Germany, Bahrain, Oman, Algeria, Japan, Slovakia and Syria

    Scientists/Institutes (data for these countries)
    Mali, India, Pakistan, Poland, Indonesia, Democratic Republic of the Congo (was Zaire), Sudan and some Caribbean Islands

    These are the only ones I can find evidence for. I’m sure there were a few others during the 1980s, but we have moved buildings twice since 1980. Not sure how you will use this data.
    Phil Jones

    I’ve made a fairly complete concordance of CRU and GHCN identifications – an obstructive omission in CRU’s information – and examined the concordance for every station in the countries for which Jones said that there was “evidence” of a confidentiality agreement.

    In every case except a few stations in Syria, every station has its data filed at GHCN. Thus any confidentiality that may have applied 25 years ago is long since waived by the public filings.

    The only confidentiality: Dr Phil and the Baath Party.

    • thefordprefect
      Posted Jul 29, 2009 at 10:02 PM | Permalink

      Re: Steve McIntyre (#109), So Jones was right when he said the data was available elswhere!
      Have you checked France, as this appeared in one of his replies (see my last response)?

      • Steve McIntyre
        Posted Jul 29, 2009 at 10:48 PM | Permalink

        Re: thefordprefect (#111),

        So Jones was right when he said the data was available elswhere!

        The CRU version of the data is only available at CRU. But remember that Jones refused to identify the stations used by CRU in multiple FOI requests, grudgingly identifying the stations only after multiple requests and appeals.

        Have you checked France, as this appeared in one of his replies (see my last response)?

        The French data is archived at GHCN.

        Maybe some of Jones’ excuses applied 25 years ago, but they had expired long before he made his excuses.

        Thus far, we have seen no evidence of any confidentiality agreement between Dr Phil and anyone other than perhaps the Syrian Baath Party.

        • Posted Jul 29, 2009 at 11:07 PM | Permalink

          Re: Steve McIntyre (#117),

          We don’t need Syrian data, the Syrian Baath party keeps reporting the same temp anyway OMA it’s hot. 🙂

          Not many people would take the time to write a confidentiality about thermometer data. Who has that much time anyway?

    • steven mosher
      Posted Jul 29, 2009 at 10:16 PM | Permalink

      Re: Steve McIntyre (#109),

      Steve See annex G of the UEA proceedures WRT entering into these types of arrangements. I’m preparing a FOIA WRT to this.

    • Ron Cram
      Posted Jul 30, 2009 at 12:47 AM | Permalink

      Re: Steve McIntyre (#109),

      Very interesting comment, Steve. It is nice to see Jones narrow it down a bit. You’ve been doing great work.

  63. steven mosher
    Posted Jul 29, 2009 at 10:10 PM | Permalink

    2. In some cases, a disclosure of information cannot be made without the consent
    of a third party (for example, where information has been obtained from a third
    party and in the circumstances the disclosure of the information without their
    consent would constitute an actionable breach of confidence such that the
    exemption at Section 41 of the Act would apply). In such instances, members of
    staff must consult that third party with a view to seeking their consent to the
    disclosure unless such a consultation is not practicable (for example because the
    third party cannot be located or because the costs of consulting them would be
    disproportionate.

    I wonder if CRU can find france. I wonder if jones, per his obligations, contacted France?

    ANNEX G and ANNEX D governing the obligations of staff when entering into contracts
    with confidentiality clauses is full of fun stuff

  64. theduke
    Posted Jul 29, 2009 at 10:25 PM | Permalink

    I find it truly annoying that the arguments over this data, i.e. who can access it and how it us used, are now primarily legal and not scientific.

    It says a lot about Jones’ integrity as a scientist.

  65. theduke
    Posted Jul 29, 2009 at 10:36 PM | Permalink

    The only confidentiality: Dr Phil and the Baath Party.

    Like Assad, Saddam Hussein was also a Ba’athist. Dr. Phil may have had ample reason here to not violate confidentiality agreements.

  66. theduke
    Posted Jul 29, 2009 at 11:30 PM | Permalink

    When the history of this scientific debate is written fifty years from now, then again 100 years from now, and ad infinitum, it will be apparent who the real guardians of science were.

    If I were Dr. Phil, I’d keep that in mind.

  67. Nylo
    Posted Jul 30, 2009 at 12:14 AM | Permalink

    I don’t think the “people” folder is supposed to be private. Some days have passed and they are still public. If they were made public by mistake, sure someone would have corrected the mistake already.

    This all looks like a distraction to me. The outcome seems evident. If the goal is to get the most up-to-date data with a FOI request, go ahead, but I would also start working with the old data already, it is certain that they won’t be able to claim confidentiality about it. (Well, they already claimed it, but I mean they would not be able to defend that in front of a court).

  68. VG
    Posted Jul 30, 2009 at 4:02 AM | Permalink

    If I was SM I would go ahead and analyze the data. If you find some major FXXud/trickeries he/or this site may become heroic/hero. If not, no one will really care due to UHI and other reasons, surface temps are not credible. After all this is “climate audit” = “to audit climatic data” and it is supposedly the purpose of this site? I still think concerning global temps the only valid way to go is satellite data.

  69. Ruth
    Posted Jul 30, 2009 at 6:30 AM | Permalink

    Apologies if this has already been posted on this or another thread, but today’s Nature has an editorial which is relevant (and encouraging, I think). It’s about a report released last week by the US National Academies on the topic of data accessibility.

    To quote one paragraph:

    “The access principle asserts the value of openness: only if results are shared can other researchers check the data’s accuracy, verify analyses and build on previous work. So unless there are very good reasons for researchers to withhold data — reasons that should be publicly posted and available for comment by other researchers — they should make provisions to supply public access in a timely manner, possibly as early as their grant proposals.”

    • Posted Jul 30, 2009 at 6:43 AM | Permalink

      Re: Ruth (#131),

      Ruth

      Very interesting! This is the report of the special NAS panel set up to address M&M’s issues with Mann’s data availability. In reality it was an attempt to kick the issue into the long grass.

  70. Ruth
    Posted Jul 30, 2009 at 7:11 AM | Permalink

    Re: Bishop Hill (#133)

    I hadn’t realised that! The NAS report’s summary can be downloaded here. It says in the summary that the whole report can be browsed freely online, but I can’t find a way to do that.

  71. Posted Jul 30, 2009 at 7:55 AM | Permalink

    Yep. Your tax money at work

    Are you as impressed by the scientific standards as I am?

  72. Keith
    Posted Jul 30, 2009 at 8:39 AM | Permalink

    Hmm, one of the people folders on the CRU FTP site is Dave Lister. He has a folder of figures for China from 2008 that is named forphilj, so he contributes to Jones’ work. But seeing that file name made me ask myself, “Are they Red Dwarf fans possibly?”

  73. Posted Jul 30, 2009 at 9:12 AM | Permalink

    RE RomanM #76, Raven 102, John A 136,
    For all we know, the “youstupidb*****” files on CRU’s people/harry/censored/ directory are viruses, so it’s probably best not to try to open them. (I’ve tried to airport-proof the filename 🙂 )

    If they are viruses, they would clearly fall under the UK anti-virus act quoted by Steve at http://www.climateaudit.org/?p=6644#comment-350926. Is there a way to download a file and scan it for malicious content without opening it?

    This creative “harry” would appear to be dendro-specialist Ian (Harry) Harris.

  74. mjt1st
    Posted Jul 30, 2009 at 10:11 AM | Permalink

    From http://www.cru.uea.ac.uk/cru/data/

    The various datasets on the CRU website are provided for all to use, provided the sources are acknowledged. Acknowledgement should preferably be by citing one or more of the papers referenced on the appropriate page. The website can also be acknowledged if deemed necessary. CRU will endeavour to update the majority of the data pages at timely intervals although this cannot be guaranteed by specific dates.

    I’m not a linguist or a lawyer but I am pretty sure SM would be covered under the “all to use” portion of this statement. I personally don’t see the need to give them a timeframe in which they can revoke the permission. Now if only they would give the new location of the data so that you can properly cite it in your work 🙂

  75. Wansbeck
    Posted Jul 30, 2009 at 10:43 AM | Permalink

    The Royal College of Surgeons has just published compelling evidence of the benefits of freely available data:
    “The report, entitled Demonstrating Quality: The Sixth National Adult Cardiac Surgical Database Report, conclusively proves wrong critics who suggested that publishing mortality data would lead to risk-averse behaviour from surgeons, with the most sick and elderly patients being turned down for surgery for fear of blotting statistics. In practice, the opposite has turned out to be true – the increase in reliable data has emboldened surgeons to take on more complicated cases. More people who would have been considered too sick to undergo an operation just five years ago are now routinely treated and doing well.”

  76. Calvin Ball
    Posted Jul 30, 2009 at 2:28 PM | Permalink

    Dr Phil and the Baath Party

    Psychoanalysis in the Jacuzzi?

  77. thefordprefect
    Posted Jul 30, 2009 at 2:33 PM | Permalink

    Steve McIntyre (#72),

    However, it was hinted at to me a year or two ago that I should also not make the station data available….I would suggest you take this up with WMO and/or GCOS. I have raised it several times with them and got nowhere

    I should warn you that some data we have we are not supposed top pass on to others.
    We can pass on the gridded data – which we do. Even if WMO agrees, I will still not
    pass on the data. We have 25 or so years invested in the work. Why should I make
    the data available to you, when your aim is to try and find something wrong with it.
    There is IPR to consider.

    Our estimate is that more than 98% of the CRU data are on these sites.
    The remaining 2% of data that is not in the websites consists of data CRU has collected
    from National Met Services (NMSs) in many countries of the world. In gaining access to
    these NMS data, we have signed agreements with many NMSs not to pass on the raw
    station data, but the NMSs concerned are happy for us to use the data in our gridding, and
    these station data are included in our gridded products, which are available from the CRU
    web site. These NMS-supplied data may only form a very small percentage of the
    database, but we have to respect their wishes and therefore this information would be
    exempt from disclosure under FOIA pursuant to s.41. The World Meteorological
    Organization has a list of all NMSs

    We cannot produce a simple list with this format and with the information you described
    in your note of 14 April. Firstly, we do not have a list consisting solely of the sites we
    currently use. Our list is larger, as it includes data not used due to incomplete reference
    periods, for example. Additionally, even if we were able to create such a list we would
    not be able to link the sites with sources of data. The station database has evolved over
    time and the Climate Research Unit was not able to keep multiple versions of it as
    stations were added, amended and deleted. This was a consequence of a lack of data
    storage in the 1980s and early 1990s compared to what we have at our disposal currently.

    and so on. The killer quote you keep giving (part of second para – just enough to ensure the context is lost) does not seem to my British mind to be antagonistic, just stating the truth with a bit of black humour. Your interpretation does not ring true. Did you contact him about this comment?

    From http://eca.knmi.nl/documents/ECAD_report_2008.pdf I keep trying to point out this organisation has the same problem they have restricted data, why should this not be the case with CRU?

    • Ryan O
      Posted Jul 30, 2009 at 3:43 PM | Permalink

      Re: thefordprefect (#151),

      From http://eca.knmi.nl/documents/ECAD_report_2008.pdf I keep trying to point out this organisation has the same problem they have restricted data, why should this not be the case with CRU?

      .
      You do have a wonderful way with the burden of proof. You simply assume that it is the case, and then excoriate us when we utilize the FOI system to force CRU to produce evidence that this is the case.
      .
      In other words, your argument boils down to “CRU might be legally restricted from providing the data and it is totally out of bounds to ask them to provide evidence that this is, indeed, the case.”
      .
      I would call this climate science’s version of “don’t ask, don’t tell”.

    • Gerald Machnee
      Posted Jul 30, 2009 at 10:29 PM | Permalink

      Re: thefordprefect (#151),

      The killer quote you keep giving (part of second para – just enough to ensure the context is lost) does not seem to my British mind to be antagonistic, just stating the truth with a bit of black humour. Your interpretation does not ring true. Did you contact him about this comment?

      Why don’t you contact him? Steve and most bloggers here are quite clear what he meant. You seem to be the only one with a problem.

    • Geoff Sherrington
      Posted Jul 31, 2009 at 2:50 AM | Permalink

      Re: thefordprefect (#151),

      There are many people, including me, who have asked P D Jones for raw data and got nothing for asking. Would there be value in making a list of rebuttals? Also add if you have kept the correspondence and that it is available. I’ve posted a lot of mine on CA already. Geoff Sherrington, Scientist.

  78. Charlie
    Posted Jul 30, 2009 at 2:34 PM | Permalink

    QUALITY OF INFORMATION GUIDELINES
    Not that it applies to the Met or CRU, but in looking at the situation in the USA I find that not only is there the well known FOI, but there are also the “Quality of Information” standards. That system governs “requests for correction” and “requests for review”.

    This path seems to be underutilized as NASA has sent in reports for several years saying that they had no requests other than two that dealt with SETI.

    NASA Quality of Info

    NASA Requirements for Documentation, Approval, and Dissemination of NASA Scientific and Technical Information

    The NOAA webpage on Quality of Information has a good explanation.
    NOAA Quality of Info Guidlines

    I may use the NASA request for correction if the JPL Earth Science Communication team can’t manage to correct the sea level graphs at Key Indicators (The error was originally pointed out by another commenter)

    The data on the graphs doesn’t match up with the U of Colorado graphs shown as the source, and the trendline captions don’t match up with the data on the graph. The only change they’ve made in the 10 days since I’ve contacted both the site manager and the webmaster is to change a caption so that rather than showing the last update as 7.15.09, now it reads “Last updated 07.51.09” (Yes, as in July 51st)

  79. Met Mole
    Posted Jul 30, 2009 at 2:34 PM | Permalink

    Willis,

    It’s not your boots he’s pissing on but up your back, and he’s telling you it’s raining.
    It’s important to get these technical details right.

    He’ll probably tell you next it’s the climate not just the weather.

  80. Willis Eschenbach
    Posted Jul 30, 2009 at 2:59 PM | Permalink

    thefordprefect, Jones said:

    Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it.

    You claim this was “stating the truth with a bit of black humour”. Kinda odd that Phil never made that claim …
    .
    Phil Jones has had ample opportunity to say “Hey, I was just kidding”, or “I didn’t really mean it, it was just black humour” or to defend himself in some other way. He has not done so.
    .
    Kinda curious, no? Almost as odd as you jumping in to defend a man who has not tried to defend himself … He was contacted later when this was all being investigated. It was a perfect time to explain how he’d been cruelly misunderstood as you claim. He did not say it was black humour. He did not say he was kidding. He simply confirmed that he had said it
    .
    But let’s suppose you are right. I know it’s a stretch and I’m asking something hard, but sometimes I’ve believed as many as six impossible things before breakfast, so I suppose I can believe that you’re right about Phil. Your claim is that with a soupçon of black humor he was merely “stating the truth” when he said “Why should I make the data available to you, when your aim is to try and find something wrong with it.”
    .
    Please tell me you understand why no actual scientist would ever advance that in defense of his hiding data from the merciless gaze of other scientists …
    .
    With apologies to Charles Dodgson,
    .
    w.

  81. Adam Soereg
    Posted Jul 30, 2009 at 4:04 PM | Permalink

    I’ve successfully downloaded a large part of the CRU station data from the ftp server mentioned above. Altough it is not the most recent version, I possess all their series located northern than 40°N.

    The gold mine is here, you can find some hidden data files even after their immediate deletion campaign. ftp://www.cru.uea.ac.uk/

    Just to prove my statement, the following Austrian records are included (numerous non-GHCN sites with so-called “sensitive” data):

    110100 Linz 1852-1956
    110200 Stift Zwettl 1883-
    110207 Hohenfurth 1817-1843
    110280 St. Poelten 1893-
    110350 Vienna – Hohe Warte 1775-
    111010 Bregenz 1870-
    111197 Wilten 1829-1858
    111450 Bad Gastein 1854-
    111460 Sonnblick 1887-
    111500 Salzburg AP 1842-
    111550 Feuerkogel 1930-
    111560 Bad Ischl 1855-
    112120 Villacher Alpe 1851-
    112310 Klagenfurt 1813-
    112450 Bad Gleichenberg 1881-
    112900 Graz-University 1837-
    112991 Seckau 1891-
    113200 Innsbruck 1795-

  82. Urederra
    Posted Jul 30, 2009 at 4:36 PM | Permalink

    @theFordPrefect,

    If I am not wrong, this “Phil Jones” is the same guy who wrote the “Jones et al (1990) paper about UHI. The data of some chinese stations used in their analysis seem to be missing too.

    You can follow the Jones et al (1990) story here.

    • tetris
      Posted Jul 30, 2009 at 10:02 PM | Permalink

      Re: Urederra (#157),
      Yes, it is the same Jones. On the basis on that paper in particular, the IPCC established the view that UHI [like other land-use factors] had no bearing on changes in weather or climate conditions.
      Our host has made it clear that he does not like my speculating about things on his site. His privilege and I will try to respect that. That said, given his background in [mining] due diligence, he, just like me [with some 25 years of due diligence experience in both sovereign and corporate finance and the technology sector] knows that as a good, basic, ground rule, it is prudent to assume that there never is smoke without some sort of fire. All else aside, we can possibly all agree that it is beyond much discussion that there has been smoke of some sort or the other around Dr Jones for many years.

      • crosspatch
        Posted Jul 31, 2009 at 12:41 AM | Permalink

        Re: tetris (#160),

        “there never is smoke without some sort of fire”

        Or mirrors.

        And, I might add, thefordperfect has been most entertaining. I find it interesting that scientific data is allowed to be restricted from those who might “find something wrong” with someone’s analysis yet is perfectly acceptable to be given to those who would validate the conclusions. tfp would have more ground on which to stand if the restriction on the data dissemination applied to all parties equally. When the keeper of the data complains about giving the data to someone who might be at risk of invalidating his conclusion, every red flag in existence should be raised. That is the entire purpose of archiving data; so that others might perform future analysis and find a flaw in the original researcher’s conclusions. To allow distribution to only those who subscribe to the same hypotheses as the original researcher and to explicitly deny it to parties who wish to take a more objective view is not a practice that gains one prestige in the field of science.

  83. Charlie
    Posted Jul 30, 2009 at 5:47 PM | Permalink

    My comment #152 seems to be stuck in the spam trap (probably because of too many links) so here’s another go, but without links.

    Although not applicable to the Met and CRU, a useful tool with US government organizations could be the Quality of Information Guidelines.

    Google “quality of information NOAA” and take the first hit to have a pretty good review of the system that include provision for “requests for correction” and also requests for data, sources, and methods.

  84. Antonio San
    Posted Jul 30, 2009 at 11:48 PM | Permalink

    OT: Steve for your own gems… please snip when read:
    Environment Canada should get their story right for a change:

    In the Vancouver — The Canadian Press Last updated on Thursday, Jul. 30, 2009 12:49PM EDT ” “A very strong ridge of high pressure is currently dominating all of B.C.,” said Gary Dickinson, a meteorologist with Environment Canada. “The ridge of high pressure also brought up from the south very warm air, which was responsible for the record-breaking temperatures.”

    And now in the Jane Armstrong Vancouver — From Friday’s Globe and Mail Last updated on Friday, Jul. 31, 2009 01:04AM EDT “As forecasters we care about two things: Is the wind blowing onshore or offshore? And if you answer that question, you can tell a lot about the weather. If the wind blows offshore for a prolonged period of time, a heat wave sets in.”

    So dear Environment Canada… did the wind that caused the 2009 heatwave blow from the South -ocean- or from the East -land-? ROTFLOL!!!!!!!

    ““As forecasters we care about two things: Is the wind blowing onshore or offshore?” says Mr. Jones, meteorologist for Environment Canada.

    And at least we know where the Hot Air is coming from…

  85. b
    Posted Jul 31, 2009 at 1:03 AM | Permalink

    Steve is showing remarkable restraint as the fordprefect wanders
    ever more deeply into a Twin Peaks land of twisted logic. But, the
    value of open discussions is very high indeed, and I guess that’s why
    Steve allows the fordprefects mysterious concoctions.

  86. steven mosher
    Posted Jul 31, 2009 at 1:40 AM | Permalink

    ford, once again your lack of attention to detail is astounding.

    and so on. The killer quote you keep giving (part of second para – just enough to ensure the context is lost) does not seem to my British mind to be antagonistic, just stating the truth with a bit of black humour. Your interpretation does not ring true. Did you contact him about this comment?

    Jones could not possibly be stating the truth. He wrote “Why should I make the data available to you, when your aim is to try and find something wrong with it.”

    In the english language we call such a locution a “question” You can go read some speech act theory ( crap who says a liberal education is a waste of time?) here is a good start: http://online.sfsu.edu/~kbach/spchacts.html.

    There are two classes of questions we can readily identify. Rhetorical and earnest. Jones thinks his question is rhetorical. He asks his question and does not expect or want an answer. That’s because he believes the question entails it’s own answer. So, he is not “stating a truth” he is asking what he believes is a rhetorical question. And that is the issue we all have with it. With regards to calling it black humor, you are wrong here as well. Black humor makes light of serious or taboo topics.

    Put your shovel down.

  87. allen mcmahon
    Posted Jul 31, 2009 at 5:00 AM | Permalink

    CRU Data should never by released in the public domain it will immediately be corrupted by the SMc virus. I would go one step further in the interests of national security no access to the raw data by anybody for 30 years. It should be placed in a biscuit tin given to MI6 and buried under a cobblestone in Copenhagen

  88. A Person
    Posted Jul 31, 2009 at 2:33 PM | Permalink

    For the record, here is the index of the FTP server as recorded by Yahoo servers, ie. public access to search robots.

    Index of /ftpdata Name Last modified Size Description

    Parent Directory –
    wwr8190.zip 10-Feb-2002 11:16 2.7M
    tpi_stan.dat 28-Jan-2007 17:00 8.6K
    tpi_hob.dat 28-Jan-2007 17:00 8.5K
    tpi.dat 28-Jan-2007 17:00 11K
    tgrid19942000.dat.gz 05-May-2000 14:01 249K
    tgrid19941998.dat.gz 27-Jan-1999 13:54 201K
    tgrid18561993.dat.gz 03-Feb-1998 12:28 3.9M
    tavesh2v.dat 18-Jan-2006 14:31 27K
    tavesh2.dat 31-Jan-2003 14:52 28K
    tavesh.dat 02-Feb-2003 23:04 27K
    tavenh2v.dat 18-Jan-2006 14:31 27K
    tavenh2.dat 31-Jan-2003 14:52 28K
    tavenh.dat 02-Feb-2003 23:04 27K
    tavegl2v.dat 18-Jan-2006 14:31 27K
    tavegl2.dat 31-Jan-2003 14:52 28K
    tavegl.dat 02-Feb-2003 23:04 27K
    soi_tah.dat 26-Feb-2009 11:52 12K
    soi_dar.dat 26-Feb-2009 11:52 12K
    soi.dat 26-Feb-2009 11:52 14K
    readme.txt 14-Jun-2000 15:56 132
    pjones_gridded_temp_anoms_1998_2002.txt 24-Jan-2003 10:01 1.5M
    old/ 31-Jul-2003 09:55 –
    newcrustnsall.hdr 24-Feb-2003 10:49 288K
    newcrustnsall.dat.Z 24-Feb-2003 11:01 4.5M
    newcruextusall.dat.Z 11-Nov-2002 13:19 2.0M
    ncep1/ 05-May-2000 11:38 –
    ncep/ 20-Jun-2003 20:07 –
    naoseajurg.dat 22-May-2002 15:46 9.9K
    naomonjurg.dat 22-May-2002 15:45 59K
    nao_ice.dat 31-Jul-2003 09:53 14K
    nao_gib.dat 31-Jul-2003 09:53 14K
    nao_azo.dat 11-Jul-2003 11:14 11K
    nao.dat 12-Dec-2000 09:56 17K
    nagasakitemp.dat 10-Sep-2002 16:33 19K
    nagasakipres.dat 10-Sep-2002 16:33 19K
    mslpnh.dat.gz 31-May-2001 12:01 1.5M
    madrasmslp.dat-2003-04-15 30-May-2001 15:39 16K
    madrasmslp.dat 15-Apr-2003 09:31 16K
    lwtlamb.dat 30-Jul-2002 11:46 157K
    lwtjenk.dat 01-Oct-2007 16:52 141K
    ltgrid19911998.dat.gz 01-Feb-1999 14:26 193K
    ltgrid19511990.dat.gz 27-Jan-1999 13:56 1.1M
    ltgrid19011950.dat.gz 27-Jan-1999 13:56 923K
    ltgrid18511900.dat.gz 27-Jan-1999 13:55 411K
    keren_smith/ 23-Sep-2004 13:42 –
    hadsst2.zip 05-Jul-2009 22:19 3.8M
    hadsst1.nc.gz 21-May-2003 09:55 2.2M
    hadsst1.nc 21-May-2003 09:57 7.7M
    hadsst1.dat.gz 21-May-2003 09:55 2.8M
    hadcrutv.nc.gz 21-May-2003 10:00 3.0M
    hadcrutv.nc 21-May-2003 10:02 7.7M
    hadcrutv.dat.gz 21-May-2003 10:00 3.8M
    hadcrut3v.zip 05-Jul-2009 22:19 7.6M
    hadcrut3.zip 05-Jul-2009 22:18 7.3M
    hadcrut2v.zip 18-Jan-2006 14:31 4.1M
    hadcrut2v.nc.gz 18-Jan-2006 14:30 3.2M
    hadcrut2v.nc 18-Jan-2006 14:31 8.1M
    hadcrut2v.dat.gz 18-Jan-2006 14:30 4.1M
    hadcrut2.zip 18-Jan-2006 14:30 4.3M
    hadcrut2.nc.gz 18-Jan-2006 14:30 3.4M
    hadcrut2.nc 18-Jan-2006 14:30 8.9M
    hadcrut2.dat.gz 18-Jan-2006 14:29 4.3M
    hadcrut.nc.gz 21-May-2003 09:58 3.2M
    hadcrut.nc 21-May-2003 10:00 8.7M
    hadcrut.dat.gz 21-May-2003 09:58 4.1M
    gridsmot5101.dat 17-Sep-2002 09:58 23M
    gat.csv 20-Apr-2004 10:31 2.6K
    crutem3v.zip 05-Jul-2009 22:18 3.8M
    crutem3.zip 05-Jul-2009 22:18 2.7M
    crutem2v.zip 18-Jan-2006 14:29 2.4M
    crutem2v.nc.gz 18-Jan-2006 14:29 1.8M
    crutem2v.nc 18-Jan-2006 14:29 9.2M
    crutem2v.dat.gz 18-Jan-2006 14:29 2.4M
    crutem2v.dat.Z 02-Jan-2003 15:34 2.0M
    crutem2.zip 18-Jan-2006 14:28 3.1M
    crutem2.nc.gz 18-Jan-2006 14:28 1.9M
    crutem2.nc 18-Jan-2006 14:29 18M
    crutem2.dat.gz 18-Jan-2006 14:28 3.1M
    crutem2.dat.Z 02-Jan-2003 15:34 3.2M
    crutem1v.nc.gz 21-May-2003 10:04 1.6M
    crutem1v.nc 21-May-2003 10:06 8.9M
    crutem1v.dat.gz 21-May-2003 10:04 2.2M
    crutem1.nc.gz 21-May-2003 09:52 1.7M
    crutem1.nc 21-May-2003 09:55 18M
    crutem1.dat.gz 21-May-2003 09:52 2.8M
    bradleyfinalversion.pdf 17-Apr-2003 09:50 9.4M
    andy/ 22-Jul-2003 15:04 –
    anders9103a.dat 24-Feb-2003 10:46 2.3M
    alp_pre_dat.txt.gz 07-Oct-2005 11:15 11M
    alp_pre_dat.nc 02-Feb-2009 15:40 30M
    alp_pre_acc.txt.gz 07-Oct-2005 09:15 1.5M
    alp_pre_acc.nc 02-Feb-2009 13:47 30M
    alp_pre.txt 13-Jan-2006 15:13 8.7K
    alp_pre.pdf 13-Jan-2006 13:45 1.3M
    allsds6190.dat 24-Feb-2003 10:50 438K
    allnorms6190.dat 24-Feb-2003 10:50 443K
    abstem3.zip 23-Jun-2006 12:51 51K
    absolute.zip 20-Aug-2003 11:18 46K
    absolute.nc.gz 21-May-2003 10:06 40K
    absolute.nc 21-May-2003 10:06 62K
    absolute.dat.gz 21-May-2003 10:06 46K
    00readme.txt 14-Jun-2000 15:57 132

    Apache/2.2.3 (Scientific Linux) Server at http://www.cru.uea.ac.uk Port 80

  89. jryan
    Posted Aug 3, 2009 at 7:25 AM | Permalink

    This has been a fascinating turn of events. In the movie version I envision this latest purge to look something like the scene in Lord of the Rings where the camera takes us on a tour of Sauruman’s war factory beneath Isengard.

    Except instead of anvils and birthing pits it’s slamming fingers on “Delete” keys and the continuous sound of paper shredders.

    The music is the same though.

    Also, if you don’t mind speaking in code: OesDay isThay eleteDay ataDay illStay owShay upay onay ethay aybackway achinemay? 🙂

13 Trackbacks

  1. By Concern about missing data « Climate Media on Jul 29, 2009 at 6:37 AM

    […] Source: Climate Audit […]

  2. […] Monday, July 27, 2009, as reported in a prior thread, CRU deleted three files pertaining to station data from their public directory […]

  3. […] Monday, July 27, 2009, as reported in a prior thread, CRU deleted three files pertaining to station data from their public directory http://ftp.cru.uea.ac.uk/. […]

  4. […] CRU Erases Data […]

  5. […] Gosh, who to believe? Somebody that fakes biotelemetry data or somebody that won’t hand over climate data for replication studies? […]

  6. […] Gosh, who to believe? Somebody that fakes biotelemetry data or somebody that won’t hand over climate data for replication studies? […]

  7. […] kids science web page 3 11 2009 Hadley CRU isn’t the only government agency that deletes web content related to climate. NOAA/NWS Southern Region Headquarters got into the act today. An interesting thing happened today. […]

  8. […] CRU isn’t the only government agency that deletes web content related to climate. NOAA/NWS Southern Region Headquarters has gotten into the act. An interesting thing happened […]

  9. […] CRU isn’t the only government agency that deletes web content related to climate. NOAA/NWS Southern Region Headquarters has gotten into the act. An interesting thing happened […]

  10. […] CRU isn’t the only government agency that deletes web content related to climate. NOAA/NWS Southern Region Headquarters has gotten into the act. An interesting thing happened […]

  11. […] how sloppy CRU was at leaving files lying around in the open (Steve McIntyre had fun with the “mole” story prior to Climategate), getting onto the […]

  12. […] Monday, July 27, 2009, as reported in a prior thread, CRU deleted three files pertaining to station data from their public directory […]

  13. […] IPCC and AGU climatologists responded by concealing data, perverting the peer review process, and destroying the careers of critics. Rat […]