Another Brick in the Wall

After years of effort, the chronologies of Briffa et al 2001 were recently made public, although the date on which these became public is itself clouded in mystery. [Update - this minor mystery is clarified: it looks like the data was unlocked on Sep 9, 2008, the day after my FOI request but before my followup request.]

The MXD data from Briffa et al 2001 had been displayed in IPCC 2001 (and again in IPCC 2007); this data had also been used in Rutherford et al 2005 (an “independent” contributor to the IPCC 2007 spaghetti graph and again via the Rutherford et al gridded version in Mann et al 2008. For years, Briffa refused to identify the locations of the sites used in Briffa et al 2001. (Much of the data had been archived at WDCP by Schweingruber, but, without knowing which Schweingruber sites were used, one couldn’t get a foothold.)

Briffa had made information available to the initiate on a password-protected basis:

The focus of the SO&P project is the climate of the last 1000 years, and data that have been collected for use in the project or that have been produced by SO&P are accessible from here. Some data are password-protected because they are not publicly available yet.

My efforts to obtain access to the password-protected site were rebuffed in 2005, which I satirized at CA here observing:

I can somewhat understand the argument for data being private for a limited period (although I would be pretty tough on enforcing the terms of the contract), but I’m having trouble understanding the rationale for password protected sites with access limited to the initiate. I’ve tried unsuccessfully to get access to the European data at SO&P, managed by Briffa and Osborn. We can only hope that Briffa’s concept of a reasonable period of exclusive use will be less than 22 years.

From time to time, I re-visited this always without success. Briffa, after all, is Phil Jones’ closest colleague at CRU, the Phil Jones of “We have 25 years invested in this. Why should we make our data available to you, when your objective is to find something wrong with it?” – a comment, by the way, not made to me but to another person and long before the start of CA.

In the wake of Mann et al 2008, I re-visited the matter, this time using the FOI act. Mann et al referred to gridded MXD data, which proved to derive from Rutherford (Mann) et al 2005. Although Rutherford et al 2005 promised in the Journal of Climate text that their data was available, the URL for the MXD data was not available at the designated website, which said to “contact Tim Osborn” (Briffa’s colleague). I wrote to CRU in September 2008 as follows:

In the Supporting Information to Mann et al (PNAS 2008), in particular http://www.pnas.org/content/suppl/2008/09/02/0805721105.DCSupplemental/SD1.xls , a number of “Schweingruber” series are listed, with nomenclature such as schweingruber_mxdabd_grid11, which I presume were provided by Keith Briffa or Tim Osborn of the UEA.

Pursuant to the Freedom of Information Act and/or Environmental Information REgulations, whichever is aplicable, would you please provide me with a digital version of these data sets in the form provided to Dr Mann, together with any relevant meta-data, manuals or literature describing the grid locations of the series and the method of their calculation.

A few weeks later, as reported at CA, the gridded data versions were posted up at CRU together with meta-data providing the lat-longs of the gridcells (which had been misreported in one of the Mann et al 2008 SI datasets, now altered to the correct values corrected with the original error notice reporting the change now deleted.)

Unfortunately, this still left some puzzles and some gaps in the data. While the majority of the sites could be cross-identified against Schweingruber data archived at WDCP, a number of sites contributed to the gridded data, but had not been archived at WDCP/ITRDB. There were some other puzzles, which I’ll discuss on another occasion. In any event, on October 31, 2008, I sent the following followup inquiry asking for the data that remained unavailable:

5) Not all series listed at the Osborn webpage are in the ITRDB data set. Some examples are:
id name type long lat start end
327 gartogfi Gartog PCBA 98.52 29.67 1709 1993
328 haizefi Haize Shan PCBA 99.50 30.30 1777 1993
329 lhamafi Lhamcoka PCBA 99.12 31.82 1784 1994
330 lhambfi Lhamcoka PCBA 99.13 31.80 1669 1994
331 lhamcfi Lhamcoka PCBA 99.10 31.82 1768 1994
332 lhamdfi Lhamcoka PCBA 99.10 31.82 1630 1994
333 qamdofi Qamdo PCBA 96.95 31.08 1406 1994
334 riwofi1 Riwoqe PCBA 96.48 31.23 1709 1994
335 riwofi2 Riwoqe PCBA 96.48 31.30 1673 1994
Can you please provide this data.

This inquiry, which I had not made public, had previously been the topic of a derogatory email by Phil Jones to the 17 Santer coauthors on Nov 11, saying:

Don’t feel picked on – we in CRU had another FOI request related to tree-ring data yesterday as well. It is in a similar vein. We put up all the individual tree-ring series (widths, densities) – i.e. what we consider the raw data. He already had the chronologies. He now wants to know why some individual series were excluded from the chronologies and why some chronologies were excluded in subsequent analyses. This time they have asked for manuals, computer code and correspondence explaining the exclusions! It seems neverending.

If they just did some paleo fieldwork with trees, corals, sediment cores they might understand why some samples are excluded.

I would urge the 4 NOAA people on the paper to make a joint response to the FOI request when it filters through that the raw data for our paper are all publically available. I know it’s not in their (skeptic) make up, but the sooner they get their hands dirty with the sorts of analyses we/you’ve done for this and many other papers the better. They seem only to want to come in at the interpretational end, particularly on the statistical side.

On other occasions, CRU has used confidentiality of correspondence as a reason to refuse FOI requests, but it’s interesting that in a case of perceived adverse interest, their policies did not seem to require them to preserve the confidentiality of my inquiry.

In any event, a few weeks later, notwithstanding Phil Jones’ complaint to the Santer 17, on Dec 3, 2008, I was informed

These chronologies are in fact already available elsewhere on our website — see: http://www.cru.uea.ac.uk/cru/projects/soap/data/proxy/

In order to lessen the number of multiple archives of the same data set on the internet, it is preferred that the ITRDB be used as the primary source wherever possible. However, as some of the chronologies that were used are apparently not available at the ITRDB, the above webpage holds a copy of the chronology data that were actually used. Important information regarding the standardisation applied in the construction of these chronologies is given at this webpage and should be read and considered carefully when using these data.

The cited webpage proved to be the SO&P webpage where the data had previously been password-protected. Later on Dec 3, I reverted to the CRU FOI officer as follows:

Thank you very much for this. I’m glad the password protection for the SO&P tree ring data has been removed (this data was password protection at one point). I presume that the password protection was done in the past month in response to the present request and I appreciate this. The covering webpage http://www.cru.uea.ac.uk/cru/projects/soap/ still refers to password-protection and you might want to suggest that that be changed. In addition, the webpage http://www.cru.uea.ac.uk/cru/projects/soap/data/proxy/ currently says “Last updated: November 2005, Tim Osborn”. I don’t think that this is correct, since, as far as I know, the page showed that the data was password protected well after that date.

I asked David Holland about this, who said on Dec 3, 2008:

SO&P was protected only a few days ago when I last looked.

When I revisited the site a couple of days later, the site now said:

Last updated: August 2008, Tim Osborn

This claim, that they updated the site in August 2008, yields a date which, if true, conveniently precedes both Mann et al 2008 and my FOI request on Sep 8, 2008 and validates their assertion that the data was “already” available when they responded to me. I must admit that I’m getting a bit cynical about these folks and I don’t believe that the webpage was “last updated” in August 2008, particularly given David Holland’s evidence on the matter. Additional evidence against this date being true is that that the link to the zipped file refers to a directory structure that did not exist until September 9, 2008. The webpage with information on the gridded sites was changed on Nov 16, 2008, a few days after Phil Jones’ complaint to the Santer 17. However, I didn’t personally check the site on October 31, 2008 and, as we know from our experience with Gavin Schmidt at Mann’s SI, even if you checked the website at 11.30 am, the data might have changed by 12.15 pm the same day, so you have to watch pretty carefully. As noted above, David Holland says that he checked a few days ago and, unless he erred, it hadn’t been unprotected then. Maybe I’ll send an FOI request asking for the exact date on which the password protection was actually lifted. [Update - a commenter below observes that the Google cache of this page taken on Sep 12 shows that the passwords have been removed. My guess is that the password protection was removed on Sep 9, the day after my FOI request on Sep 8, perhaps by coincidence. The dating is a small curiosity and I think that the Sep 9 date is pretty much established.]

Aside from being an important chapter in a data request that has been going on for years now, there are some interesting features to the new version of the data (which differs in important aspects from other versions), which I’ll discuss in another post.

And, oh yes, Another Brick in the Wall which nicely articulates the Team’s attitude towards questions.

22 Comments

  1. Tom
    Posted Dec 7, 2008 at 7:06 AM | Permalink | Reply

    Great work in motivating the Team to reveal more Mann “data”. I read your website daily.

  2. Dave Dardinger
    Posted Dec 7, 2008 at 8:26 AM | Permalink | Reply

    It would seem to me that, notwithstanding any past animosities, when team members make available data that there’s been a request for in the past, they should send a short e-mail to the requester concerning this fact. Given the message your gave that was sent to the Santer 17 they obviously know you’re after certain data, so why not get credit instead of the Dan Rather treatment?

  3. Steve McIntyre
    Posted Dec 7, 2008 at 8:34 AM | Permalink | Reply

    #3. Contrary to how people try to pigeonhole my attitudes, I do not have any “animosity” towards these folks. That sort of personal “animosity” is an attitude far more prevalent in academia than in business, where you learn to get along with yesterday’s opponent. I think that the Team have data obligations that they do not observe. I view it as a tribal custom. I think that it is deserving of satire and ridicule. But I don’t have “animosity” towards them on this count, though the opposite is probably not the case.

    As to your point about notice, I agree. In a professional operation, if there were outstanding requests for data and the data status changed to available, a professional operation would send out notices to interested parties or provide some other notice. But these operations are pretty “amateur”.

    • Dave Dardinger
      Posted Dec 7, 2008 at 3:45 PM | Permalink | Reply

      Re: Steve McIntyre (#4),

      Sorry if you thought I was saying you had animosity toward them. It was their animosity toward them I meant. The plural form had to be used because they got mad at you at various times over various points.

  4. bill-tb
    Posted Dec 7, 2008 at 8:36 AM | Permalink | Reply

    Your efforts are to be applauded by all. You have to wonder, what has science become.

    • fFreddy
      Posted Dec 7, 2008 at 9:16 AM | Permalink | Reply

      Re: bill-tb (#5),

      You have to wonder, what has science become.

      I prefer to think that science hasn’t changed, it’s just that climatology is not science.

  5. Posted Dec 7, 2008 at 8:42 AM | Permalink | Reply

    I look forward to your post on the recent changes of state in the data flux continuum.

  6. jae
    Posted Dec 7, 2008 at 9:39 AM | Permalink | Reply

    Ahhh, it’s great to see Steve’s persistence paying off, at last. Congrats. Parts of climate science have lost a lot of credibility over this stonewalling and obfuscation nonsense. If the general public understood what has been going on, some of these guys might be flipping hamburgers, instead of touring the world in CO2-spewing jets in the name of saving it from CO2.

  7. Andy
    Posted Dec 7, 2008 at 12:29 PM | Permalink | Reply

    It’s breathtaking that such activities are allowed to continue.

  8. David Holland
    Posted Dec 7, 2008 at 12:35 PM | Permalink | Reply

    Steve,

    I may have erred. I did check the SO&P site and it did have lots of little key icons and statements saying some files are password protected. In all honesty I cannot remember if I went as far as getting to the password page for any of them. However I have just looked now and can only find one key icon like this, where previously there were lots. However I have not actually found any data that is still password protected. They are not open and transparent.

    My guess is that they took off the passwords first and have been following up by editing out references to passwords so in all honesty I cant say when they did it. But they mucked up their server a few days ago which is the sort of thing that happens when you do this sort of change.

    However the point is this. SO&P was funded by the EU who also pushed for the Aarhus Convention under which (via the Environmental Information Regulations 2004 in the UK) all public authorities have to proactively disseminate environmental information. They have had this duty since 1 Jan 2005 and should not have had any passwords after that date. But on page 47 of their final report on SO&P dated September 06 is this statement.

    Note that some of the deliverables are password-protected because they are not appropriate for completely open dissemination. The password can be requested from the project coordinator (Tim Osborn, t.osborn@uea.ac.uk). Data will be provided in most cases for agreed collaborative projects.

    This is obviously not Aarhus compliant but just another way of saying “we are not going to let you have our data if all you want to do is find fault”.

    It is going to take a while to get looked at but my UEA/CRU complaint will be with the ICO tomorrow and unless the long overdue MOD/Met Office ‘internal review’ agrees to disseminate all the IPCC information which I have asked for, they will be next.

    • Paul Foote
      Posted Dec 8, 2008 at 11:55 AM | Permalink | Reply

      Re: David Holland (#10), Good show, David Holland! I appreciate your effort there and I am sure many others will too.

  9. nanny_govt_sucks
    Posted Dec 7, 2008 at 1:51 PM | Permalink | Reply

    If they just did some paleo fieldwork with trees, corals, sediment cores they might understand why some samples are excluded.

    Might understand?

    So even if one went and did field work in each of the areas of dendro, corals, AND sediment, then you might understand which selections were made!

  10. David Holland
    Posted Dec 7, 2008 at 2:29 PM | Permalink | Reply

    I did not carefully enough. These pages at the SO&P site are still password protected.

  11. Soronel Haetir
    Posted Dec 7, 2008 at 3:13 PM | Permalink | Reply

    We don’t need no bad proxies,
    We don’t need spliced series!
    No infilled records in the data!

    Hey dendros, leave those trees alone!

  12. Posted Dec 7, 2008 at 5:34 PM | Permalink | Reply

    It almost seems like it would be worth some effort to have a robot crawl all their sites every 12 hours or so. You wouldn’t catch the really quick changes, but it would put the lie to data being updated in August when it was actually updated in October.

    Is there someone who has the technical skill and computing power available who could take a list of Web sites provided by Steve and create a robot to do the crawling on a timely basis that wouldn’t put undue stress on the “helpful” researchers schedules?

  13. Nick Moon
    Posted Dec 7, 2008 at 6:39 PM | Permalink | Reply

    Looking at the google cache, that page had been modified by 12th September – password icons had been removed. But the date at the bottom of the page still reads November 2005. I think it might be reasonable to assume they did update the web page in August but forgot to change the date. The date has been fixed some time after September 2008.

    The Wayback Machine at archive.org has snapshots of the page – with the password protection – dated to 2004.

    So cockup rather than conspiracy.

  14. Steve McIntyre
    Posted Dec 7, 2008 at 7:03 PM | Permalink | Reply

    #17. The date has only been changed to August 2008 in the last few days – I can attest to that because I notified them of the incorrect date a couple of days ago.

    I still don’t believe the August 2008 date. The file structure is dated Sep 9, 2008 and the link shown in the page as of Sep 12 could not have existed prior to that.

    My FOI was sent in on Sep 8, 2008. Data directories are stamped the next day. PErhaps it was a coincidence, perhaps not. In any event, the evidence looks very like they unlocked the data following my FOI request on Sep 8, but before my follow-up on Oct 31. This was not mentioned in their response to my FOI request. While they unlocked the data, key webpages continued to say that data was password-protected, tho not the one where the zip file was directly called.

    In any event, I think that this timing is now pinned down pretty accurately (and that the data was probably unlocked on Sep 9).

  15. stan
    Posted Dec 7, 2008 at 7:22 PM | Permalink | Reply

    Sounds like some folks need to get a Congressional subpoena for testimony before a committee.

  16. nevket240
    Posted Dec 8, 2008 at 7:02 PM | Permalink | Reply

    This is obviously not Aarhus compliant but just another way of saying “we are not going to let you have our data if all you want to do is find fault”.

    What these “people” conveniently avoid is responsibility for their incompetence. If they are wrong, and it is shown where they are wrong, surely mature professionals would have the behavioural standards to accept and progress?? (my bad)
    regards

  17. Sam Urbinto
    Posted Dec 9, 2008 at 11:40 AM | Permalink | Reply

    If only certain people wouldn’t publicly mis-categorize situations in press releases, web sites, blogs, and emails, it wouldn’t be required to establish dates and the like to show the actual situation.

    It would seem sooner or later, it would get tiring having the details show you were incorrect, and you’d be more careful on such matters in the future. It would certainly save Steve a lot of time. Perhaps that’s the point, eh?

    I’m sure the directories being stamped the day after the request is simply a coincidence.

  18. Posted Dec 9, 2008 at 3:02 PM | Permalink | Reply

    Steve, one more thank you to add to your collection. Your work is stalwart.

    As a retired businessman, not scientist, it is quite amazing to me that Dr. Jones of CRU displays the petulance and defensiveness he does over his findings. Regrettably, if everything I read elsewhere on the AGW thesis is an indicator, his behaviour is common amongst the scientists working to promote AGW.

    Such contact as I have with ethical scientists reinforces the view that real scientists work harder than anyone else to interrogate their own work to seek out weaknesses. The ones who dislike your enquiries fail this test.

    I do hope you keep it up.
    Yours aye
    Bob V T

One Trackback

  1. [...] of the struggle to bring data in to public view is seen in these posts at Climate Audit:  here and [...]

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 3,145 other followers

%d bloggers like this: