One of the most important IPCC representations is the supposedly tremendous quality control of its review process. I’ve mentioned in passing on a number of occasions that, when I sought to obtain supporting data for then unpublished articles, IPCC threatened to expel me as a reviewer.
I’ve had a few requests to recount my experience with trying to get data from IPCC for unpublished studies. So here’s a short summary of my correspondence with IPCC.
On August 1, 2005, I was invited by IPCC to act as a reviewer. (I guess this makes me one of the 2500 scientists who support IPCC conclusions, although my review comments have all been ignored as far as I can tell.)
You have been nominated to serve as an Expert Reviewer for the Working Group I contribution to the IPCC Fourth Assessment Report, Climate Change 2007: The Physical Science Basis. The first draft of this report will be available for expert review from Friday, 9 September 2005, with all review comments due by Friday, 4 November 2005.
I accepted. In September 2005, I noticed that the Paleoclimate chapter cited two then unpublished studies by D’Arrigo et al (later D’Arrigo et al 2006) and Hegerl et al (later Hegerl et al J Clim 2006). In order to carry out my responsibilities as a reviewer, I wanted to see the supporting data for these studies and I accordingly wrote to the IPCC Technical Services Unit at UCAR in Boulder on Sep 20, 2005 as follows:
I have been unable to locate supplementary information or data archives for several of the articles posted at the pdf location for Chapter 6 and would appreciate assistance in this regard.
1) Hegerl et al, submitted. Can you provide me with an ftp location for the proxy data used in this study (which does not even list the proxies used) or post it at your website.
2) D’Arrigo et al, submitted. Again, this data has not been archived at WDCP. Can you provide me with an ftp location for the proxy data used in this study or post it at your website.
On Sep 22, 2005, Martin Manning of the IPCC/UCAR TSU wrote back refusing to provide this data in the following terms:
… It is normal practice that expert reviewers of scientific works check the references given and the way they are used. We certainly expect this during the review of the first draft of our report and are grateful that you have identified an issue that the authors will need to deal with in the next draft if that can not be done now.
The second issue is availability of data used in cited literature. As you have recognized some of this is available at data centers. Often the original authors of the cited papers will release their data on request. However, the IPCC process assesses published literature, it does not involve carrying out research, nor do we have the mandate or resources to operate as a clearing house for the massive amounts of data that are used in the climate science community or referred to in the literature used by our authors. Given the many different approaches to intellectual property and data release in different countries and agencies such an undertaking would in any case not be possible.
I was obviously unsatisfied with their failure to provide supporting data and re-iterated my request for supporting data as follows:
My request for data pertains to two papers which are presently unpublished and for which the data is unarchived. One of the papers does not even list the data used. I request that you simply contact the authors who submitted the articles in question and ask him/her to provide an FTP location for the data so that it can be reviewed. The request can be made through a simple email and does not require resources beyond those available to you. You could have submitted the request as quickly as it took you to draft your reply to me. If the authors refuse to provide their data pursuant to a request from you, then that would be a factor in my review, as it should be for IPCC itself, as to whether the article should be referenced by IPCC.
The next day, Sep 23, 2005, Manning made the following shirty reply:
Let me repeat – If you wish to obtain data used in a paper then you should make a direct request to the original authors yourself. It would be inappropriate for the IPCC to become involved in that communication and I have no intention of allowing the IPCC support unit to provide you with what would in effect be a secretarial service. There are over 1200 other scientists on our list of reviewers and we simply can not get involved in providing special services for each. I gave you the courtesy of a detailed response earlier to ensure there was no confusion about our process which is my responsibility. Acting as an intermediary with other scientists is not. I will not be responding to further correspondence on this matter.
Now I had presumed that a unit entitled Technical Services Unit would not view “secretarial services” as beneath their dignity. Perhaps they’d been watching too many episodes of 24 and got the TSU and CTU units mixed up. In addition, by requiring me to contact the authors directly, obviously the anonymity of the process was forfeited. I thought that Manning’s answer was unacceptable and immediately appealed his decision to Susan Solomon, Chair of WG1, this time referring to specific IPCC policies on data, on Sep 23, 2005, as follows:
Dear Dr. Solomon,
As a reviewer of IPCC 4AR, I requested the assistance of the WG1 TSU in obtaining data pertaining to unpublished articles referenced in the Draft WG1 Report. This request was made in accordance with the document “PROCEDURES FOR THE PREPARATION, REVIEW, ACCEPTANCE, ADOPTION, APPROVAL AND PUBLICATION OF IPCC REPORTS” http://www.ipcc.ch/about/app-a.pdf , which states:
“The Working Group/Task Force Bureau Co-Chairs should make available to reviewers on request during the review process specific material referenced in the document being reviewed, which is not available in the international published literature.”
Dr. Manning of the WG1 TSU replied that he would not do so and that he had no intention of allowing the TSU to perform what would be a “secretarial service”. I would have thought that the TSU’s functions would include certain “secretarial services”, including the provision of support for Co-Chairs in discharging their obligations under the above policy. In any event, pursuant to the above policy, I request the following Chapter 6 information previously requested from Dr Manning:
1) An ftp location for the proxy data (or the proxy data itself) for Hegerl et al. submitted
2) An ftp location for the proxy data (or the data itself) for D’Arrigo et al., submitted.
3) Copies of Briffa et al., 2005 and Wilson and al. 2005, referenced in Chapter 6.
Thank you for your attention.
In addition, I wrote directly to the two authors, Hegerl and D’Arrigo, as Manning had directed me to do. On Sep. 24, 2005, I wrote to author D’Arrigo as follows (with a similar letter to Hegerl):
Dear Dr D’Arrigo,
I have accepted an invitation to act as a reviewer for IPCC 4AR. Your submitted article “On the Long-Term Context for Late 20th Century Warming” is cited. For my review, could you please provide me with an FTP location for (or otherwise provide me directly by email) the measurement data used in this study. Thank you for your attention.
Yours truly, Stephen McIntyre
In both cases, the authors refused to provide any data – including even the identification of sites. On Sep 25, 2005, Hegerl wrote, politely refusing as follows:
Once my paper comes out, the data will be accessible to the community via an electronic supplement, but I cannot release the data before the review process of the PAPER is finished. Please direct any corresponding concerning the IPCC draft to the TSU.
On Sep 26, 2005, a D’Arrigo co-author wrote to me, advising that the article had been submitted to JGR, that they would not provide any data to me and that:
Any data requests (and their availability) will have to be made via the Editors of any relevant journals.
Following these refusals by D’Arrigo and Hegerl, I wrote again to Susan Solomon, notifying her of these refusals and asking her to re-consider the IPCC refusal to require these authors to provide supporting data, with one letter about Hegerl as follows (and a similar letter about D’Arrigo):
Dear Dr Solomon,
I have attempted to obtain the data for Hegerl et al 2005, submitted, from the original authors without success. As I previously mentioned to you, it is my understanding that IPCC review is independent of journal review and, as mentioned before, that the following IPCC policy require you to make this unpulished data available. “The Working Group/Task Force Bureau Co-Chairs should make available to reviewers on request during the review process specific material referenced in the document being reviewed, which is not available in the international published literature.”
I reiterate my request that you provide the requested proxy data for Hegerl et al, 2005 pursuant to this policy.
Thank you for your attention,
I also pursued the avenue with the journal, which the D’arrigo et al group had identified as having control of the matter. On Sep 27, 2005, I wrote to the JGR editor saying:
Dear Dr O’Dowd,
I am a reviewer for the IPCC Fourth Assessment Report (IPCC 4AR) and am writing in respect to a submission to your journal by D’Arrigo et al., entitled “On the Long-Term Context for Late 20th Century Warming.” This article was referenced in chapter 6 of the Draft IPCC 4AR and made available to IPCC reviewers. In the course of my review, I contacted the senior author, Dr. D’Arrigo, for the FTP location of the data used in this article or for alternative access to the data. Dr D’Arrigo categorically refused and I was referred to the journal editor if I desired recourse.
Data Citation and Archiving
I point out that AGU policies for data citation and data archiving (http://www.agu.org/pubs/data_policy.html ) specifically require that authors provide data citation according to AGU standards and require that contributors archive data in permanent archives, such as the World Data Center for Paleoclimatology. For example, the policy states:
1. Data sets cited in AGU publications must meet the same type of standards for public access and long-term availability as are applied to citations to the scientific literature. Thus data cited in AGU publications must be permanently archived in a data center …
2. Data sets that are available only from the author, through miscellaneous public network services, or academic, government or commercial institutions not chartered specifically for archiving data, may not be cited in AGU publications.
On page 21 of D’Arrigo et al., there is a listing of “regional groupings” of data. In some cases, part of the data is archived at WDCP; in other cases, the data has been collected by the authors, but has not been archived. In cases, where the data has been archived, it has not been cited according to AGU policies. For example, the Torntraesk site is presumably swed019w, but this is not stated. The Polar Urals site appears to be a combination of russ021w, russ176w and russ022w, but this is not stated. The Quebec site appears to be a version of cana036, but a version that differs from the one archived, as it includes more series. The “Mongolia” site appears to be the authors’ mong003 site, but a different version than the one archived (which commences at a different date). The “Yukon” series is a combination of two sites, which are not stated. At least one of the sites is a different version from the one archived. The Icefields site is again a different version than the one archived. Other data sets e.g. Seward, NW North America, Central Alaska, Wrangells, Coast Alaska, Central NWT, Southern Alaska, have been collected by the authors and are either not archived at all or archived in obsolete versions.
In order that this submission comply with AGU policies on data archiving, I request that you require D’Arrigo et al. do (1) provide accurate data citations complying with AGU policies for all data sets presently archived at WDCP; (2) archive all “grey” data used in the article.
[Update Sep. 2012] On Sep 28, 2005, (CG1- 590. 1128000000.txt), Rob Wilson emailed Osborn and Briffa, both IPCC Authors, and complained about my email to O’Dowd (which had been copied to Wilson). Osborn immediately replied, calling the request that the data be archived an abuse of my position as an IPCC peer reviewer. Rosanne D’Arrigo (CG2-2590) wrote to Osborn and Briffa, advocating that I be “fired” as an IPCC reviewer. D’Arrigo urged the Climategaters to be “very cautious about our emails as Lord V will stop at nothing”:
i am leary of passing all of this around but in this case i am glad in that osborn et alneed to know what is going on – they should fire him as a reviewer of IPCC – i cant believe they included him in the first place! So, please email him back and tell him that he should as he says take it up with the ipcc authors and see whether it is still appropriate to include him as a reviewer. we should however be very cautious about our emails, lord v will stop at nothing (this is sort of fun in a harry potter way)…
The next day (Sep 29), Osborn wrote (CG2 -4868) to IPCC Chapter Authors (Overpeck, Jansen) and Jones; Jones had undertaken to take the matter up with IPCC WG1 Chair Solomon.
On Sept 30, 2005, I received the following reply from Susan Solomon, Co-chairman of IPCC Working Group 1:
iThis is in response to your several messages regarding the IPCC review process.
First, as has already been explained to you in previous correspondence, your interpretation of IPCC procedures in relation to what is made available to reviewers is not correct. The term “materials referenced” used in our rules is unambiguously defined by the list of such “references” given at the end of each chapter. The term does not extend beyond those cited references to such material as datasets, computer codes, or other sources of information that those papers may themselves cite or use. As has already been detailed for you by Dr. Manning, the IPCC does not and cannot provide datasets associated with each of the papers cited in the review, whether published or unpublished.
In order to access the IPCC review material you explicitly agreed to our conditions for doing so which include the following:
“This site also provides access to copies of some submitted, in-press, or otherwise unpublished papers and reports that are cited in the draft WG I report. All such material is made available only to support the review of the IPCC drafts. These works are not themselves subject to the IPCC review process and are not to be distributed, quoted or cited without prior permission from their original authors in each instance.”
We are now aware that you have used your access to unpublished material on our review web site to attempt to influence editorial decisions by the Journal of Geophysical Research. This is clearly a substantial breach of the conditions of access on two counts.
First you have not used the material solely for the purpose of reviewing the IPCC drafts, and second you have, without prior permission, cited unpublished material to the JGR editors in your attempt to influence them. Such actions are not appropriate. The IPCC process can not supercede or alter the scientific review of papers followed by individual scientific journals.
These considerations and the explanations you have already received made it clear that it is inappropriate to cite a function as a reviewer in the IPCC process as entitling you to access to additional information from authors of the unpublished papers available at our web site, which you have also done.
Dr. Manning has already answered your other repeated question regarding two missing references: that, and any other views you may have, can be pointed out in your review comments, to be submitted via the formal process.
Finally, we must insist that from now on you honor all conditions of access to unpublished, and therefore confidential, material made available for the IPCC review process. The IPCC rules for reviewing draft reports have served the scientific and policy communities well for numerous past international assessment rounds. If there is further evidence that you can not accept them, or if your intent is to use your access to the review process to challenge them, then we will not be able to continue to treat you as an expert reviewer for the IPCC.
On Sep 30, 2005, I replied to Solomon as follows:
Dear Dr Solomon,
First of all, any of my inquiries have been for the sole purpose of submitting a review to IPCC.
In my opinion, examining the underlying data is an important part of reviewing materials. This opinion is obviously widely shared as the provision of supplementary information is standard for most paleoclimate articles (though not as widely shared as I would like) and many journals have policies requiring the archiving of data used in articles (although the policy is not always upheld.) Unavailability of underlying data would be a point that I would raise in my planned review.
I did not instigate any contact with any authors without first contacting TSU. My first request for supporting data was to IPCC TSU, not from the originating authors. It was my view (and remains my view) that this data should have been obtained by TSU and avoid my being put in the awkward position of having to have direct contact with the authors, which I did not wish,
In reply to my original inquiry, Dr Manning did not advise me that it was inappropriate for me to try to examine the supporting data. On the contrary. His response was that, if I wished to do so, I should contact the authors directly and that it was not TSU’s function to carry out a “secretarial service” in the following words:
“If you wish to obtain data used in a paper then you should make a direct request to the original authors yourself. It would be inappropriate for the IPCC to become involved in that communication and I have no intention of allowing the IPCC support unit to provide you with what would in effect be a secretarial service. There are over 1200 other scientists on our list of reviewers and we simply can not get involved in providing special services for each. I gave you the courtesy of a detailed response earlier to ensure there was no confusion about our process which is my responsibility. Acting as an intermediary with other scientists is not.”
The issue for Dr Manning was not obtaining the data — it was just that TSU wasn’t going to do it for me. I would have much preferred it if TSU had done so, as I had originally requested. This contradicts the statement in your email::
“These considerations and the explanations you have already received made it clear that it is inappropriate to cite a function as a reviewer in the IPCC process as entitling you to access to additional information from authors of the unpublished papers available at our web site, which you have also done.”
The explanation which I had “already received” said exactly the opposite: if I wanted to examine the underlying data, I should approach the individual authors. The issue of seeking data had been fully discussed with TSU and I was merely following their instructions in contacting the original authors.
I apologise for any misunderstanding, but I do not acknowledge any breaches of IPCC reviewer obligations on my part.
Yours truly, Stephen McIntyre
I received no answer to this letter. Later in 2005, at the CCSP workshop, I suggested to Susan Solomon as a panelist that IPCC should require authors who submitted papers for citation to consent to provide data; she said that that would be interfering with the journals.
(Fast forward briefly: In late 2006, I asked Hegerl for detailed information on Mongolian and Urals site used in her study. They have undertaken to provide the requested information but I still don’t have it. Despite the reminder to JGR about AGU data archiving policies, JGR did not require D’Arrigo et al to adhere to AGU data archiving policies and DArrigo et al did not archive their data with ITRDB at the time of publication or provide a SI with data citations compliant with AGU policy. At present, there is still no listing of the sites used in D’Arrigo et al 2006 and the majority of sites remain unarchived.)
Oct 2009 – Crowley, a Hegerl et al coauthor, has challenged me in correspondence with Andy Revkin see- DotEarth here. . In his email to Revkin, Crowley acknowledged that he was “overdue in getting you some data set you’d requested.” It is, of course, the data originally requested in fall 2005 in connection with the review of AR4.
On October 19, 2009, I re-iterated my request to JGR editor O’Dowd for the D’Arrigo et al data, a request immediately passed on to the Climategaters (CG2-2374). My letter to O’Dowd:
Date: Mon, 19 Oct 2009 11:00:21 -0400
From: Steve McIntyre
To: ‘Colin O’Dowd’
CC: Rosanne D’Arrigo
Dear Dr O’Dowd,
In fall 2005, I corresponded with you in your capacity as Editor, JGR seeking data for D’Arrigo et al 2006 (then submitted), which was being cited by IPCC WG1. In that email, I referred to AGU policies for data citation and data archiving www.agu.org/pubs/data_policy.html which requires that data cited in AGU pulications be located in a permanent archive.
Despite this correspondence, the data archive for D’Arrigo et al 2006 remains abysmally incomplete. There is no data section describing where the data sets referenced in their Table 1 can be located. Indeed, for some of the regional composites, there is not even any information on which sites are included in the regional composite nor is the measurement data for individual sites available (e.g. Coastal Alaska, Central NWT and others.) Nor are the “chronologies” that result from the analysis carried out in the paper archived.
I request that you take steps to ensure compliance with AGU data archiving policies by ensuring that proper listing of the sites for each Table 1 composite is provided and that the measurement data for each such site is located in a public archive.
There is also an important error in the description of a key series. For the RCS chronology labelled “POL”, core counts are shown for the Polar Urals site, while the RCS chronology actually shown is from a different site entirely (Yamal), which has much lower replication in the modern period.
Jones immediately expressed confidence to Wilson and Briffa that JGR/AGU would not require D’Arrigo and Wilson to comply with AGU policies:
date: Mon Oct 19 16:44:47 2009
from: Phil Jones
subject: Re: [Fwd: D’Arrigo et al 2006]
to: Rob Wilson , Keith Briffa
This might constitute joining the Team – although as I said the Team doesn’t exist! I doubt JGR/AGU will follow this up.
Sep. 2010 This dispute is discussed in Climategate email 590. 1128000000.txt. On Sep 28, 2005, Rob Wilson emailed Osborn and Briffa, complaining about my email to O’Dowd (attached to his email. Osborn immediately replied, calling the request that the data be archived an abuse of my position as an IPCC peer reviewer. Osborn then wrote to the IPCC Chapter Authors (Overpeck, Jansen) and Jones on Sep 29. The next day, Susan Solomon replied.