Myles Allen, a declared supporter of open data archives, has, in blog comments here, proposed “name and shame” as a first tactic against data obstructionists (as opposed to FOI).
Journal editors can and should enforce a simple “disclose or retract” policy if a result is challenged, and almost all of them do: if any don’t, then the solution is to name and shame them, not set up a parallel enforcement system.
I partly agree with this; I’ve used FOI primarily as a last resort. And in the case of climate scientists and journals that withhold data unashamedly, I believe that it remains a valuable tool of last resort. Obviously I’ve not been shy about naming data obstructionists at Climate Audit, though this longstanding effort has typically encountered resentment rather than encouragement from the community. Perhaps Allen will add his voice in more direct communications with editors rather than just at blog comments. Regardless, it’s nice to get even some moral support, since, for the most part, the community has united in solidarity behind data obstructionists.
By coincidence, Myles’ comments come in the midst of another data non-archiving incident that I haven’t reported on.
Real Climate recently praised a new multiproxy study (Gergis et al 2012). Gergis et al is the fourth or so multiproxy article co-authored primarily by Raphael Neukom and Joelle Gergis. Several articles in this corpus are cited prominently in AR5.
One of the coauthors, Ailie Gallant, was a featured performer in the recent We Are Climate Scientists anthem, with her cameos occurring during the memorable declaration:
We’re climate scientists. What we speak is true. Unlike Andrew Bolt, our work is peer reviewed. Haaaa…
None of the data for the earlier articles was archived. Or, more accurately, it was archived in a secret Swiss databank only accessible to the illuminati. CA readers will recall that I requested data for an earlier article from co-author Neukom and was blown off ( see CA post here.)
Gergis et al 2012 and the “Screening Fallacy”
CA readers will recall the long-standing blog criticism of the “Screening Fallacy”, not just here, but at other technical blogs as well. Not understanding the problem is almost the litmus test of being a professional climate scientist.
The error is committed once again by Gergis et al. They described the selection of 27 proxies from a network of 62 as follows:
Our temperature proxy network was drawn from a broader Australasian domain (90E–140W, 10N–80S) containing 62 monthly–annually resolved climate proxies from approximately 50 sites (see details provided in Neukom and Gergis, 2011)…
Only records that were significantly (p<0.05) correlated with the detrended instrumental target over the 1921–1990 period were selected for analysis. This process identified 27 temperature-sensitive predictors for the SONDJF warm season (Figure 1 and Table 1) 228 henceforth referred to as R27.
On the surface, screening a network of proxies for correlation to temperature seems to “make sense”. But the problem is this: if you carry out a similar procedure on autocorrelated red noise, you get hockey sticks. If you think that a class of proxy is a valid temperature proxy, then you have to define the class ahead of time and take it all. No after the fact “screening”. [Note - June 1] Gergis et al 2012 say that their screening is done on de-trended series. This measure might mitigate the screening fallacy – but this is something that would need to be checked carefully. I haven’t yet checked on the other papers in this series. (Update 2: Despite the above statement in their article, Gergis et al did not screen on detrended data after all. The article is presently removed from the Journal of Climate website and “on hold” – see other posts in this sequence)
I discussed the screening fallacy (not by that name) in early CA posts e.g. here, as a criticism of Jacoby’s selection of the 10 “most temperature sensitive” sites from a network of 36. (Jacoby compounded the problem by refusing to provide data from the “other 26 sites” when requested, a refusal supported by the journal (Climatic Change) and the U.S. National Science Foundation.
David Stockwell wrote about the fallacy in 2006 – blog post here, article in AIG News here. Ross and I wryly cited Stockwell, 2006 (AIG News) in our PNAS Comment on Mann et al 2008; Mann et al fulminated against the temerity of citing AIG News but did not refute the point.
Mann et al 2008, better known for the Upside-Down Tiljander problem that also baffles climate scientists, also committed the screening fallacy. (Mann et al 2008, like his earlier article, is a laboratory of statistical horrors.) Jeff Id, then a new blogger, wrote two excellent blog posts in Sept 2008 here here commenting on this error in Mann et al 2008. Lubos Motl’s follow-up article here is a further demonstration of the phenomenon.
Lucia re-visited the problem in October 2009 (on the eve of Climategate), pondering why it so befuddled climate scientists. See her clear exposition, aptly entitled Tricking Yourself into Cherry Picking.
Unlike their earlier articles, Gergis et al at least archived the 27 “temperature-sensitive predictors”. But like Jacoby, they did not archive the 35 “climate proxies” that they didn’t use. I asked them for these other proxies. While I was at it, I also asked them for digital versions of the proxy networks used in Neukom and Gergis 2011 (Holocene), Neukom et al 2011 (Clim Dyn) and Neukom et al (2010).
None of my requests have thus far been acknowledged by the authors.
I also sent these requests to editors of the four journals, urging them to require the authors to archive their data, and, if the authors were unwilling or unable to do so, require the retraction of the article – a remedy that Myles Allen seems willing to support.
I also wrote to Valerie Masson-Delmotte, CLA of the paleoclimate chapter of AR5, notifying her of the data archiving issue for articles cited in AR5. (I met her at AGU one year and talked to her for a while and got a good impression of her.)
John Matthews of Swansea Univesity, editor of The Holocene, responded promptly saying that The Holocene does not require authors to archive data, that the article satisfied the referees and that was that.
The Holocene does not have a policy of requiring its authors to archive their data sets. We offer the facility of publishing Supplementary Material on-line. It is up to the authors whether or not they archive their data sets. Authors are expected to describe their data sources and methods adequately, which in this Research Review was done in detail and to the satisfaction of our referees. There is, in my view, no case for retraction.
I forwarded Matthews’ refusal to Masson-Delmotte, urging that IPCC accordingly de-certify The Holocene as an eligible citation in AR5.
As Dr Matthews says in his letter, the journal Holocene does not have an adequate or indeed any policy of requiring authors to archive data. Unless it establishes such a policy, I suggest that IPCC not permit the citation of articles from Holocene in the forthcoming assessment report.
Perhaps Myles Allen can encourage Matthews of the merits of adopting a data policy.
I next heard from Anthony Broccoli, editor of Journal of Climate. (Broccoli had acted as editor of O’Donnell et al 2010 and had required us to make major revisions and resubmissions to accommodate Eric Steig, aka the anonymous Reviewer A.) Broccoli unresponsively told us to contact the authors – even though the reason for contacting Broccoli had been the past uncooperativeness of the authors and the authors had not responded to the present email:
Thank you for your inquiry. Please communicate directly with the authors regarding access to their data.
Lest a ball remain in my court, I emailed Neukom and Gergis one more time, including an extra pretty please this time (literally):
Gergis et al 2012 states:
Our temperature proxy network was drawn from a broader Australasian domain (90E–140 W, 10N–80S) containing 62 monthly–annually resolved climate proxies from approximately 50 sites (see details provided in Neukom and Gergis, 2011).
You’ve archived the 27 series that you screened from the 62, but have not archived the original population of 62 series that entered into the analysis. Could you please provide me with a copy of this data.
Pretty please with sugar on it,
Despite using these magic words, I haven’t heard back from the authors.
I did hear back from Valerie Masson-Delmotte, whose response was somewhat positive about ensuring that the Second Draft would be constructed around publicly available data:
I thank you for your concern about the IPCC AR5 assessment, and the references cited in the first order draft. The IPCC First Order Draft review process would have been a standardized way to send your comments and suggestions to all Chapter 5 authors. It is our duty to make this assessment as transparent as possible, following IPCC guidelines and IAC recommendations. We are aware that not all funding agencies or publishers follow a consistent strategy with regard to the public release of data associated with published articles. Regarding your specific concerns, we are confident that the next draft of our chapter will be based on new publications associated with publicly available datasets.
This is encouraging.
Update: June 1 12>30 am:
Joelle Gergis has responded blowing off my request (cc to David Karoly, Valeria Masson-Delmotte (AR5 CLA) and the editor of Journal of Climate). She says that I should try to get the unarchived data from the original authors, saying snottily that this is “commonly called ‘research'” and that they “will not be entertaining further correspondence” on the matter.
We have already archived all the records needed to replicate the analysis presented in our Journal of Climate paper with NOAA’s World Data Center for Palaeoclimatology:
While the vast majority of the records contained in the full Australasian network are already lodged with NOAA, some records are not yet publically available. Some groups are still publishing their work, others have only released their data for use in a particular study and so on.
The compilation of this database represents years of our research effort based on the development of our professional networks. We risk damaging our work relationships by releasing other people’s records against their wishes. Clearly this is something that we are not prepared to do.
We have, however, provided an extensive contact list of all data contributors in the supplementary section of our recent study ‘Southern Hemisphere high-resolution palaeoclimate records of the last 2000 years’ published in The Holocene (Table S3):
This list allows any researcher who wants to access non publically available records to follow the appropriate protocol of contacting the original authors to obtain the necessary permission to use the record, take the time needed to process the data into a format suitable for data analysis etc, just as we have done. This is commonly referred to as ‘research’.
We will not be entertaining any further correspondence on the matter.
Dr Joelle Gergis
Climate Research Fellow
UPDATE: For the record, I have had totally the opposite experience with Fredrik Ljungqvist. I’ve contacted him in the past and re-contacted him today. He has been promptly helpful and consistently cordial.