per inquired recently about obtaining a copy of Gerry North’s presentation to the newly minted NAS Panel on Assuring the Integrity of Research Data, which held its first hearings last week. Gerry North was appropriately the first speaker, as the new panel was occasioned by problems left unanswered by the North panel, although its terms of reference are much broader. The North presentation is here. Some background and thoughts follow.
The NAS Panel was formed at the request of the House Science Committee, protecting their turf from previous questions sent by the House Energy and Commerce Committee to MBH, IPCC and the NSF. Because of the controversy over replicability, one of the key Boehlert questions was:
(2) (c) Has the information needed to replicate their work been available? (d) Have other scientists been able to replicate their work?
At the NAS panel hearings, the panel was nonplussed when von Storch put this question on the board as Ralph Cicerone, the President of NAS, had excluded this question from the terms of reference of the panel and they knoew nothing of the Boehlert questions (although these were widely disseminated). My contemporary discussion, Sir Humphrey and the Boehlert Questions, is here. It seemed that the NAS Panel was more interested in dealing with “big” questions rather than giving an opinion on the controversy that had occasioned their appointment.
At the end of the first day, David Goldston of the House Science Committee spoke up in the public portion, asking that the panel actually take some of the smaller controversies off the table, noting that there was going to be ongoing discussion of the “big” questions and lots of other occasions to deal with them. Several weeks later, the terms of reference were modified (contemporary CA report here) as follows:
- Comment [Evaluate -deleted] on the overall accuracy and precision of such reconstructions, relevant data quality and access issues, and future research challenges.
I thought that it would be very unlikely that the panel would grasp this nettle in view of their virtual total failure to even ask Mann about the questions raised by the House Committees and their total failure to follow up on Mann’s bizarre denial of even calculating the verification r2 statistic. And they didn’t, with only some highly generalized observations that providing data was a good thing. A contemporary report at CA is here. The panel did not investigate or report on actual availability, merely observing platitudinously:
Our view is that all research benefits from full and open access to published datasets and that a clear explanation of analytical methods is mandatory. Peers should have access to the information needed to reproduce published results, so that increased confidence in the outcome of the study can be generated inside and outside the scientific community. Other committees and organizations have produced an extensive body of literature on the importance of open access to scientific data and on the related guidelines for data archiving and data access (e.g., NRC 1995).
I recall the idea of a new committee on Data being in the air around the time, but didn’t notice any contemporary notes. My guess is that Sir Humphrey Cicerone decided that, if they had no choice but to respond to these matters, his best approach was to dilute the paleoclimate problems where an answer was easy with complicated questions about software and every other scientific discipline under the sun – sort of like trying to develop a Napoleonic Code instead of making common law decisions.
Be that as it may, the new panel is entitled: Assuring the Integrity of Research Data (PIN: CSEP-Q-06-02-A ). Its terms of reference here say:
Project Scope: An ad hoc committee will conduct a study of issues that have arisen from the evolution of practices in the collection, processing, oversight, publishing, ownership, accessing and archiving of research data. The key issues to be addressed are:
1. What are the growing varieties of research data-?. In addition to issues concerned with the direct products of research, what issues are involved in the treatment of raw data, pre-publication data, materials, algorithms, and computer codes?
2. Who owns research data, particularly that which results from federally-funded research? Is it the public? The research institution? The lab? The researcher?
3. To what extent is a scientist responsible for supplying research data to other scientists (including those who seek to reproduce the research) and to other parties who request them? Is a scientist responsible for supplying data, algorithms and computer codes to other scientists who request them?
4. What challenges does the science and technology community face arising from actions that would compromise the integrity of research data? What steps should be taken by the science and technology community, research institutions, journal publishers, and funders of research in response to these challenges?
5. What are the current standards for accessing and maintaining research data, and, how should these evolve in the future? How might such standards differ for federally-funded and privately-funded research, and for research conducted in academia, government, nongovernmental organizations, and industry?
The agenda for their first meeting is here. At my request, North sent me his presentation and gave me permission to post it up. My comments here are similar to my earlier comment. The PPT leads with an anti-Barton editorial from the Houston Chronicle and then as separate slides on both realclimate and climateaudit. One of his last slides is my letter post-NAS Panel to North asking for help getting data. (Not a single piece of data has been provided; the committee apparently did not ask whether any of the information was ever provided to me.)
Some of the points that North highlighted give a hugely misleading impression of paleoclimate data issues. For example, Phil Jones obstructionist refusal to identify stations is well-known to CA readers. At the NAS Panel hearings, Hans von Storch posted a slide with Phil Jones famous refusal: “We have 25 years invested in this, why should we let you see the data when your only objective is to find something wrong with it?” von Storch said that he could not believe that a responsible scientist had made such a statement and that he asked Jones to confirm the truth of this story – which Jones did. Von Storch condemned this attitude in the strongest possible terms. Instead of bringing this to the attention of the Data Integrity Panel, North posted up the following trivializing email exchange involving Phil Jones as follows:
You don’t need to ask permission to get HadCRUT2v. It is sitting on our web site.
http://www.cru.uea.ac.uk/cru/data/temperature/ . . .Cheers Phil
Readers of CA are familiar with the fact that key data sets used over and over by the Team in multiproxy studies – Yamal, Taymir etc. – are not archived and that Briffa refuses to disclose the data. These problems were brought to the attention of the NAS Panel. Indeed, I specifically confronted two presenters to the NAS PAnel, D’Arrigo and Hegerl, about the unavailability of data and asked the NAS panel to get it. The NAS panel did nothing. Instead of reporting on these issues, North cited a 1997 study saying the following:
A considerable portion of tree ring data collected on all inhabited continents is freely available online (Grissino-Mayer and Fritts 1997)
While its true that much tree ring data is online, the trouble is obviously that this isn’t the case for some key series used by the Team. In his covering email to me, North said that he verbally took the position that data archiving procedures need to be tightened up:
My suggestion (not really on the slides) is that when an agency awards a grant like some many of these we have talked about, there should be a negotiation as to what is to be saved and in what form. There needs to be some consideration for the costs, etc. But the bottom line is that these things need to be agreed upon before the money is awarded.
In his covering email, he also said that he’d said that the paleoclimate community was “shocked” to find themselves thrust into the limelight and “totally unprepared” for it. That seems only partly true – they seem quite prepared for awards from Scientific American and quite prepared for adulation. Indeed, not only were they prepared for it, they went so far as to issue press releases for many scientific studies – you don’t issue press releases if you aren’t seeking the limelight. What they seem not to have been “prepared” for is someone saying: if you communicate with the public, then you have responsibilities of disclosure and due diligence that exceed those applicable to discussion in seminar rooms.