Nature has two online articles pertaining to the NAS Panel – one about the NAS Panel primarily relying on the press conference and another mentioning a prospective new NAS investigation into the data access problems left untouched by the past panel. The Nature reporter asked an interesting question at the press conference. I’ve provided a partial transcript of this exchange and compared it to the article.
The Nature reporter (at about 25 minutes) used the O-word in asking:
I was wondering if the …. whether the community had oversold the original research or if there were any lessons generally that the community has learned from this experience.
North answered (these are reasonable but probably not 100% transcriptions):
The community probably took the results to be more definitive than Mann and colleagues originally intended. That has happened. Perhaps They took it much more seriously than the original paper. That was done not just in the press, but even in the scientific community.
The essential part of the answer is the Mann et al wok was really the first of its kind… I don’t think that the scientific community really felt that it was oversold… Science works over time as a community process…
The overselling or the perception of overselling came very much not from the science community but from the interaction of part of the science community with the broader public discourse and in particular with the way the way. This reconstruction was used by the IPCC in the 2001 report. The IPCC 2001 report was very careful to give the original Mann et al reconstruction., which was only 2 years old, a level of confidence that was fairly low. They characterized it as likely which, in IPCC terminology, means 2:1 odds, which is well short of standard that we expect from quantitative argument.
However, the IPCC also used the reconstruction as a visual prominently in the report, including in the SPM and that sends a very confusing message, a misleading message about how resolved this part of the scientific research on this part of climate change was.
North then added:
Lessons… The community has to understand that science is a process and no individual paper really tells the whole story. It’s not a bad idea to temper statements when they’re new, usually several year vetting. Science really advances by trial and error. It’s very dangerous to pull one paper of the literature fresh before it’s had time to season.
In the Nature article, they said:
In its report, released on 22 June, the NAS committee more-or-less endorses the work behind the graph. But it criticizes the way that the plot was used to publicize climate-change concerns. And it leaves open big questions about whether researchers should be obliged to make their data available (see Plotting a course). …
Panel members were less sanguine, however, about whether the original work should have loomed so large in the executive summary of the IPCC’s 2001 report. "The IPCC used it as a visual prominently in the report," says Kurt Cuffey, a panel member and geographer at the University of California, Berkeley. "I think that sent a very misleading message about how resolved this part of the scientific research was."
"No individual paper tells the whole story," agrees North. "It’s very dangerous to pull one fresh paper out from the literature."
In the Nature article on a new NAS investigation on data access – which is the first that I’ve heard of this – they said:
The US National Academy of Sciences (NAS) report may help put the ‘hockey-stick’ debate to rest, but it leaves open a larger question about who should have access to researchers’ data. …
The NAS panel largely demurred on questions of data sharing, saying that although openness is generally good, such issues are "discipline dependent". "We thought that question was a little big for what we were trying to do here," says committee chair Gerald North.
That doesn’t mean the matter is closed. Bill Kearney, the academy’s director of media relations, says that the NAS is putting together a panel to examine issues of data sharing in all fields. The panel, which will convene later this summer, "will be a broad look at access across the board", says Kearney.
For now, politicians will probably intervene as they see fit, says David Goldston, chief of staff for the House Committee on Science: "As the scientific community continues to wrestle with this, it will continue to be an issue."
I’ve pointed out the tension between NAS and the House Committees on the specific question of data. These issues bothered the House Committees – and, in my opinion, correctly so, as there are applicable federal policies on data that are routinely flouted by the climate science community, with the National Science Foundation making no discernible effort to ensure or even monitor compliance by grantees with federal policy.
NAS established terms of reference for the panel, which excluded specific House questions on data access. This obviously bothered Goldston who raised the problem on presentation day, but the panel had little interest in the topic.
A month later, the terms were modified so there must have been some behind-the-scenes negotiations between NAS and the House Science Committee which has been unreported. Even though the House Science Committee presumably asked for the terms to be changed for a purpose, the NAS Panel basically snubbed its nose at the House Science Committee on data issues. They carried out no investigation or due diligence on the topic. (Of course, had they had a replication expert on the panel, as certain people suggested, and complied with NAS policies on panel composition, this might not have been such a problem.) The panel’s comments (as I predicted) were trivial – one paragraph of generalities. They said nothing that could not have been said 15 years ago or that wasn’t said better 15 years ago.
It looks like even NAS is a bit embarrassed about this. Of course, now, instead of answering a specific question, they are proposing a panel to study everything under the sun. In my opinion, there are some pretty simple issues with data and methods access for paleoclimate that are separable from biotechnology. One of the wise aspects of common law traditions is case law. Sometimes you can decide particular problems and from a tradition of solutions to particular problems, you can build more general principles. The paleoclimate case is easy. Why try to develop a Napoloeonic Code? Sounds like too much Sir Humphrey to me.
Reading between the lines of the quote from Goldston, it sounds like they’re not very happy with NAS, that they put their own credibility on the line in their dispute with the Energy and Commerce Committee relying on NAS’ representations that they could resolve the questions actually asked by the committees. Instead, NAS changed the terms of reference and the panel dodged key questions. Goldston: "For now, politicians will probably intervene as they see fit. As the scientific community continues to wrestle with this, it will continue to be an issue." Sounds to me like he feels that he was sandbagged by Sir Humphrey. He was sandbagged.