Nature has two online articles pertaining to the NAS Panel – one about the NAS Panel primarily relying on the press conference and another mentioning a prospective new NAS investigation into the data access problems left untouched by the past panel. The Nature reporter asked an interesting question at the press conference. I’ve provided a partial transcript of this exchange and compared it to the article.
The Nature reporter (at about 25 minutes) used the O-word in asking:
I was wondering if the …. whether the community had oversold the original research or if there were any lessons generally that the community has learned from this experience.
North answered (these are reasonable but probably not 100% transcriptions):
The community probably took the results to be more definitive than Mann and colleagues originally intended. That has happened. Perhaps They took it much more seriously than the original paper. That was done not just in the press, but even in the scientific community.
Cuffey added:
The essential part of the answer is the Mann et al wok was really the first of its kind… I don’t think that the scientific community really felt that it was oversold… Science works over time as a community process…
The overselling or the perception of overselling came very much not from the science community but from the interaction of part of the science community with the broader public discourse and in particular with the way the way. This reconstruction was used by the IPCC in the 2001 report. The IPCC 2001 report was very careful to give the original Mann et al reconstruction., which was only 2 years old, a level of confidence that was fairly low. They characterized it as likely which, in IPCC terminology, means 2:1 odds, which is well short of standard that we expect from quantitative argument.
However, the IPCC also used the reconstruction as a visual prominently in the report, including in the SPM and that sends a very confusing message, a misleading message about how resolved this part of the scientific research on this part of climate change was.
North then added:
Lessons… The community has to understand that science is a process and no individual paper really tells the whole story. It’s not a bad idea to temper statements when they’re new, usually several year vetting. Science really advances by trial and error. It’s very dangerous to pull one paper of the literature fresh before it’s had time to season.
In the Nature article, they said:
In its report, released on 22 June, the NAS committee more-or-less endorses the work behind the graph. But it criticizes the way that the plot was used to publicize climate-change concerns. And it leaves open big questions about whether researchers should be obliged to make their data available (see Plotting a course). …
Panel members were less sanguine, however, about whether the original work should have loomed so large in the executive summary of the IPCC’s 2001 report. "The IPCC used it as a visual prominently in the report," says Kurt Cuffey, a panel member and geographer at the University of California, Berkeley. "I think that sent a very misleading message about how resolved this part of the scientific research was."
"No individual paper tells the whole story," agrees North. "It’s very dangerous to pull one fresh paper out from the literature."
In the Nature article on a new NAS investigation on data access – which is the first that I’ve heard of this – they said:
The US National Academy of Sciences (NAS) report may help put the ‘hockey-stick’ debate to rest, but it leaves open a larger question about who should have access to researchers’ data. …
The NAS panel largely demurred on questions of data sharing, saying that although openness is generally good, such issues are "discipline dependent". "We thought that question was a little big for what we were trying to do here," says committee chair Gerald North.
That doesn’t mean the matter is closed. Bill Kearney, the academy’s director of media relations, says that the NAS is putting together a panel to examine issues of data sharing in all fields. The panel, which will convene later this summer, "will be a broad look at access across the board", says Kearney.
For now, politicians will probably intervene as they see fit, says David Goldston, chief of staff for the House Committee on Science: "As the scientific community continues to wrestle with this, it will continue to be an issue."
I’ve pointed out the tension between NAS and the House Committees on the specific question of data. These issues bothered the House Committees – and, in my opinion, correctly so, as there are applicable federal policies on data that are routinely flouted by the climate science community, with the National Science Foundation making no discernible effort to ensure or even monitor compliance by grantees with federal policy.
NAS established terms of reference for the panel, which excluded specific House questions on data access. This obviously bothered Goldston who raised the problem on presentation day, but the panel had little interest in the topic.
A month later, the terms were modified so there must have been some behind-the-scenes negotiations between NAS and the House Science Committee which has been unreported. Even though the House Science Committee presumably asked for the terms to be changed for a purpose, the NAS Panel basically snubbed its nose at the House Science Committee on data issues. They carried out no investigation or due diligence on the topic. (Of course, had they had a replication expert on the panel, as certain people suggested, and complied with NAS policies on panel composition, this might not have been such a problem.) The panel’s comments (as I predicted) were trivial – one paragraph of generalities. They said nothing that could not have been said 15 years ago or that wasn’t said better 15 years ago.
It looks like even NAS is a bit embarrassed about this. Of course, now, instead of answering a specific question, they are proposing a panel to study everything under the sun. In my opinion, there are some pretty simple issues with data and methods access for paleoclimate that are separable from biotechnology. One of the wise aspects of common law traditions is case law. Sometimes you can decide particular problems and from a tradition of solutions to particular problems, you can build more general principles. The paleoclimate case is easy. Why try to develop a Napoloeonic Code? Sounds like too much Sir Humphrey to me.
Reading between the lines of the quote from Goldston, it sounds like they’re not very happy with NAS, that they put their own credibility on the line in their dispute with the Energy and Commerce Committee relying on NAS’ representations that they could resolve the questions actually asked by the committees. Instead, NAS changed the terms of reference and the panel dodged key questions. Goldston: "For now, politicians will probably intervene as they see fit. As the scientific community continues to wrestle with this, it will continue to be an issue." Sounds to me like he feels that he was sandbagged by Sir Humphrey. He was sandbagged.
Climate Audit 
35 Comments
Steve,
The article in Nature was as expected. Ignore the facts. Stay the course.
The editorial was more interesting touching as it did on Mann’s withholding of information. In my wide readings in the history and philosophy of science (Lately mass extinctions, Burgess Shale, and mammal-like reptiles.) I can’t recall a scientist who withheld information or data, if it supported his conclusions.
In this, Mann, truly is breaking new ground.
JSP
Well, Steve, it sounds like you have another trip to Washington in your future. At least to me, it would be a great injustice not to have you testify before this other NAS panel as well.
Of course, them emphasis would not be on who had or hadn’t provided data, but rather the benefit to both society and the science community of having as much data available as possible both to provide accountability and to ease replication and thereby enhance reliability.
Is it just me who thinks that the NAS Panel is deliberately widening the scope of the investigation in order to avoid making specific findings against certain people and certain controversial reports?
If the NAS Panel wants to look at data sharing and related issues, then these are ethical issues and there should be ethicists on the Panel.
re #3: John, not just ethical issues, but legal issues too if the terms of grants are being breached!
I thought that North’s “big question” comment was in reference to Boehlert’s list of questions, and not just the data sharing issue. From my quickly notes from the webcast question and answer session on 6/22/06, I noted the following. I think that this was the first question asked.:
Previously posted here.
If my notes are correct (they are paraphrased) and I interpretted the comment correctly, I think that it was ironic for North to say this. It sounds like he is saying that the original narrowly defined questions were too big, so they expanded the scope of the report. John_A, it sounded a little odd to me too.
It seems to me that the whole investigation went soft once Barton ceeded control to Boehlert.
I would like to see Rep. Barton get re-involved. Barton should demand to know why key questions were not answered or obfuscated.
It looks like Barton’s efforts are not quite finished on this subject. The last paragraph of a 6/23/06 article in the Boston Globe states:
The NAS panel has overshadowed this work. I seem to remember reading that Barton had appointed one professor to head the report. I don’t remember remember his name, but I seem to remember being impressed with his credentials. I haven’t been able to find much about this. Does anyone else here any any info?
I was talking with a colleague last week about the NAS report, and the suspicion hanging over how Mann crafted his 1998 hockey stick.
My friend recalled that, while he was an impressionable chemistry grad student in 2002, a professor at his university gave a supportive, detailed talk on Mann’s 1998 work . My friend remembers that, after the talk, several of his chemistry professors expressed surprise and some doubt about Mann’s methodology. But, their doubts were expressed quietly.
My friend, who enjoys debate, felt that global warming was one of those off-limits academic topics that serious graduate students dare not challenge in any way.
This is simply his recollection (he’d fall into the GW believer, not skeptic, camp, by the way). I have no way of knowing the content of the presentation or the quiet reservations, or whether this was just an isolated event. But, it does make me wonder if there are those in the academic would who recognize some of the problems with methodology and ethics (my word) in the climate science field, but speak quietly due to political correctness considerations.
this whole science panel stuff of the NAS smells like big time political panels. Why? Because they didn’t answer many of the hot questions, but rather summarized something already known and now they want to open a “new” panel and discussion for exactly some of the hotter questions (which again will turn out to be a paraphrasing of already known things, without much progress thus giving way for a “newer” panel)….
A strategy of small steps, or as I call it, politics…
I like the idea of a “Mann et al wok…” Suitable for cooking up some spurious results?
Re #7, Jason Lewis
Strongly agree that I want to hear more about what these folk are doing. One of Barton’s staffers wrote to New Scientist a month or two ago about this.
Steve M, has Barton’s team had any contact with you ?
So Mann is being painted as a victim, whose statements have been taken out-of-context by the press and his colleagues, and whose hockey stick graph was unintentionally over-emphasized by the IPCC in 2001 (with him being one of the authors, of course).
It seems to me that every time the media misconstrues a publication about global warming, or something in the press release that doesn’t fully agree with the actual body of work, etc, there’s someone who comes-out to try and set the record straight. There was an NAS report years back where the press release was misleading, and Richard Lindzen went ballistic about it. When Chris Landsea thought Kevin Trenberth was promoting his views at a conference as that of the IPCC instead of his own personal views, Landsea resigned from the IPCC in protest. So where was Mann when all of these people were going overly ga-ga with his hockey stick? When did he make an effort clear the air?
Of course, the folks at RC say nothing has changed from 1998 – they say that Mann’s “likely” is the equivalent of the IPCC’s 2:1 and the same as the NAS’s “plausible.”
This really makes me mad. It is simply not true; they did NOT “more-or-less endorse” Mann’s work. They actually agreed with everything MM have been saying! The NAS folks are nothing but a bunch of waffling politically correct clowns.
Here is more on the ongoing Barton effort. A Feb 10, 2006 WSJ article states:
Info on Edward Wegman can be found here and here. His office is at the “Center for Computational Statistics” at George Mason University. He appears to have background in hardcore statistics and computational analysis. This could be interesting. I’m a little suprised that M&M haven’t been contacted.
I’m not. As it says, “Mr. Barton has already sought a separate analysis.” Seems like going to a 3rd party and letting them do as much work as they can independently would be reasonable. Now if they hit snags and have trouble replicating undescribed steps in MBH98, then I expect they’ll contact MBH and M&M.
And they noticeably leave out the fact that none of this would be an issue if they had originally listed their confidence as only “plausible” in the first place. The papers would never have been published. I suppose, then, Mann wouldn’t be the… never mind.
Mark
re. 16:
Just as a reminder, MBH 98 only goes back to 1400. Here is the abstract from MBH 1999. Emphasis added.
Abstract.
Several recent studies have sought to estimate seasonal or annual Northern Hemi-
sphere NH mean temperatures during past centuries based on available networks
of proxy and historical data. Because of interest in the possibility that tempera-
tures were globally warmer in Medievaltimes, preceeding any plausible anthropogenic
climate influences, we investigate here the possibility of millennial-scale reconstruc-
tions of hemispheric temperature based on proxy data. We focus on the statistical
properties of the underlying data and the reconstructions themselves, and assess the
reliability of the reconstructions and uncertainties therein, back in time. Taken at
face value, the 20th century appears to be the warmest century this millennium, the
1990s the warmest decade, and several recent individual years the warmest on record.
****However, the expanded uncertainties in early centuries preclude, as yet, any definitive
conclusions prior to about AD 1400.****
Re:17, Mann and Jones (2003):
ABSTRACT:
We present reconstructions of Northern and Southern Hemisphere mean surface temperature over the past two millennia based on high-resolution ‘proxy’ temperature data which retain millennial-scale variability. These reconstructions indicate that late 20th century warmth is unprecedented for at least roughly the past two millennia for the Northern Hemisphere. Conclusions for the Southern Hemisphere and global mean temperature are limited by the sparseness of available proxy data in the Southern Hemisphere at present.
And it’s all based on the limited proxy coverage shown here.
(lee, I guess I should have said “since 2003” in #12…tough to keep publication dates straight, even for RC and NAS)
re #17: I don’t know what you are quoting, but my MBH99 abstract says (emphasis added):
re 20 – grabbed the wrong paper- that was MBH 98 in GRL.
Your 1999 abstract includes the first half of the sentence you emphasized, and the sentence prior to that, as well.
Now, if you want to blast them for a lot of other things, go for it – you may have good reason – but dont blast them for not including caveats that are right there.
re #21: Hah! I’m not native English speaker but “at moderately high levels of confidence” does sound much more than just “plausible”. Now we are on to it, let’s check also their (MBH99) Conclusions:
So as far as I can understand, they are saying 1990s was likely (caveats included) the warmest decade in the millenium, but more confident conclusions needed more data. Now re-read Mark’s #16 you were responding in #17.
Of course, making the claim of anomalous now, without anything prior to the LIA is not only disingenuous, it is outright fabrication. IMO, even if it IS warmer now than the MWP, proving the concept of “anomalous” is difficult if not outright impossible.
Mark
So you’re hitting them for saying, basically, it’s likely we’re right, but here are some ways we could be wrong and more data is needed, as overstating their case?
Bottom line, this parsing of phraseology in attempts to find yet another reason to attack Mann et al, is fundamentally boring and useless – at least.
I came to CA becaue I knew there were issues with the dendro data, and I wanted to understand what the issues were, whether I needed reo reassess my beliefs about AGW because of that, and whether it was ineesting and important enough to decicd eto devote the time and effort to learnign the field and mathematics well enough to evaluate it myself. The tentative conclusion I’ve come to, given discussions here and more importantly the NAS report, is that there is interesting data there, that interpretation and analysis is difficult and in at least some (but not necessarily all) cases has not been done well, and that at best it is supporting evidence for other lines of evidence. I even understand some of the reasons people are unhappy with Mann on the issues.
But the reasoning behind these attempts to manufacture reasons (through parsing of phraseology in this case) to vilify him for trivia is beyond me, and it frankly discredits those who engage in it.
Lee: It ain’t the trivia; it’s the deliberate witholding of data and methodology, and his refusal to make ANY concessions that has “vilified” him. I gladly note that you are changing your mind on some of these issues.
Then drop the damn trivia.
re #24: No, I’m hitting them (mainly Mann because I’m not sure about the level of involnment of B/H) because I’m confident that MBH98/99 was no “accident”: it was not a result of several “honest but bad” methodolical and data selection choices, it was crafted in purpose.
Anyhow, your abstract quote in #17 comes from the SUBMISSION version of MBH99 (there is no such a thing as MBH98 in GRL), which seems to be anymore only available (according to Google) from Google Scholar cache. I find it curious that out of the 17 (according to Google Scholar) copies of MBH99 available, you were able to locate the only one with the outdated abstract. Anyhow, thank you for leading me to this copy (pdf can be recovered through Wayback Machine from here), I did not know existed. It will be interesting to compare that to the final article. I already find this original conclusion (compare #22) interesting:
“almost certainly” Hmmmm….
Steve, check the preprint (#27). There are rather interesting textual changes, and moreover, two figures were removed from the final version 🙂
Lee, relax, your “anomaly” fears are completely unfounded. We’re still well below the Holocene thermal max ~10000 yrs ago. Most of the long-term proxies show the LIA was the coldest period since then.
Jean S, in Goosse et al 2003, [Update- Gerber et al 2003, available at Mann’s website) they mention that they adjust confidence intervals by calculating the "effective degrees of freedom" because of autocorrelated residuals and use that to adjust N. Presumably they use AR1. That might be the missing link in the MBH99 confidence interval calculations.
I’ll check the preprint. It’s also interesting to look at the contemproary press release which I’ll link up to. It was pretty lurid – a worthy example for climateprediction.net.
Re: 14
That is interesting; Wegman sounds like a real statistical heavyweight. I’m surprised that Steve hasn’t heard from him as yet. Does anyone have any further info on when (or if) Wegman’s findings will be released, and in what venue?
Regards;
“The article in Nature was as expected. Ignore the facts. Stay the course.”
My impression was different. This was a retreat just about as far as Nature could go given its prior stands.
Which reminds me of a comment I made a while back to the effect of that the NAS report was bad, very bad. They used the pretense of investigating the dendros to open up on everything to push anthro. AGW. Each of the names on the report will get carbon credit lobby/Brit. Petroleum “study grants” in the future. Just a wild “guess.”
But as I said before maybe the report isn’t so bad for mankind after all. Because it was so, so bad and so public no scientist not directly on the BP dole could endorse it. Whatever you think of the dendro studies at least more than one was analyzed. How others like glacier studies? Just one was used. For the glacier studies, like the NAS’ selective others, all showed AGW, NAS used these to confirm each other and used descriptive writing that sounded like professional lobbyists wrote – or perhaps the lobbyists’ language has worked itself into scientific writing styles on the topic.
I’m interested in the glacier studies because they’re the most comical. Retreating glaciers? Heat melts ice therefore AGW. Advancing glaciers like in Greenland? AGW melts ice at rock edge faster which sinks between rock/ice interface lubricating and pushing glaciers fasters. Or something like that. I would really like to see the NAS resolve conflicts like that! Might have to defer the writing of the study 100% to the professional spinmeisters supplied by the lobbies.
re #30: Steve, I was unable to locate the reference (Goosse et al 2003) please give the full details.
My bad, I meant Gerber et al 2003 here
re #34: No problem. Eduardo gave the reference to Gerber et al article here and as you can see, it has much smaller “uncertainties” than MBH98/99. This is rather strange when you are talking about the same reconstruction… I’m still interested if someone could provide the data used to create the figure 1(B) in the paper.