A Slight Change to NAS Panel Terms of Reference

Readers may recall the consternation of the NAS Panel when von Storch (and ourselves) started presenting answers to some of Boehlert’s questions. I received notice today from NAS that:

You might also notice that there have a been a few minor changes to the Committee’s Statement of Task.

I posted up the original terms of reference here. The slightly revised terms of reference are here. The changes are not huge, but it’s always interesting to check. (If they were completely "minor", why make the changes at all?) . I’ve italicised changes.

Project Scope
The committee will describe and assess the state of scientific efforts to reconstruct surface temperature records for the Earth over approximately the past [1,000-(deleted)]2,000 years. The committee will summarize current scientific information on the temperature record for the past two millennia, describe the main areas of uncertainty and how significant they are, describe the principal methodologies used and any problems with these approaches, and explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change. As part of this effort, the committee will address tasks such as:

[[Identify the variables for which proxy records have been employed (e.g., temperatures averaged over specific geographical areas and time periods, extreme temperatures).-deleted]]

- Describe the proxy records that have been used to estimate surface temperatures for the pre-instrumental period (e.g., tree rings, sediment cores, isotopes in water and ice, biological indicators, indicators from coral formations, geological boreholes, historical accounts) and evaluate their limitations.

- Discuss how proxy data can be used to reconstruct surface temperature over different geographical regions and time periods. [new]

- Assess the various methods employed to combine multiple proxy data to develop large-scale surface temperature reconstructions, the major assumptions associated with each approach, and the uncertainties associated with these methodologies.

- Comment [Evaluate -deleted] on the overall accuracy and precision of such reconstructions, relevant data quality and access issues, and future research challenges. new

Note: statement of task revised on 03/30/2006.

The National Academies will be sponsoring this study. The approximate starting date for the project is 01/19/2006. The committee will issue a Final Report in approximately four months

For me, the most interest change is the insertion of the "relevant data quality and access issues". I wonder how they”’ deal with that. We barely touched the surface of this topic in our presentation.


24 Comments

  1. Brooks Hurd
    Posted Mar 31, 2006 at 6:21 PM | Permalink

    Steve,

    “relevant data quality and access issues”. I wonder how they”‘ deal with that. We barely touched the surface of this topic in our presentation.

    True, but von Storch did make it abundantly clear that data access is an important issue. I believe that the NAS panel will now need to deal with this issue in some way.

  2. jae
    Posted Mar 31, 2006 at 8:20 PM | Permalink

    Wonder why these changes were made. I doubt NAS came up with these changes without some kind of pressure.

  3. Armand MacMurray
    Posted Mar 31, 2006 at 9:00 PM | Permalink

    I find it interesting that rather than “evaluate” the “… the overall accuracy and precision of such reconstructions,” they will now only “comment on” those reconstructions. Seems like a possible out from being pinned down on how good certain past temp reconstructions really are…

  4. John Lish
    Posted Apr 1, 2006 at 1:00 AM | Permalink

    Comment [Evaluate -deleted] on the overall accuracy and precision of such reconstructions, relevant data quality and access issues, and future research challenges. new

    I agree Armand that this is the most interesting change however, lets look at this in context with the relevant Boehlert request:

    2) (b) What are the principal scientific criticisms of their [Mann, Bradley and Hughes] work and how significant are they? (c) Has the information needed to replicate their work been available? (d) Have other scientists been able to replicate their work?

    It would appear that Ciceroni’s ToF have been brought more in line with Boehlert’s questions as raised by M&M and Von Storch. Also, remember back to Goldston’s comments at the end of the Thursday session and the fact that the panel (chaired by North) were unaware of the precise request by Boehlert. It makes me wonder whether more pressure has occurred from the panelists themselves on Ciceroni? After all, I would be livid to have been misled as they were.

    I also disagree that replacing ‘evaluate’ with ‘comment’ is necessarily about avoiding being pinned down. Given that no-one apart from Mann had any confidence in predicting temperature within 0.5C, I would think that it would be hard to evaluate confidently the methodology in this context other than in harsh tones. ‘Comment’ allows a widening of the context and will, I think, lead to a reduction of significance for this methodology as a temperature reconstruction (probably to where the likes of Rob Wilson are positioned). I also suspect that the more wacky statistical reconstructions and behaviour will come in for criticism. Politically, this makes sense as you don’t want to throw away the baby with the bathwater. As I’ve said before, that 0.2C comment will come back to bite Mann in the backside. If I was a panelist with sympathies towards the AGW argument, to try and incorporate Mann’s position would conflict with a sense of personal professionalism (regardless of external pressures).

    So I’m imagining that the report will say something along the lines of the basic science is sound but that its significance as a determinator of temperature has been overplayed (thinking back to Schrag’s testimony). Also in the report something like: There needs to be greater rigour as regards statistical analysis and a more open culture as regards data-sharing to ensure that dendro-studies confidently provides its contribution to the understanding of the climate/environment. I wouldn’t have a problem with such a report, nor would I suspect would Boehlert.

  5. fFreddy
    Posted Apr 1, 2006 at 3:18 AM | Permalink

    For me, the most interest change is the insertion of the “relevant data quality and access issues”. I wonder how they deal with that. We barely touched the surface of this topic in our presentation.

    Given that they have changed the terms of reference, doesn’t this open the door for you to make a supplemental submission to the Panel ?
    This change will reduce Barton’s ability to twist Boehrlert’s arm if the Panel does do a whitewash. It might be a good idea to list up some of the problems you have had in getting access to data, and get them on the record.

    (Suggesting yet more work for you, again … sorry, Steve … )

  6. Jeremy
    Posted Apr 1, 2006 at 12:01 PM | Permalink

    They changed it from evaluate to comment on. Sounds like they want to avoid confrontation and give the ol’ rubber stamp.

  7. per
    Posted Apr 1, 2006 at 1:03 PM | Permalink

    I think it is axiomatic for most scientific studies that “data quality and access issues” are a given; a sine qua non; an essential of science. The fact that they are asked to look at this is salient.

    The next question is how they do so. It is one thing to look at the state of play today; which data sets are available, what methodology is available. That kind of ignores the fact that it took a request from a Senate committee to force MBH to release their computer code, so that one could see (most of) what they did in a paper in 1998 ! The history here- what was available when and to whom, is really kind of key, and it would be most disconcerting to see that they don’t take proper evidence here. A series of statements from one’s friends that you gave the data to them, is not the same as having your data publically available for inspection…

    This one must surely have legs, for how can they answer this question ?
    yours
    per

  8. Steve McIntyre
    Posted Apr 1, 2006 at 2:01 PM | Permalink

    #7. It will be interesting indeed. I didn’t get the impression that it was a topic that the panel was very interested in. Their natural inclination, I’m sure, was more towards “big picture” issues. Answering the data access issues for a range of paleoclimate studies will require quite a bit of slogging through swamps. I suspect that a bibliography of my CA posts on the topic would be about as useful; as anything else.

    They also sent an email asking for names of reviewers for their report, with the reviewers to be suggested no later than April 4, to be available commencing mid-April. It will be hard to make a substantive response on the added terms of reference (changed on March 30) for a report whose review is scheduled for mid-April. I’d have a hard time writing it up by then and I know the details inside-out.

  9. John Lish
    Posted Apr 1, 2006 at 2:32 PM | Permalink

    #8

    I didn’t get the impression that it was a topic that the panel was very interested in. Their natural inclination, I’m sure, was more towards “big picture” issues.

    Not entirely convinced by this statement – after all the very ToF provided by Ciceroni directed them towards this, over and above their selection for the panel.

    #7 Per, I don’t think that we can necessarily assume that “The fact that they are asked to look at this” is just pressure from Boehlert’s committee. I suspect that the dynamics are more complicated and interdependant. But it is salient although I can envisage this not being the major issue that you assume.

  10. John A
    Posted Apr 1, 2006 at 3:07 PM | Permalink

    Um, how often is it that the terms of reference for a scientific investigation change AFTER you’ve done the hearings and heard the evidence (or at least, that which you can winkle out of people in only 45 minutes each)?

    The major difference between Boehlert’s questions and the NAS Panel remains: the specific questions about Mann’s method, statistical control and quality of result.

    If I were Congressman Boehlert, I would feel extremely let down by the conduct of the NAS Panel and I’d feel strongly that I should have done the investigation myself.

  11. per
    Posted Apr 1, 2006 at 3:25 PM | Permalink

    Re: #9

    #7 Per, I don’t think that we can necessarily assume that “The fact that they are asked to look at this” is just pressure from Boehlert’s committee. I suspect that the dynamics are more complicated and interdependant. But it is salient…

    I stayed away from how this became part of the terms of reference; but I am intrigued about the consequences now that it is part of what the committee must address.

    … although I can envisage this not being the major issue that you assume.

    I think it has been reported that the chairman appeared to be quite relaxed about data being hidden until publication. However, I find it very difficult to imagine how a committee can accept that peer-reviewed, published science should be performed on the basis of (1) hidden data and/ or (2) hidden methods; especially when public policy may depend upon the results of these studies. I take this issue to be black and white. If the committee examines the status quo, they may be surprised. If they examine the history, I can only imagine they will be taking legal advice to avoid saying anything too defamatory.

    Steve, did you provide a list of outstanding data / methodology issues, where it is not publically known what the facts of particular papers is ?

    cheers
    per

  12. Doug L
    Posted Apr 1, 2006 at 6:12 PM | Permalink

    I think they have no sensible choice but report that further investigation is needed.

    They added a second statistician at the last minute because they knew they were unprepared. When they got to the meeting they learned of the Boehlert questions, so again they were unprepared. Now the terms of reference are mostly reduced suggesting that they still feel unprepared.

    They were asked to evaluate, it appears they don’t feel in a position to do that.

    The pattern now seems obvious.

    I did send a comment to them way back when, it went something like this:

    You need at least three independent statisticians who would control the report, and so what if it takes longer than the scheduled time.

    My crystal ball says: they punt.

  13. John Lish
    Posted Apr 2, 2006 at 4:39 AM | Permalink

    It is very messy but I find that quite cheering. For one thing, it shows that Ciceroni has cocked things up rather nicely. Thats the thing with public hearings, they often don’t go the way vested interests (or indeed Sir Humphreys) would want them to. And because this is a politico-academic space that the NAS panel are operating in, they are subject to a number of conflicting pressures. I wouldn’t want to be them.

  14. Brooks Hurd
    Posted Apr 2, 2006 at 9:58 AM | Permalink

    Re: 12, Doug,

    It did seem that most were unprepared to answer Boehlert’s questions. However, von Storch, like Steve, was prepared. Once Pandora’s box was cracked open, Ciceroni was placed in the position of dealing with Boehlert’s questions.

    I think that Per’s question in 11 is excellent:
    Steve, did you provide a list of outstanding data / methodology issues, where it is not publically known what the facts of particular papers is ?

    This change begs for such a supplemental submission.

  15. Steve McIntyre
    Posted Apr 2, 2006 at 10:01 AM | Permalink

    I included some infor along those lines in Appendix B of my handout. However, that was very summary.

  16. Doug L
    Posted Apr 2, 2006 at 2:19 PM | Permalink

    Further reflection on the changes has brought on the following thoughts.

    From “Global Warming Probe Prepares to Punt”

    http://www.geocities.com/poncedeleon_1/ClimateChange/CCProbePreparesToPunt.html

    “……. In spite of adding an additional statistician, it appears they are unprepared to evaluate, they are now only being asked to “comment” on the accuracy and precision of the research.

    The original terms of reference do include some of what the Boehlert questions ask, but not the issue of data quality and access. Could it be that data is of insufficient quality to do the evaluation that was asked for?………..”

    In other words the committee gets itself out a difficult spot by noting something like lack of access to unused data makes the evaluation a pointless exercise. This kind of thinking can be buried in bureaucratic language.

    No need then for the committee to suggest further investigation, that could be left up to Rep Boehlert’s replacement, or perhaps someone else.

  17. BradH
    Posted Apr 3, 2006 at 5:01 AM | Permalink

    Discuss how proxy data can be used to reconstruct surface temperature over different geographical regions and time periods. [new]

    This seems to assume that proxy data can be used to reconstruct surface temperature.

    This, to my mind, is yet to be proven, therefore it seems an odd assumption to make.

  18. Doug L
    Posted Apr 3, 2006 at 5:25 AM | Permalink

    At least the following is still in there:

    “explain how central the debate over the paleoclimate temperature record is to the state of scientific knowledge on global climate change”

    It will be interesting to see how much they are willing to say about that. Surely they can’t just say that it’s not central, and not explain why they continue to believe that CO2 is the chief culprit in climate change.

    At the very least they ought to say what it’s based on, what the strengths and weaknesses are in those lines of evidence (or at least a picture of the uncertainties).

    Perhaps they will have to modify their terms of reference again! ;-)

    Nah! Notice carefully, the word “CO2″ appears nowhere in the terms of reference.

    Another thing that does not appear are the names of three scientists referred to repeatedly by the Boehlert questions. When top scientists complained about the Barton questions they implied that the scientific establishment could keep its own house in order, yet they seem unwilling to name names.

    Will be interesting to see if the report shrinks from naming the names of people or gasses.

  19. per
    Posted Apr 5, 2006 at 7:40 PM | Permalink

    re: #18

    Surely they can’t just say that it’s not central, and not explain why they continue to believe that CO2 is the chief culprit in climate change.

    I am confused by this comment. The NAS panel doesn’t have to justify the IPCC’s stance, or anyone else’s. It just has to say what the scientific weight of evidence on a point is, and it can come to a different view from another NAS panel !

    In short, they have no obligation to justify that CO2 is the chief culprit…
    yours
    per

  20. Doug L
    Posted Apr 6, 2006 at 4:04 PM | Permalink

    Re #19

    The basic thrust of my post was that it appears that the report is going to try to be as uncontroversial as possible.

    If my comment was ignorant, I apologize for any confusion.

    CO2 is believed to be a major aspect of climate change, and in the mind of many, the reconstructions bolstered that theory. If the NAS panel feels the theory is unaffected or weakened, it’s hard to see how that is reported without some basic explanation of the theory.

    I should add that, the theory apparently is not just that CO2 traps heat, but that when the various forcings without CO2 are added, that a negative sum results. I believe that this sort of information is generally buried in the fine print rather than being emphasized.

    I thought the point of the NAS was to clear up things for the policy makers, not continue to bury things in fine print. If I have the science wrong, it is because the information is not being properly presented. Or I suppose I just have a screw loose. :-)

    In case anyone doesn’t know, the NAS says:

    “The National Academies perform an unparalleled public service by bringing together committees of experts in all areas of scientific and technological endeavor. These experts serve pro bono to address critical national issues and give advice to the federal government and the public. “

  21. Steve McIntyre
    Posted Apr 6, 2006 at 4:41 PM | Permalink

    In fairness to the NAS Panel, Goldston of the House Science Committee said that there are many huge questions and each huge question will be the subject of great debate; but they were hoping for clarification on some specific questions. I think that he was disconcerted at the panel broadening its scope so that the specific questions got lost. I agree with him – sometimes you need to find your footing on the smaller questions.

    I’m sure that NAS would be more than happy to wade into the bigger quetions, but they’ll probably wait until after IPCC 4th Assessment Report and it will be a different panel. I have no problem with that.

  22. John A
    Posted Apr 6, 2006 at 4:55 PM | Permalink

    In case anyone doesn’t know, the NAS says:

    “The National Academies perform an unparalleled public service by bringing together committees of experts in all areas of scientific and technological endeavor. These experts serve pro bono to address critical national issues and give advice to the federal government and the public. ”

    What WordPress desperately needs is irony tags.

  23. per
    Posted Apr 6, 2006 at 5:24 PM | Permalink

    Re: #20
    IPCC TAR summary says (inter alia):

    Reconstructions of climate data for the past 1,000 years (Figure 1b) also indicate that this warming was unusual and is unlikely7 to be entirely natural in origin.

    I think you are correct; the reconstructions are supportive, especially MBH, which suggests very little variability until the increased CO2 came along. There is some difficult in knowing how well the GCMs model climate, because you cannot perturb the system and test your GCMs. The period that we know is very short.

    But formally, there is a distinction between how good the GCMs are, and how reliable the historical reconstructions are. You don’t need a historical reconstruction to have a GCM which is 99.99% accurate. I will say that it is a bit of a broad topic to opine on; I can’t really see how you could do anything more than arm-wave on this, especially with this panel’s already broad remit.

    cheers
    per

  24. Doug L
    Posted Apr 6, 2006 at 7:38 PM | Permalink

    Re #23

    I can’t open the link to verify, but I believe Von Storch’s presentation indicated the reconstruction problems had no bearing on detection.

    One might call that hand waving and expect it to get passed on into the report. Slightly more elaborate hand waving in the report would be preferable.

    Perhaps they just ignore the elephant at the cocktail party instead?! :-).

2 Trackbacks

  1. By Cicerone Then and Now « Climate Audit on Feb 4, 2010 at 5:53 PM

    [...] this controversy, there was a slight change in the terms of reference of the NAS panel (see CA post here), which resulted in the North panel making a few references to data archiving, but they did not [...]

  2. [...] deal with them. Several weeks later, the terms of reference were modified (contemporary CA report here) as follows: – Comment [Evaluate -deleted] on the overall accuracy and precision of such [...]

Follow

Get every new post delivered to your Inbox.

Join 3,381 other followers

%d bloggers like this: