It is very difficult for the general public to understand that IPCC does not carry out independent due diligence. The answers of Michael Mann in 2003 to questions from Sen Inhofe are well worth re-reading in the present context. Mann says that it is “against the mission” of IPCC to “carry out independent programs” or to “carry out studies”. Instead, they have “lively exchanges”.
Perhaps that’s how Gerry North got his understanding of how expert panels, such as the NAS panel, should carry out business. You may recall North’s statement that the NAS Panel didnt do any research, that they just took a look at papers, that they got 12 people around the table and just kind of winged it, saying that’s what you do in these sort of expert panels.
Here is an excerpt of Mann questions and answers (but all are worth reading).
30. Did IPCC carry out any independent programs to verify the calculations that you made in MBH98 or MBH99? If so, please provide copies of the reports resulting from such studies.
It is distinctly against the mission of the IPCC to “carry out independent programs”, so the premise of the question is false. However, the IPCCs author team did engage in a lively interchanges about the quality and overall consistency of all of the papers as the chapter was drafted and revised in the course of review.
31. Did IPCC carry out any independent quality control on the data that you used in MBH98 and MBH99? If so, please provide copies of the reports resulting from such studies.
The IPCC doesn’t “carry out studies”, so the premise of the question is false. The IPCC instead depends that the normal scientific peer-review process, especially when done in a leading journal, has ensured an acceptable level of quality. In addition, the IPCC does check to see if any criticisms have been raised postreview in comments and response to the journal articles.
32. Did IPCC carry out any studies to validate the statistical procedures and methodologies used in MBH98 and MBH99? If so, please provide copies of the reports resulting from such studies.
The IPCC doesn’t “carry out studies”, so the premise of the question is false. Instead, as indicated above, the IPCC relies on earlier stages of review to cover such matters.
34. Has any organization other than IPCC conducted independent quality control on the data that you used in MBH98 and MBH99? If so, please provide copies of the reports resulting from such studies.
The IPCC doesn’t “carry out studies”, so the premise of the question is false. The data used by MBH98 (and MBH99) were produced by other researchers, not Mann and colleagues. It is thus not clear what kind of “independent quality control” is being referred to here. However, it is fair to say that each of these papers has been subject to rigorous peer review in a leading scientific journal, which is considered by scientists to be an independent quality control process. We are aware of no criticisms of the datasets in the peerreviewed scientific literature.
35. Has any organization other than IPCC carried out any studies to validate the statistical procedures and methodologies used in MBH98 and MBH99? If so, please provide copies of the reports resulting from such studies.
The IPCC doesn’t “carry out studies”, so the premise of the question is false. If the question were asked: Have other independent groups tested the methodology of Mann et al (1998) in a publication in the peerreviewed climate literature, the answer would be “yes”. I would refer the questioner to the following paper: Zorita, E., F. Gonzalez-Rouco, and S. Legutke, Testing the Mann et al. (1998) Approach to Paleoclimate Reconstructions in the Context in a 1000-Yr Control Simulation with the ECHO-G Coupled Climate Model, Journal of Climate, 16, 1378-1390, 2003. The paper arrives at the conclusion that the methodology of MBH98 performs well with networks of data comparable to those used by MBH98.
37. Did the peer reviewers for Nature in MBH98 carry out any independent quality control or validation studies? If so, please provide copies of such reports.
Neither I, nor authors of peer-reviewed journal articles in general, are made privy to the detailed analyses that peer reviewers may or may not have performed in the process of reviewing a manuscript. Authors only receive the comments that were selected to be made available to them by the reviewer and editor. This question is thus impossible to answer. Numerous other groups (see the dozen or so independent estimates of various groups shown in Figure 1 of: Mann, M.E., Ammann, C.M., Bradley, R.S., Briffa, K.R., Crowley, T.J., Hughes, M.K., Jones, P.D., Oppenheimer, M., Osborn, T.J., Overpeck, J. T., Rutherford, S., Trenberth, K.E., Wigley, T.M.L., On Past Temperatures and Anomalous Late 20th Century Warmth, Eos, 84, 256-258, 2003) have produced reconstructions that are remarkably similar to those of MBH98 based on a variety of data and methods. See my answer to question #4.
38. Did the peer reviewers for Geophysical Research Letters in MBH99 carry out any independent quality control or validation studies? If so, please provide copies of such reports.
See response to question #37.
A couple of other things that I noticed in my re-read. At the time MM2003 was just in the news. In question 11, he refers to “a manuscript detailing the numerous fundamental errors made in the Energy and Environment paper has been submitted to the peer-reviewed literature”. Does anyone remember seeing that article? You don’t, because it was rejected – and you must admit, it’s hard to imagine circumstances under which Stephen Schneider would reject a Mann defense, but it happened. (And yes, I know much of the story.)
In questions 8 and 9, Mann refers to:
A more recent paper (in press) extends proxy-based hemispheric temperature reconstructions through the mid 1990s, demonstrating the ability of the reconstruction to capture the accelerated warming evident in the instrumental record since 1980.
This is presumably the calculation in Jones and Mann 2004, about which I’ll comment some time.
Climate Audit 
50 Comments
11
Any news?
25
He’s funny guy 😉
Some additional questions comes to my mind,
Did you use sparse reconstruction when computing verification statistics and in MBH99 Fig 2 ?
Is it possible to obtain that astronomical cooling trend without CO2 adjustment for PC1? ( 2007 Follow-up; Why Rutherford et al 2005 recon starts at AD1400? )
So what I guess I take home from this is that the IPCC doesn’t carry out studies. I’m glad he cleared that up.
SteveM
Have you noticed that when a “bad” paper makes it through peer review
Mann and Gavin say that peer review is necessary but not sufficient,
but that here Mann seems to imply that peer reveie insures quality?
Here is another consequence of the fact that IPCC doesn’t do independent due diligence and doesn’t conduct studies.
Section 9 of chapter 2, WG1 report, 4thAR, gives a table (see here, page 203: http://www.ipcc.ch/pdf/assessment-report/ar4/wg1/ar4-wg1-chapter2.pdf) that we’ve all seen before. It shows the identified anthropogenic radiative forcings and assigns a level of scientific understanding to each. Note that the level of understanding is much better on the positive forcings than on the negative forcings.
Why is this so? Is it because the positive forcings are simpler and easier to evaluate — while the negative forcings are complicated and difficult to evaluate? Or is it because the overwhelming majority of the research being conducted is aimed at validating the positive forcings? And if the former is the case, shouldn’t more resources be focused on the more difficult forcings? Wouldn’t an independent, objective effort to find the truth focus at least equal resources on both sets of forcings, until both are well understood?
The answers are somewhat mind-boggling if not mind-blogging. If one asks whether the IPCC has verified something, I don’t think that such a question makes any assumption – perhaps only the assumption that the IPCC could verify these things. When a Yes/No question is being asked, we usually assume that both answers might a priori be possible. That’s why we ask.
In a normal world, the possibility that the IPCC verifies something would not be such an unusual assumption: if the world organization hypothetically collects hundreds of experts who dedicate some of their time, it could be a good idea to use them to verify certain key things more properly than what is possible in a conventional peer-review process that only involves one, two, or three largely unmotivated busy random professionals who may have other idiosyncratic reasons not to do their work in a perfect way.
Even though we don’t live in an ideal world, it would still be plausible for Mann to answer “No, Sir, the IPCC has unfortunately not done such a thing” instead of “It is distinctly against the mission of the IPCC” to do the things that a sane person would assume the IPCC should do according to its definition. For example,
http://en.wikipedia.org/wiki/Intergovernmental_Panel_on_Climate_Change
defines the IPCC as a scientific body tasked to evaluate the risk of climate change. Is it such a weird assumption to think that such a massive body could also evaluate papers and refine some reviews? Needless to say, it is revising and suppressing reviews in the case of papers from the other side, using a certain obvious polarization, anyway.
I think that most people here know what is the actual “mission” of the IPCC as understood by Mann and many other men and the “lively interchanges” probably reflect this “mission”. Many of them are probably interchanges answering how to achieve their “mission”. But it would be interesting anyway to hear Mann’s answer to the question “What is the mission of the IPCC?”.
Steve remarked
Well, don’t keep us in suspense!
Reading this sort of thing, it’s amazing Mann still has his reputation. And his job.
The worm will turn, however slowly.
Until then, PT
—
“The book is worth reading, in part because it is enjoyable to read of
other people’s folly, not to mention their avarice and stupidity.”
— Roger Lowenstein, reviewing “Devil Take the Hindmost: a History
of Financial Speculation”, WSJ 6-1-99
(Steve M):
One might contrast that with the NASA panel on solar cycle prediction, where the panel members are people that actually did the underlying [and possibly most relevant] research, and where research is continuing during the evaluation period.
FWIW, here’s the IPCC “mandate” (which I assume is what Mann is calling the “mission statement):
It does appear that all they’re chartered to do is compile literature and then write a summary (not necessarily in that order).
Lots of platitudes. Not much meat.
‘Lively Interchanges’, ///// I’m glad I didn’t have to make ICs work via that methodology. As a matter of fact, I had to ignore the perspectives of the experts to find the fix where they refused to look. The documentation to get from R&D to manufacturing methods which lead to a certain outcome makes the IPCC ‘documentation’ a joke.
#8. Leif, this is a large topic. IPCC authors are experts, but all have horses in the race and frequently review their own material with all the potential for bias that that creates. In business and engineering, independent experts are used all the time; they may be less knowledgeable about the details than the specialist, but they are independent. Personally, I think that a completely arms-length review of the climate models is long overdue – it would be a big budget enterprise by say aerospace engineers and such; people like Gavin Schmidt would be required to cooperate by the NSF; but it would be a report by non-fellow travelers. Who knows – maybe they’d say that the situation is worse than we think and we need to do more.
10,
It seems like resistance to independent review indicates a lack of confidence.
10 (Steve M): Yes, I know this is a large topic, but so is solar physics. What we do at the Panel is having face-to-face meetings combined with tele-conferences and email. Five of the 12 members are stakeholders [have horses in the race], the other seven are critics. The stakeholders are also critical [at least of other people’s work]. The critics grill the stakeholders about how they obtain their data, demand to see the source code, suggest sensitivity studies that must be performed by the stakeholders before the next meeting, demand the theory to be fully exposed, including assumptions, adjustable parameters, and the like. Then at regular times, open voting is done. People are asked what new evidence would cause them to reverse their vote. Everything is done above board and transparently. Now, I recognize that with 2000 ‘experts’ that is not feasible, but may I suggest that quantity does not guarantee a better result. Maybe the IPCC [or each subsection] should have 20 experts instead. The idea of a completely independent body sounds nice, but will run up against the reluctance of the stakeholders to share data, perform further tests, show source code, etc. In the NASA panel, the stakeholders are directly involved and it is hard for them to duck and hide.
#12, Leif
Indeed. It also helps not to have $$$billions riding on the outcome, PI’s with an agenda, slanted journos writing alarmist BS, opportunistic politicos winning Nobels….
Still, it sounds like you folks have your house in better order. In the long run, truth will out — science is self-correcting, that’s the beauty of the scientific method. The real question is, when?
Cheers — Pete Tillman
Re #8, Larry, Jan 5th, 11.19am:
Thank you for providing the IPCC mandate:
You say:
Actually, if they took their mandate seriously, there is at least some meat. Here it is again, with some key words bolded:
I submit that these bolded phrases have clear meaning, and impose on IPCC a requirement that they act in a sound and professional manner. That they have demonstrably failed to do so is an indictment.
Example of “lively interchange”:
Mann: I think the climate sensitivity is 2 degrees.
Hansen: I think it’s 3.
Mann: It’s 4.
Hansen: 5.
Mann: Do I hear 6?
Hansen: 7, and that’s my final offer.
Mann: Ok, Jim, now how much is the sea level going to rise?
Hansen: usufruct…
Ah, the word parsing war, a RC specialty. “Carry out studies”. IPCC does not carry out studies in the sense of doing primary resarch. It merely carries out studies in the sense of synthesizing primary research results into a coherent statement. Gotcha – that’s not “study”. And since the premise is “wrong” I am under no abligation to address the spirit of the question.
The attitude is disgustingly anti-accountability. Decidely unhelpful for the policymaker trying to understand the factual basis underlying a policy direction. A young aspiring policy wonk would be disgusted.
Back to their mandate:
Apparently human-induced climate change is a foregone conclusion.
Does this seem adequate for an organiosation that is demanding a world-wide economic recession, if not suicide?
#17 Dan
It shouldn’t be. Rescan your quote:
Risk is ‘probability’ multiplied by ‘impact’. Visit society of risk analysis website http://www.sra.org/ for formal definitions:
Risk: The potential for realization of unwanted, adverse consequences to human life, health, property, or the environment; estimation of risk is usually based on the expected value of the conditional probability of the event occurring times the consequence of the event given that it has occurred.
i.e. You have to address both components, probability of AGW and impact of AGW.
Is it possible IPCC misunderstood its mandate, and that they only did half the job?
Very important question.
16,
Yes. I’m waiting for her condemnation.
#21 Want to read a magazine while you wait?
http://www.theonion.com/content/node/27509
dr bender will examine her presently.
One other observation:
Any lawyer will tell you that the word “should” is a weaselword, contracts always use the word shall, and never should. Should is optional. Shall isn’t. So the escape hatch is built in.
Re #23: Larry, 5th Jan, 3:14pm:
Well observed Larry. You are of course correct.
Re 12,
Leif, it is becoming clear the stakeholders are the people of the world. Wrong decisions now on climate issues could cost trillions of dollars or major percentages of GDP’s of countries. Steve has talked about Engineering reports. I think the world deserves such a report and to allow public comment like the FCC and other organizations require on issues affecting the public before reports are released.
25 (Bob B): I’m not disagreeing with you, just pointing out even an Engineering Report can benefit from the engineers participating. We saw with the NASA panel that open dialog at all stages was beneficial, and the ability to grill and demand data and sources from people doing the reserach was central. Now, one can accomplish that in other ways, perhaps, e.g. by excluding papers from consideration when the authors are not forthcoming or responsive to the demands of the Panel.
Leif Svalgaard says:
The panel already excludes papers that don’t meet their standards (pro-AGW). If they excluded papers from authors that “are not forthcoming or responsive to the demands of the Panel”, then the majority of the current Team wouldn’t be included. They’d be able to send the next IPCC report out on a postcard…
But it would be a waste of postage.
======================
I have trouble with two consecutive sentences in the IPCC mandate:
I would really like to find out how any group can assess scientific and technical literature on a comprehensive basis without monitoring related data and parameters.
#8 Lubos, I don’t think the panel asked those questions because they didn’t know the answers to them. My guess is they wanted the responses on the record in the context of their own independent review (the Wegmann Report). The choice they gave Mann was to answer directly with a No! or obfuscate, be combative, whatever, which is what he chose to do.
This set the stage for the only independent survey of a small part of the AGW theory and found that part flawed (i.e. Wegmann). So people not in climate science are justified in being skeptical of any part not independently reviewed.
Re: #19
IMO, that implies an assumption or a conclusion that human-induced climate change is risky; i.e., it has a potential for realizing unwanted, adverse consequences to human life, health, property, or the environment . But does it? I agree with Lomborg, that warmer is better than colder. Additionally, that only humans can induce climate change or is the major contributor to climate change is unproven.
#31 pjaco, I don’t see any discussion at all of the quality of the IPCC process in the mainstream science or press. So far they have accepted everything the IPCC says uncritically as far as I can see.
I’m not sure the mainstream press have ever properly reported what the IPCC have to say. If people actually take the time to read the IPCC reports rather than learning about it second hand from people with an agenda they might find that the lower end of the range predictions are very much in “what’s the big deal?” sort of territory and that there are plenty of caveats and mentions of possible uncertainties. Because the problem is so complex it maybe shouldn’t be surprising that the upper end of the range predictions which would be highly problematic for mankind can’t be discounted at this stage. Talking about changes in the 1.5 to 6.0 C range or whatever is a scientific way of saying “we can’t really say for sure yet but this could turn out to be a very major problem”. The environmentalists tend to focus on the latter portion of that message and ditch all the caveats to try to scare people into following their wider back to nature sort of agenda while journalists tend to go for the most stunning and attention grabbing headline they can dream up so what the general public eventually winds up hearing can be a bit distorted to say the least.
We can all dance around the truth, but anyone that doesn’t see that the IPCC is doing exactly what it wants to do, and what it really was set up to do, is blind to what is going on. They are not misunderstanding anything. Politicians don’t care that it is not objective, nor do most of the scientists that participate*. The pols get exactly what they want: fuel through fear for increased support. The scientists get exactly what they want: more money for more research for more alarmism for… That the process by which the IPCC conducts its business is so openly corrupted is merely a slap in the face of those that have the wherewithal to actually understand it. Most average joes, which includes the majority of journalists (they’re on board, too), simply don’t care, or don’t have the ability to understand why things could be in any way considered wrong. They want A to be true, they need A to be true, and even a sign on the IPCC door that says “we really don’t care about the science since we already know A is true, and if it isn’t, what we propose to do is the right thing anyway.” (Think of the first episode of Psych when Gus and Shawn discuss the name of their company… such a sign would be too big to fit in the window).
Mark
*Sure, there are a few that truly would like the IPCC to be a scientific entity with true objectivity, but they are in an extreme minority.
Here is an interesting response from Mann:
First the question:
Mann’s response:
So here we have Mann validating his climate reconstructions by noting that they agree with the models. Go over to the IPCC reports and you’ll read that the models are valid because they agree with past climate reconstructions.
#1 cont. ,
In 25
In univariate calibration this can be seen in term
of (approximate) variance of the estimator
In multivariate calibration, the equations are just bit more complicated ( see http://www.climateaudit.org/?p=2445 ), but the same dependence of magnitude of the true value is present. That’s what scale factor errors do. Mann cannot accept this equation, too much uncertainty, see http://www.climateaudit.org/?p=2445#comment-173404 . (Even IPCC does not accept this equation, see explanation for fig 6.10.c )
Thus, he throws in some non-sense about spectrum of calibration residuals, unknown method of MBH99, and cross validation with a reference that he chooses. Nice.
Michael,
Good point. Each were leaning on a weak reed.
One can almost imagine Jim Hanson (or his representative Gavin Schmidt) talking to Michael Mann about how these two could support each other. I wonder if they realize it is a house of cards or if they have actually fooled themselves?
Michael Smith said: “So here we have Mann validating his climate reconstructions by noting that they agree with the models. Go over to the IPCC reports and youll read that the models are valid because they agree with past climate reconstructions.”
Gotta love the mutual reinforcement there. There obviously is a PhD program in Tautology out there somewhere producing these types of scientists.
I find the following IPCC items to be the most telling:
“adaptation and mitigation” is engineering in that it is the physical application of science. How can one claim to be objective when climate change is deemed to be human induced, and that something must done about those changes and then list possibilities for what can be done based on those assumptions.
Does young policy wonk see a link between under-estimated uncertainty in (1) the paleocliamte time-series and (2) the GCM output? Or is it all just coincidence? A PhD in ClimaTautology might get you trained up the right way.
Getting back to the opening post – these, errr, “lively interchanges” – are they documented somewhere? It would be interesting to see what could be so provocative about a science that is so clearly “settled”. Maybe it’s just the inherently lively nature of the average climatologist that leads to such interchanges?
Climate-science-style messing around uncertainties gets even more funny, when you read Hegerl et al [1]. Where _only_ scaling errors were considered. Tapio Schneider criticized this [2], and Hegerl’s response [3] was that
and added priceless
And indeed, Nature’s webpage tells that
New data seems to be corrected, but Fig.1 in the original paper remains uncorrected. These uncertainty issues are not that important for climate scientists (or for Nature, a journal with huge impact factor). Supplementary still says that
Which explains why, in the original version, the Upper Confidence limit is sometimes Lower Confidence limit .
[1] G. C. Hegerl, T. J. Crowley, W. T. Hyde & D. J. Frame Nature 440, 10291032 (2006)
[2] T. Schneider Nature 446, doi:10.1038/nature05707 (2007) ; Received 19 May 2006; accepted 13 December 2006.
[3] Hegerl et al. reply; Replying to: T. Schneider Nature 446, doi:10.1038/nature05707 (2007)
RE: #4 – To add to that, imagine doing something like Fault Tree Analysis / FMEA / Hazard Analysis on the climate system. What would be the severity factors / downsides of the negative forcings in a worst case scenario? Which is harder to adapt to and causes more death, +5 deg C or -5 deg C?
RE: #34 – Planning for a +2.5 deg C future is a really great government program. It’s actually not all that high of a risk, even if nothing is done. It wins votes from the ecologically oriented voters. It increases energy independence (assuming that actual action is taken). Great way to score politically (and, more cynically, make a few extra bucks on the side). Now, consider the “what if we’re completely wrong future. Huge risk, so if you say you’re doing something about it and fail, cold hungry mobs will come after you. The things you can practically do to mitigate it are mostly environmentally “ugly” and may do things like increase cancer and respiratory risks. Not really lots of money to be made when sunk costs are factored in. Most countermeasures would be exceedingly politically unpopular and multiple wars may be needed. Well, that explains it.
Prediction; this entire subject, for whatever reasons, should become a non-issue soon. It is very likely that will be within a reasonable amount of time.
Not only is there a lack of due diligence on the science but the IPCC does not seem too fussy about demonstrating that they operate as laid down in their Governing Principles, mentioned by Dan in #17, which call for a comprehensive, objective, open and transparent basis etc. Clause 10 says Differing views on matters of a scientific, technical or socio -economic nature shall, as appropriate in the context, be represented in the scientific, technical or socio-economic document concerned
That Steve and Ross thought Wahl and Ammann was inappropriate, seeing as how the IPCC never considered the accepted version, is to be expected. However when the reviewer for the Government of the United States points out that it does really does not cut the mustard ( see reviewers comment ID #: 2023-415) one might expect some mention in the text. The umpire on Chapter 6 was a Brit, the Chief Scientist of the of our Met Office, sent, I presume, at the taxpayers expense to act in accordance with paragraph 5 of Annex 1 to Appendix A to the Principles Governing IPCC Work which say the Review Editors function is to assist the Working Group/Task Force Bureaux in identifying reviewers for the expert review process, ensure that all substantive expert and government review comments are afforded appropriate consideration, advise lead authors on how to handle contentious/controversial issues and ensure genuine controversies are reflected adequately in the text of the Report.
Clearly the Review Editors advice would be interesting to read and paragraph 5 goes on to say that Although responsibility for the final text remains with the Lead Authors, Review Editors will need to ensure that where significant differences of opinion on scientific issues remain, such differences are described in an annex to the Report. Review Editors must submit a written report to the Working Group Sessions or the Panel etc.
From the reviewers comments it is clear that many areas are contentious in several of the chapters but I have seen no annexes. Rejected hardly seems appropriate consideration. Clearly the Review Editors written reports are something we would all like to see. In reply to my request to the British Chapter 6 Reviewer Editor I was directed to TSU who were said to be in the process of disbanding. Perhaps this is the Littauer hand off.
I shall, of course, be pressing to get see the review editors reports through FoI in the UK (did no one get sent a copy in the UK?) but in these matters some of you are in more congenial jurisdictions and I hope that someone might press to see these reports which must have been circulated to their governments or the IPCC is even less in anyones control. Either the Review Editors’ reports say nothing because the umpires are biased which means the checks and balances are worthless or they were ignored. Either way the system does not work as specified.
#40 bender
Congrats on a very good witticism
The problem as I see it
The problem as I see it is that most people recieved their information not from the IPCC report but from the SPM (summary for policymakers) The SPM was writen primarily by governement appointees who directed THEIR views, not neccessarily those of the participating scientists.
These political appointees claim that a consensus of view was achieved by “thousands” of scientists, but as Arthur Roarch points out in his paper “Pseudo Scientific Elements in Climate Change Research” (in press)”McLean and Holland calculated that some 800 researchers were involved in ALL the reports” and that”the real number of expert reviewers who univocally sustain the conclusions of the most important chapter in the recent assessment (WGI, 9 ‘attribution’) is limited to five”
Great information! I have been hearing more about this subject matter and thank you for bringing this information to our attention.
2 Trackbacks
[…] Review Editor Comments David Holland has written in raising an excellent point about the failure of IPCC WG1 to release the Review Editor […]
[…] A clear statement of this occurs in Mann’s 2003 answers to questions from Inhofe (noted up at CA here). There were a series of questions, starting with: 30. Did IPCC carry out any independent programs […]