IPCC Review Editors have an extremely important function under IPCC procedures. In prior discussion of the Replies by WG1 Chapter Authors to Review Comments, we noted their unresponsiveness on issues that we were familiar with e.g. the deletion of the inconvenient post-1960 Briffa reconstruction results, the handling of the HS dispute. When the IPCC WG1 (grudgingly) placed the WG1 Review Comments and Replies online- url here they did not place the Review Editor comments online, despite the importance of review editors. Through the diligent efforts of David Holland, the IPCC WG1 and WG2 Review Editor comments have now been obtained and are now online for the first time here – at this point, another Climate Audit exclusive.
When you examine these review comments, as I urge you to do, please remember that this is supposed to be the most carefully reviewed document in human history, where entire stadiums of scientists have carefully weighed each word. Compare that impression to the actual review editor comments, which as you will see do not rise above a form letter for 64 of 69 Review Editor comments discussed here.
Duties of Review Editors
First some comments on the obligations of Review Editors set out by the IPCC here. The workload for a Review Editor is said to be “heavy”. They are supposed to ensure that all substantive comments receive “appropriate consideration” and that “genuine controversies are reflected adequately” in the Report.
Review Editors will assist the Working Group/Task Force Bureaux in identifying reviewers for the expert review process, ensure that all substantive expert and government review comments are afforded appropriate consideration, advise lead authors on how to handle contentious/controversial issues and ensure genuine controversies are reflected adequately in the text of the Report.
There will be one or two Review Editors per chapter (including their executive summaries) and per technical summary. In order to carry out these tasks, Review Editors will need to have a broad understanding of the wider scientific and technical issues being addressed. The workload will be particularly heavy during the final stages of the Report preparation. This includes attending those meetings where writing teams are considering the results of the two review rounds. Review Editors are not actively engaged in drafting Reports and cannot serve as reviewers of those chapters of which they are Authors. Review Editors can be members of a Working Group/Task Force Bureau or outside experts agreed by the Working Group/Task Force Bureau.
If there are particular points of controversy or areas of major differences – and readers of CA can probably think of a couple -, the Coordinating Lead Authors “in consultation with the Review Editors” are encouraged to organize a “wider meeting with principal Contributing Authors and expert reviewers” – something that obviously did not take place in the topic that was the most controversial in chapter 6.
Coordinating Lead Authors, in consultation with the Review Editors and in coordination with the respective Working Group/Task Force Bureau Co-Chairs and the IPCC Secretariat, are encouraged to supplement the draft revision process by organising a wider meeting with principal Contributing Authors and expert reviewers, if time and funding permit, in order to pay special attention to particular points of assessment or areas of major differences.
If “significant differences of opinion on scientific issues remain”, Review Editors are obliged to “ensure” that such differences are described in an annex to the Report.
Although responsibility for the final text remains with the Lead Authors, Review Editors will need to ensure that where significant differences of opinion on scientific issues remain, such differences are described in an annex to the Report.
Review Editors are required to submit a “written report”, though, as seen below, this proves to be nothing other than signing a form letter.
Review Editors must submit a written report to the Working Group Sessions or the Panel and where appropriate, will be requested to attend Sessions of the Working Group and of the IPCC to communicate their findings from the review process and to assist in finalising the Summary for Policymakers, Overview Chapters of Methodology Reports and Synthesis Reports.
WG2 Review Editor Reports
David has obtained 43 WG2 Review Editor reports, collated into one package here. All but 5 reports consisted of the following form letter:
The review process for the development of the Chapter in the Working Group II Fourth Assessment, as laid out in the Principles Governing IPCC Work, has been properly followed.
My reading of the Final Government Draft of the Chapter confirms the satisfactory completion of this process.
My reading of the Final Government Draft of the Chapter confirms that it properly reflects scientific controversies.
The form letter stated that they would welcome other comments:
We welcome any other comments on the report which may be appended, or written below. In particular, if you are unable to answer in the affirmative to any of the statements above, please provide further explanation.
Of the handful of 5 Review Editors that actually assessed whether the Authors had responded to the Review Comments, the observations of John Zillman (page 59-63) on WG2 chapter 19 (edited by Stephen Schneider and 2 others) is a striking departure from the standard signature to a form letter. Zillman observes of Schneider and others:
(b) Significant points not fully taken on board. In several places (eg E-19-9 (p 18), E-19-91, 92 (p 44)) where the balance of comments suggests an approach different from the Lead Authors’ preferred approach and they have chosen to stick with their own preference, responses such as ‘Unfortunately we cannot please everyone . . . ‘ do not really do justice to the points being made and give an arguably misleading justification of the basis on which they have rejected the Reviewer’s proposals.
(c) Misleading implications that comments have been taken on board. There are numerous places (eg G-19-86 (p 21), G-19-107 (p 22), G-19-111 (p 27), G-19-149 (p 34), G-19-166-169 (p 37)) where, by using words like ‘yes’, ‘OK’, ‘Text Revised’, ‘Text revised to deal with many of these issues’ etc, the Lead Authors imply that they have actually taken the Reviewers’ points on board whereas, in fact, they have either ignored them, deleted any relevant text where they might have been incorporated, or
actually taken a different position in their revised text.
(d) Uncompleted responses. There are several places where, in the ‘Notes of the Writing Team’ column, the Lead Authors have not gone back and tied off their final response. …
(e) Erroneous statements. There are some (but not many) places where it is stated that something will be done, or has been done, and it clearly hasn’t, …
4 I should also draw attention to one particular point on which it is arguable that the Lead Authors have tried to give the impression of having responded to Review Comments but have subtly avoided doing so to the extent implied. In lines 18-19 of the Introdution (p 5 of the Final Draft) they leave ambiguous whether ‘other judgements’ are ‘value judgements’ or ‘scientific judgements’ (‘other judgements informed by the state of scientific knowledge’); and then, while asserting (Notes of the Writing Team in response to G-19-22 (p 7)) that ‘the key vulnerabilities in this chapter are intended to provide information useful to policy makers in making their DAI judgements, not to provide those judgements’, the Lead Authors appear to build in quite a lot of value judgements and equate DAI with ‘key vulnerabilities’ in the later text (eg p 29, line1, p 30, line 35).
It would be interesting for someone to assess chapter 19 in the light of Review Editor Zillman’s comments. I might add that Zillman’s comments seem completely descriptive of WG1 chapter 6 as well, where there also, for example, are many cases of “misleading implications that comments have been taken on board” when they haven’t.
WG1 Review Editor Comments
David Holland has obtained 26 Review Editor responses, all submitted between mid-November 2006 and Jan 7, 2007. Again all but one Review Editor response is a form letter, although the form letter differs slightly from the WG2 form letter and reads only as follows:
I can confirm that all substantive expert and government review comments have been afforded appropriate consideration by the writing team in accordance with IPCC procedures.
Notice that the WG1 form letter does not even require the authors to confirm (as in WG2) that the document “properly reflects scientific controversies.”
The only Review Editor who did not submit the above form letter was John Mitchell in respect of Paleoclimate chapter 6 who stated:
I can confirm that the authors have in my view dealt with reviewers comments to the extent that can be reasonably expected.
There will inevitably remain some disagreement on how they have dealt with the reconstructions of the last 1000 years, and there is further work to do here in the future, but in my judgement, the authors have made a reasonable assessment of the evidence they have to hand. The other possible area of contention (within the author team) is on some aspects of sea level rise – this has gone some way towards reconciliation but I sense not everyone is entirely happy
With these caveats I am happy to sign off the chapter, to thank the lead author team for their cooperation, and congratulate them on the chapter.
Notice that he did not certify that Briffa and the chapter 6 authors had complied with IPCC procedures. Although IPCC procedures state clearly that “where significant differences of opinion on scientific issues remain, such differences are described in an annex to the Report”, Mitchell specifically noted the existence of such a controversy in respect to the 1000 year reconstructions, but failed to meet his obligation to ensure that there was an “annex”.
Aside from the particularities of the HS dispute, the overwhelming impression though is that of the Review Editors simply rubber stamping the process. Other than a few editors of WG2 (and at this point we don’t know whether even their comments had any effect), the recorded comments of Review Editors merely show rubber stamping.
It seemed almost impossible that these form letters constituted the entire corpus of Review Editor contributions. David Holland has followed up with John Mitchell, double checking whether there were some more detailed comments that perhaps had not been provided to Holland by IPCC. On Jan 31, 2008, he wrote to Mitchell asking (After some pleasantries):
Can you confirm that the attached is the complete report or let me have a copy of any supplemental information?
On February 20, Mitchell replied as follows:
I can confirm that you have had the complete Review Editors report and that there was no supplemental information submitted with the Review Editors report. I hope this answers your enquiry.
On Feb 22, Holland sent Mitchell a letter concerning chapter 6, asking inter alia:
Can you tell me if you sent a copy of your Review Editors’ Report, or any other report on the IPCC AR4 process, to the Met Office, DEFRA or any other UK Government agency?
On March 27, Mitchell responded, stating that he had not “kept any working papers” as he was not “required to do so”:
You should note that the review editors do not determine the final content of the chapters. It is the authors that are responsible for the content of their chapters and responding to comments, not the review editors. All of the comments and all of the authors’ responses have been made available, and are the proper source for anyone wishing to understand what comments were made and how the authors dealt with them. It would be inappropriate to provide more information beyond the web pages already freely provided.
For my own part, I have not kept any working papers. There is no requirement to do so, given the extensive documentation already available from IPCC. The crux of the review editors’ work is carried out at the lead authors meetings going through the chapters comment by comment with the lead authors.
When Mitchell says that there is no “requirement” to keep working papers, this seems at odds with the explicit IPCC statement:
All written expert, and government review comments will be made available to reviewers on request during the review process and will be retained in an open archive in a location determined by the IPCC Secretariat on completion of the Report for a period of at least five years.
IPCC Principles Appendix A state that Review Editors should be drawn from the ranks of “independent experts based on the lists provided by governments and participating organisations” – so even though Mitchell is a Review Editor, he is also an “expert” and “all” of his comments are supposed to be retained for a period of “at least five years”. Why would Mitchell discard his papers on the topic? And BTW I’d be surprised if all trace of his comments and correspondence have disappeared from the Met Office as well (which can be FOIed).