Although the IPCC calendar webpage doesn’t link to session documents of the 34th session (Kampala Nov 2011), David Holland has alertly located the documents – see here.
IPCC rejected a proposal for anonymous peer review – see document here (page 12 on).
We haven’t discussed this topic previously (in an IPCC context). On reflection, the adoption of a form of anonymous peer review by IPCC seems to me to be a very good idea and might somewhat mitigate some problems. There is no doubt in my mind that review responses are strongly conditioned by who is making the suggestion. The experiences of Ross and myself are vivid examples.
In the IPCC’s Special report on Carbon Dioxide Capture and Storage of WG III of the Fourth Assessment Report, reviews were anonymous. The review document described the results of anonymous review as being entirely positive:
The TSU prepared a list of Review comments with numbers. The TSU had a conversion table linking the numbers to the Reviewer’s names. During the treatment at the lead author meetings and the formulation of the authors’ responses the authors only saw the numbers. The authors knew that in case they would need to consult an Expert Reviewer for getting some clarifications about his comments, the anonymity could be lifted and the coordinates of the Expert Reviewer would have been made available to the authors. In practice it turned out they did not need to use this provision. The anonymity was continued until finalization of the final draft report.
The Reviewers and authors have been informed beforehand about this procedure. The number of comments was normal compared to other special reports. No Reviewer used improper or inappropriate language. The WG III co chairs and TSU held an enquiry among the authors and Review Editors. They considered the anonymity an improvement, because it made them concentrate fully on the content of the matter, disregarding the persons and their background, which was more time efficient.
The review document summarized the advantages of anonymous review as follows:
• Authors will concentrate on the content of the matter, excluding (subconscious) biases.
• There is positive experience in WG III AR4 – also the Task Force on the Greenhouse Gas Inventory Program (TFI) has practiced anonymous reviews with a positive judgment of the authors and Review Editors.
• Authors cannot be criticized anymore of ignoring comments of specific individuals or representatives of scientific schools or interest groups, as happened in the past.
• It remains possible for authors to contact Expert Reviewers if there is a need for clarification.
These seem pretty convincing arguments. Here are the arguments against:
• The Task Group on Procedures was installed in order to consider the recommendations of the InterAcademy Council (IAC). The IAC did not recommend anonymous Reviews, so there is no compelling reason to address this.
• There is a risk that Exper Reviewers could take advantage of their anonymity by burdening authors with unprofessional or inappropriate comments.
• Measures against biases are already sufficiently taken by having Review Editors and by having a authors working as a group.
• Transparency is crucial to the IPCC process. There would be an imbalance in transparency when authors are known by name and Reviewers are not.
• IPCC needs a consistent approach with regard to its Expert Reviews. Changing the approach to require anonymous Review comments would imply that there is a problem with the named Reviewer approach, which is not the case.
• Named Expert Review is more efficient as it allows writing teams to liaise with Reviewers when there is a need for clarification.
In my opinion, none of these reasons stands up.
The Task Group on Procedures was installed in order to consider the recommendations of the InterAcademy Council (IAC). The IAC did not recommend anonymous Reviews, so there is no compelling reason to address this.
The first argument starkly shows the hypocrisy and opportunism of IPCC. IAC did not recommend (or even consider) Jones-Stocker enhanced confidentiality. This was not not mentioned in the briefing documents for the Jones-Stocker amendment. Worse, they represented the changes as addressing issues raised by the IAC.
There is a risk that Exper Reviewers could take advantage of their anonymity by burdening authors with unprofessional or inappropriate comments.
The IPCC’s own experience with the Carbon Dioxide Task Group was reported to be the opposite. To the extent that accountability was an issue, the reviewer names could be removed from the version given to authors for comment, but restored in the final publication of review comments, thereby ensuring accountability.
Measures against biases are already sufficiently taken by having Review Editors and by having authors working as a group.
Neither of these measures had the slightest deterrent to AR4 author responses. In practice, authors seem to have divided up responsibilities in their chapter and to have been busy handling their own sections without worrying too much about how, for example, Briffa handled review comments in his section.
Transparency is crucial to the IPCC process. There would be an imbalance in transparency when authors are known by name and Reviewers are not.
I agree that transparency is “crucial” to the IPCC process. As discussed elsewhere, IPCC has opposed transparency in favour of confidentiality, with the situation getting worse with the furtive adoption of the Jones-Stocker amendment. In addition, it would be easy enough to add back the reviewer name when the review comments were published. The present system is designed not for transparency, but to enable authors to decide how to respond, depending on who the reviewer was.
IPCC needs a consistent approach with regard to its Expert Reviews. Changing the approach to require anonymous Review comments would imply that there is a problem with the named Reviewer approach, which is not the case.
This is perhaps the stupidest argument – even by IPCC standards. Once again, the pretence of infallibility. There are problems with the named reviewer approach. I can understand an argument that, after considering a balance of problems, an institution might choose one method rather than another. But worrying about the impact on infallibility is not a valid reason.
Named Expert Review is more efficient as it allows writing teams to liaise with Reviewers when there is a need for clarification.
Again, this is a fatuous argument. Their own experience with the Carbon Dioxide Capture Task Group permitted authors to locate reviewers for follow-up if necessary. In addition, there is little evidence from the Climategate emails that AR4 reviewers bothered to do this. Briffa, for example, didn’t try to clarify things with me or Ross.