Although the IPCC calendar webpage doesn’t link to session documents of the 34th session (Kampala Nov 2011), David Holland has alertly located the documents – see here.
IPCC rejected a proposal for anonymous peer review – see document here (page 12 on).
We haven’t discussed this topic previously (in an IPCC context). On reflection, the adoption of a form of anonymous peer review by IPCC seems to me to be a very good idea and might somewhat mitigate some problems. There is no doubt in my mind that review responses are strongly conditioned by who is making the suggestion. The experiences of Ross and myself are vivid examples.
In the IPCC’s Special report on Carbon Dioxide Capture and Storage of WG III of the Fourth Assessment Report, reviews were anonymous. The review document described the results of anonymous review as being entirely positive:
The TSU prepared a list of Review comments with numbers. The TSU had a conversion table linking the numbers to the Reviewer’s names. During the treatment at the lead author meetings and the formulation of the authors’ responses the authors only saw the numbers. The authors knew that in case they would need to consult an Expert Reviewer for getting some clarifications about his comments, the anonymity could be lifted and the coordinates of the Expert Reviewer would have been made available to the authors. In practice it turned out they did not need to use this provision. The anonymity was continued until finalization of the final draft report.
The Reviewers and authors have been informed beforehand about this procedure. The number of comments was normal compared to other special reports. No Reviewer used improper or inappropriate language. The WG III co chairs and TSU held an enquiry among the authors and Review Editors. They considered the anonymity an improvement, because it made them concentrate fully on the content of the matter, disregarding the persons and their background, which was more time efficient.
The review document summarized the advantages of anonymous review as follows:
• Authors will concentrate on the content of the matter, excluding (subconscious) biases.
• There is positive experience in WG III AR4 – also the Task Force on the Greenhouse Gas Inventory Program (TFI) has practiced anonymous reviews with a positive judgment of the authors and Review Editors.
• Authors cannot be criticized anymore of ignoring comments of specific individuals or representatives of scientific schools or interest groups, as happened in the past.
• It remains possible for authors to contact Expert Reviewers if there is a need for clarification.
These seem pretty convincing arguments. Here are the arguments against:
• The Task Group on Procedures was installed in order to consider the recommendations of the InterAcademy Council (IAC). The IAC did not recommend anonymous Reviews, so there is no compelling reason to address this.
• There is a risk that Exper Reviewers could take advantage of their anonymity by burdening authors with unprofessional or inappropriate comments.
• Measures against biases are already sufficiently taken by having Review Editors and by having a authors working as a group.
• Transparency is crucial to the IPCC process. There would be an imbalance in transparency when authors are known by name and Reviewers are not.
• IPCC needs a consistent approach with regard to its Expert Reviews. Changing the approach to require anonymous Review comments would imply that there is a problem with the named Reviewer approach, which is not the case.
• Named Expert Review is more efficient as it allows writing teams to liaise with Reviewers when there is a need for clarification.
In my opinion, none of these reasons stands up.
The Task Group on Procedures was installed in order to consider the recommendations of the InterAcademy Council (IAC). The IAC did not recommend anonymous Reviews, so there is no compelling reason to address this.
The first argument starkly shows the hypocrisy and opportunism of IPCC. IAC did not recommend (or even consider) Jones-Stocker enhanced confidentiality. This was not not mentioned in the briefing documents for the Jones-Stocker amendment. Worse, they represented the changes as addressing issues raised by the IAC.
There is a risk that Exper Reviewers could take advantage of their anonymity by burdening authors with unprofessional or inappropriate comments.
The IPCC’s own experience with the Carbon Dioxide Task Group was reported to be the opposite. To the extent that accountability was an issue, the reviewer names could be removed from the version given to authors for comment, but restored in the final publication of review comments, thereby ensuring accountability.
Measures against biases are already sufficiently taken by having Review Editors and by having authors working as a group.
Neither of these measures had the slightest deterrent to AR4 author responses. In practice, authors seem to have divided up responsibilities in their chapter and to have been busy handling their own sections without worrying too much about how, for example, Briffa handled review comments in his section.
Transparency is crucial to the IPCC process. There would be an imbalance in transparency when authors are known by name and Reviewers are not.
I agree that transparency is “crucial” to the IPCC process. As discussed elsewhere, IPCC has opposed transparency in favour of confidentiality, with the situation getting worse with the furtive adoption of the Jones-Stocker amendment. In addition, it would be easy enough to add back the reviewer name when the review comments were published. The present system is designed not for transparency, but to enable authors to decide how to respond, depending on who the reviewer was.
IPCC needs a consistent approach with regard to its Expert Reviews. Changing the approach to require anonymous Review comments would imply that there is a problem with the named Reviewer approach, which is not the case.
This is perhaps the stupidest argument – even by IPCC standards. Once again, the pretence of infallibility. There are problems with the named reviewer approach. I can understand an argument that, after considering a balance of problems, an institution might choose one method rather than another. But worrying about the impact on infallibility is not a valid reason.
Named Expert Review is more efficient as it allows writing teams to liaise with Reviewers when there is a need for clarification.
Again, this is a fatuous argument. Their own experience with the Carbon Dioxide Capture Task Group permitted authors to locate reviewers for follow-up if necessary. In addition, there is little evidence from the Climategate emails that AR4 reviewers bothered to do this. Briffa, for example, didn’t try to clarify things with me or Ross.
Climate Audit 
49 Comments
Curious, but no-where in this document does it state that the IPCC rejects anonymous review. In fact, the actions taken are specifically (p5):
“not to amend the IPCC Procedures in any way with respect to the anonymity or non-anonymity of expert reviews and not to preclude a different approach in future assessments”
How is that a rejection?
So here’s a test, will you amend the blog title to something more factually based? ‘IPCC discusses anonymous review’ perhaps. Or is that not incendiary enough?
The status quo of AR5 is that reviewers’ comments are not anonymous. In its explanation the proposal document says,
“It has been suggested that the Expert Review process could be made more objective by making it anonymous.”
The authors decided not to. What exactly is the problem with blog heading?
The authors’ responses are anonymous in the status quo. That was a problem in AR4. In the TAR they were never published – even worse.
I have some sympathy with Steve’s view but come from a different standpoint, which is that my government signed a binding Convention on access to environmental decision-making and then acquiesces in wholesale breaches of it.
David
As you know, a few threads ago Don Keiller noted that Brussells had signed the EIR Convention as well, but in practice blithely ignored it. So too do Whitehall bureaucrats
Our Aus bureaucrats are even worse – simply removing from internet publication and without comment, papers and articles that have proved embarrassing, then ignoring any questioned criticism of this … happened and is happening many times
One cannot vote bureaucrats out. Politicians are too short-lived to remove them. Passive aggression works well if you cannot be sacked for it
Re: ianl8888 (Feb 3 00:09),
Ian, you are right about the bureaucrats. See my comment at 8:36am. How long has David Warrilow been the UK’s head of delegation? As far as I can tell he does what he likes on IPCC matters. And he pops up in a few of the Climategate emails.
Warrilow turned up in a number of emails in the campaign against Soon and Baliunas. They prepared questions to be asked of Soon at a briefing conference in Washington. Later, on the eve of the Senate committee hearing featuring Mann on the one hand and Soon on the other (see 1595), Hans Verolme of FCO wrote to Simon Brown of the Met Office and Mike Hulme of CRU, copies to Warrilow, Geoff Jenkins, Peter Stott, Cathy Johnson of DEFRA and Maria Noguer of DEFRA) telling them that an article by Brown and/or Hulme “debunking” Soon and Baliunas, “stripped of its [UK] origins”, had been provided to “sympathetic Senate staff”:
While I think anonymous review would be good, I don’t believe it is enough. Until authors are forced to address review concerns, it doesn’t matter how bad the work is.
Quite frankly, the goal of the IPCC was never about understanding the science. From that standpoint, they cannot ever allow a legitimate process to take hold. They need to have absolute control over the message, and anonymous review – with legitimate consequences, will deteriorate that control.
Mark
Ftzr, somehow in digging out your quote, you missed all the language that went with it.
I’m sure that was merely an oversight on your part.
But of course, Ftzr is a big fan of anonymous review, just not for important stuff.
But the IPCC rejected anonymous review, what’s wrong with that as a title to the piece?
What IPCC did was to reject an amendment of existing rules, i.e. an amendment establishing anonymous reviews, leaving in place instead the existing system of non-anonymous reviews. Thus, FITZR comment is, IMHO, utterly misleading.
It will be interesting to see [names withheld] comment on this, given their comments extolling the virtues of anonymous review in other cases.
Good one Terry…
Im writing a book with a partner on real estate development and suggested a chapter on the fallacy of green buildings. It is, but to write about it is a career killer.
Simple solution – list the reviewers so it is open and transparent who is taking part in the review process, but provide no named attribution for their comments.
Normally I support McIntyre says, but in this case I disagree.
We know by the ClimateGate emails that anonymity doesn’t work. We know that the ClimateGate folks are fully capable of ignoring input by content.
We know that the ClimateGate folks may not know who an anonymous reviewer is, but they still know that it couldn’t have been one of “them”.
So, I don’t see anonymity as anything but a way for review leads to reject by authors via content while justifying themselves to the world as being non-biased.
Far better to be able to track it when their rejections turn out “coincidentally” to be by author.
In disagreeing with Steve it’s worth remembering what he wrote yesterday:
I fully agree with that. I also agree with you that anonymity of reviewers probably wouldn’t add much to the five year literature review, based on what we witness in CG1&2. And I agree with Steve that it’s rank hypocrisy the way the IAC’s silence is used as cover in this case but its express advice is ignored, indeed mispresented, on other crucial matters of transparency.
I disagree.
By not having the ability to review anonymously, they are purposely making it so any climate scientist that would fear for his/her academic future and fiscal future to not be able to provide quality reviews and opinions.
It make no difference as to the identity of the reviewer, just what his/her review would be.
DOH !
Forgot to add:
Climate Gate has shown that despite the established groupthink, there is a plethora of disagreement amongst the CAGW promoters. I’m sure some of those opinions would make fantastic reviews.
I agree with that! But the whole point about Climategate is that it’s what the Team, the magic circle, were saying to each other when they thought nobody else could hear, with the express purpose of portraying a completely different ‘consensus’ view in public.
I am for full openness of the IPCC – all review comments seen in real time by anyone who wishes to browse them, anywhere in the world. With a Real Names policy for all comments. And only selected people able to comment. But I also think that the five year summary of the literature is a flawed idea. My advocacy of openness is partly to make that plain.
It’s not just about the ‘rules’, it’s about the culture which has become deeply embedded in the senior people involved – that the IPCC five-yearly report has to be authoritative, as the lady from Botswana desired, leading as David Holland says to all kinds of falseness.
Our one hope is that many more governments will follow the lead of Canada in making clear that IPCC output is no longer considered holy writ – and finally begin to cut off the funding. I do expect this to happen but as things stand it’s not as certain as it should be, given the dystopia that could be facing us the other route.
The IPCC appears to have pushed the stick all the way forward. At any moment, I expect to hear the wings ripping off.
If expert review comments were rendered anonymous, then someone might suggest extending anonymity to comments made by government-appointed reviewers. If some anonymous government-appointed reviewer suddenly comes over all independently-minded, and starts singing from the wrong hymn-sheet, then how much harder is it for an activist or NGO to influence said government?
Re: michael hart (Feb 3 01:23),
Government comments are largely anonymous and (not a lot of people know this) many are not government comments anyway but come from NGO’s and others bodies that governments ask to comment on the SOD.
Examples? Sources?
Not that I disbelieve you …
Re: Richard Drake (Feb 3 06:50),
Richard,
My very first ever FOIA/EIR request was to Defra in 2007 for info on AR4. They ignored it until, as part of my long running correspondence with them, on 15 May 2008 they wrote to me reproducing in line a letter that they had prepared to send on 27 January 2008 but never did. Crazy or what?
To cut a long story short eventually they sent me their copy of the comments they submitted on the AR4 SOD. But it did not tally with what the IPCC published. Eventually – and it will take me a while to find the stuff – Defra told me who wrote what comments. I will dig it out and post a link to it later.
Re: Richard Drake (Feb 3 06:50),
Richard,
I can send you a 2.6MB pdf file of my Defra AR4 WGI FOIA correspondence if you want to wade through it. On pdf page 39 Defra states
On pdf page 98 Defra wrote
I had specifically asked who wrote SPM-19 because it had amused me
David, I enormously appreciate everything you’ve done and are doing on this. Do send the big PDF. But what you’ve extracted from it here should be a major story in its own right.
It is worth remembering that the IPCC is a horse designed by a committee. John Zillman has written,
The key word is authoritative. Remember Milliband’s and Rooker’s statements following TGGWS.
The problem is that some people instinctively distrust authoritative bodies that are not open and transparent and it is difficult to be authoritative on matters of great uncertainty and still be open and transparent. That’s why for years, while no one thought to test it, the IPCC boasted about how open and transparent it was whereas in truth it had never been. Now the gloss has peeled off it is behaving like a banana republic.
Totally off topic, but at Bishop Hill they are discussing the charging of Chris Huhne, our Secretary of State for Energy and Climate Change, with perverting the course of justice. Of course, he is innocent until proved guilty but if so he will be the third Minister in charge of climate change, in just a few years, with an ignominious end to his career.
Chris Huhne (ex) Minister for Energy and Climate Change in the UK has been charged with “perverting the course of justice” and has resigned his position.
http://www.bbc.co.uk/news/uk-politics-16866127
If convicted he will do jail-time.
Huhne is the person mainly responsible for plastering the landscape and sea in and around the UK with near-useless windmills.
It has costed every taxpayer some £200/year in increased energy costs and the number of UK citizens in “Fuel Poverty” has doubled on his watch.
Huhne and his obsessive policies have undoubtedly resulted in people in the UK dying of hypothermia.
What is it with “Green” politicians and scandal?
Maybe you have to be “economical with the truth” when trying to sell a scam.
On first reading, I had thought anonymity in the reviews a bad idea. After all, who says something often does matter. But this is exactly the reason that anonymity should be preferred. An anonymous comment must stand on its own. It must be compelling and conclusive. It must carry its own water.
Maybe they haven’t thought about this enough.
There’s a good example from the Climategate emails showing a completely difference response to Jones and I making the same comment to IPCC authors.
In the thread of email 609 (2005-08-01), Jones had asked AR4 chapter 6 author Tim Osborn why they hadn’t shown the Law Dome d18O series (shown in Jones et al 1998 and Jones and Mann 2004) as one of the (scarce) SH proxies in their graphic, a query reported by Osborn to his chapter 6 co-authors as follows:
In my review comments to the AR4 Second Draft, I asked exactly the same question.
Coordinating Lead Author Overpeck (CG1- 709. 1153233036.txt) sneered that a so-called “expert” reviewer would ask such a question:
In the incident in question, they were aware that the Law Dome d18O record, one of very few well-dated SH proxies, had a warm MWP. They elected not to show Law Dome d18O in the figure, but to insert a short CYA discussion of Law Dome in the running text.
The difference in attitude to the same comment being made by Phil Jones and by me could hardly be more stark.
While I’m sure that they could with considerable probability identify comments by McKitrick and/or me, by knowing for sure, they were disposed to argue against the comments rather than consider them seriously.
It seems to me that there’s a process interaction between two unusual features of IPCC review here: (1) that the Lead Authors ultimately decide on whether or not to consider a review comment; (2) the naming of the reviewers.
If anonymity is good for journals so the reviewer does not face reprisal from a couple of authors, how much more so when you are reviewing the big beast IPCC and your comments are going to be published? Being critical could be career-ending.
Craig: Anonymity in these proceedings is a double-edged sword. Ironically, it’s nearly the same fix that Stocker and Jones are implementing anonymously (or so they thought) behind the scenes. Anonymity is currently being used to allow individuals to hide behind the group. The sword cuts deeper when people like Stocker and Jones are using it.
RDCII has it right in a comment above. The people entrenched at the higher levels of the review process will abuse anonymity to cover up their machinations. Until the process used to select lead writers and other honchos is dramatically reformed, anonymity as it is being proposed here will be used to circumvent openness and transparency.
I think your analogy is faulty. These reviews need to be viewed more as public proceedings– more like governmental committees coming to determinations on matters that effect public policy. We need more sunshine, not less.
And if a junior academic is criticize of the pet ideas of a senior one and that senior one is in the old boys group that sits on funding boards, then what. So to comment that Professor X’s paper has been superseded by later research and having Professor X then review one’s next grant would not be a career limiting move.
I have had experience in which a senior researcher has held an entire community of researchers in terror. His ideas were 25 years old and commonly thought, and discreetly expressed, to be out of date but he had strong influence with funding groups and was quite eager to use it to suppress other ideas. Could a junior academic withstand the king of pressure to which Steve McIntyre has been subjected?
On the other hand, anonymity could be counterproductive in allowing reviewers to relax and not be as thorough as they might be. Some people seem to like to treat the status of “IPCC expert reviewer” as something to put on their CV, but do they all do a thorough job in reviewing the areas relevant to their expertise? If someone is an expert in a particular field and they know they’re going to have their name at the end of the report as an expert reviewer, then I think for most people this is an incentive to try to make sure there are no errors related to their area of expertise. Anonymous reviewers, however, can feel free from such pressures as errors in the final report will not reflect badly on them.
Also, if the authors reject the reviewer’s comments, it then becomes a matter of record that the reviewer suggested changes but in vain. Then if the suggested change turns out to be a genuine error that was not fixed, the reviewer can clearly show they tried to do the right thing.
It would appear to me that this thread has gone in 2 directions:
In one we are talking about attempts to avoid personal biases against the reviewers from affecting the authors responses and in the other about attempts to avoid the reviewers from facing recriminations in their professional careers for how they might choose to review.
Eventually revealing the reviewers names would not address the recriminations issue above and thus that “fix” would require never revealing the reviewer names and also require the authors’ capability to clarify a review be handle through a third party who would have to be sworn to secrecy forever.
I have problems with both approaches/rationale for anonymous reviewers. In the first instance of protecting against biases against the reviewers, I think might well be an illusion and used as marketing issue for fairness where it does not necessarily exist. I do not think it would take much surmising to determine who might have made a review, or at least who of a few individuals, and particularly so if it came from a reviewer with a minority point of view.
If the fear of reprisal runs that deep amongst the reviewers who might risk their careers for making review statements counter to the consensus thinking, I would not think that a reticent member of the reviewers would risk their careers by abruptly changing course and making bold anonymous reviews for which the source well could eventually be leaked or discovered by other means.
Agreed on both counts Ken.
“In the incident in question, they were aware that the Law Dome d18O record, one of very few well-dated SH proxies, had a warm MWP. They elected not to show Law Dome d18O in the figure, but to insert a short CYA discussion of Law Dome in the running text.”
Two different author responses to two different reviewers but with the same result. In does not appear that an anonymous review would have affected the final product.
A sample of one is hardly going to give any leaning one way or the other. I am sure the Team would work most of the anonymous reviews out but if a small % still got through it would be better than the current situation.
One of the major changes needed is the pre-release of the draft SPM and AR5 proposed final version to a completely outside group of specialists. Past inquiries have said that statisticians need a look, but given the way that SPM involves social reorganization as much as physics, chemistry, stats & math, there should be a prior overview by economists and top business managers with a proven track record.
I’d call that “transparency”. The opposite, being cloistered academic secrecy, has already revealed methodology dangers with global consequences.
Assuming the current five-year-summary scenario enacting this kind of review would add considerably to transparency as I also woulddefine it. Even if my radical ‘goldfish-bowl’ proposal was adopted from earliest drafts there might well be virtue in introducing different types of expert at different times. The most valuable people won’t have enough time to follow every change. That’s the kind of thing to be considered in a truly rational system that desires to maximise transparency and integrity.
But do those at the centre right now want to maximise such things? Why don’t we ask Jones and Stocker their opinion on that?
The Vatican has spoken. Take heed and obey.
Steve, this is off topic, but did you know the fate of the whole world is in your hands?
email form climategate II
http://tomnelson.blogspot.com/2012/02/email-2378-oct-2009-desmogblog-warmist.html
I happen to think though that you actually may have helped save mankind’s use of the ‘evil’ carbon based energy
There is much that Karl Popper would have had to say about this, so I leave it at one quote:
There are all kinds of sources of our knowledge; but none has authority … The fundamental mistake made by the philosophical theory of the ultimate sources of our knowledge is that it does not distinguish clearly enough between questions of origin and questions of validity.
If the lead author needs some clarification from Anon13 say, all they have to do is
send their query blindly to Anon13, who then responds and answers the question. What I
really want to see is the rough draft A, then the expert Anon reviews to that, then the
next draft that takes into account comments from the reviewers. It may be possible to see where the author totally ignores some key comment from a reviewer. This process is repeated to the final document.
I think most of us wish to see that
Such a public process is precisely what the IPCC will avoid
The best I can hope for is that the various drafts are leaked (as happened with ZOD and, so far, some of FOD Chapter 5), allowing us to assess the evolution of AR5
Steve, this is unrelated, but pertinent.
http://www.eva.mpg.de/denisova/
Complete openness and production of all raw data, no passwords, free of charge in researching human anthropology.
Why should a body that is mostly funded by public money have to go through some sort of audit process of its people, methods, ideals? Its ultimate aim is only to change the very basics of the way we use our economies to live, trade, and interact both nationally and internationally.
It is, of course, a settled fact that the IPCC is only staffed by the very best hard working scientific professionals of the highest morals and ethics, above the normal standards that mire everyday scientific bodies in controversy. This in itself puts them beyond the ordinary strata of scientific probity and therefore requires that they can only be self regulating.
.
.
.
.
.
.
/sarcoff
Believe any of that and I got a computer model to sell you!
Some in the press are pointing out that the Pacific institute is a “non profit” suggesting that only profit based organizations lie etc. See:
http://blogs.kqed.org/climatewatch/2012/02/24/pacific-institutes-work-rises-above-the-gleick-fiasco/
Hiding behind the non profit label (The Pacific Institute is a registered 501(c)3 non-profit ) suggesting that this provides some immunity or more integrity is ludicrous. Non profit per above is a “Tax Status” also given to others orgs. like churches. Many of these organizations are well thought of others are mediocre and some are basically ideologues who say whatever fits with their ideology.
From Dr. Judith Curry’s website:
“There are five attributes of ideologues:
1. Absence of doubt
2. Intolerance of debate
3. Appeal to authority
4. A desire to convince others of the ideological “truth”
5. A willingness to punish those that don’t concur
Note that each of these characteristics is anathema to science.”
This institute under Gleick’s leadership has become ideologically driven. He has faked a memo as Steve Mosher and Steve McIntyre have clearly pointed out. He has ruined his career and destroyed whatever reputation this institute had. I feel sad for the man but the facts are damning!
In the latest (09/07/2012) IPCC WG1 AR5 “Coordinating Lead Authors[CLA] , Lead Authors and Review Editors” listing I see only two of the old hard core Team members: Tim Osborn in Ch5. as one of 15 lead authors for “Information from Paleoclimate Archives” and Kevin Trentberth as one of 4 review editor for Chapter 14: “Climate Phenomena and their Relevance for Future Regional Climate Change”. Who are the new Team representatives?
The USA has 69 scientists in, the UK 29, France and China 17, Germany and Australia 13, India 9 (amazing knowing the miserable level of government climate science here) and Russia just 5.
One Trackback
[…] Climate audit […]