John Nielsen-Gammon writes as follows in a comment in the preceding post:
If you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse, it is necessary for you to pretend to not be a reviewer. Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed.
I welcome John’s comments, which have been constructive and interesting, but I don’t think that he has fully thought through what happens when the scientist wishes to talk to the public.
Scientists may expect “disingenuous” conduct from other scientists, but the public doesn’t. The last thing that the climate debate needs right now is more climate scientists being “disingenuous” in their communications with the public – there lies the road to “hide the decline”.
There has to be something wrong with a policy that results in or condones a climate scientist talking “disingenuously” to the public. In the case at hand, I think that N-G goes wrong on his premise:
lf you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse
It seems to me that a reviewer, wishing to engage in public discourse on the reviewed paper, should disclose that he was a reviewer. If he does so, there is no need for dissembling. If he doesn’t want to disclose that he was a reviewer, then he doesn’t need to make public commentary. (I present this as a suggestion, rather than a settled position.)
Steig could easily have avoided the conduct that led Lucia to characterize him as the “Rod Blagojevich” of Science. Steig was on record as saying that he disdained blog commentary on his peer reviewed articles (an odd position for a realclimate founder but that’s another story). If Steig didn’t want to disclose to the public that he was a reviewer of O’Donnell et al 2010, then he could have refrained from public commentary. He could have said that he would reply to O2010 in the peer reviewed literature. But that’s not what he did. After disdaining blog commentary by critics, he himself published blog commentary against those critics and made comments at blogs and to blogs about O2010.
Under the protocol that I suggest here, if he decided to comment to the public either at blogs or to reporters, he should have stated that he had been a reviewer of O’Donnell et al 2010, that, as a reviewer, he had argued that the article was flawed and that the article was ultimately published over his protests. All of this would have been true and would have forestalled at least some of the recent controversy.
In addition, once a reviewer embarks on a path of public criticism of an article that he reviewed, it seems to me that this is (or should be) an implied waiver of any confidentiality claim that the reviewer might expect on his review (in those cases where the journal has confidentiality policies binding on the authors.) It might be worthwhile for journals to think about this issue.
Reasonable people can disagree about precisely how reviewers should deal with this sort of situation. But any policy that results in more climate scientists being “disingenuous” in their communications with the public cannot be right. End of story.
Climate Audit 
126 Comments
“The situation at hand is a reviewer wanting to engage in “public” discourse, such as commenting at realclimate, blogs of the co-authors and other blogs, rather than private or semi-private discourse.”
I think the core issue is that Steig had a conflict of interest and was using the tradition of reviewer confidentiality to hide the conflict. The ethical issues that arise from this are compounded by Steig’s making public commentary, particularly since this commentary seemed to reveal that he had engaged in a -snip – trick as a peer reviewer. Both his conflict of interest -snip – could only be properly evaluated by the public if his role as a reviewer were known.
Just a quick note to clarify that the word “publicly” in my comment meant “publicly”.
Steve-
got it. I’ve edited accordingly. I don’t see that the change raises controversy and have not left a strikethrough. For those worried about such things, I removed a contrast between talking to the “public” and talking to semi-private workshops following JOhn’s clarification. See above comment for prior language.
Re: John N-G (Feb 17 13:17),
John, what do you think of the policy suggested in this post?
Steve- I disagree with your premise:
The various alternative forms of peer review are all suboptimal and involve tradeoffs, and the jury is still out on the question of which one is best in terms of the quality of the resulting papers and the fairness of the system. I would agree that requiring an often tortuous path to preserve anonymity is a negative factor, but not a disqualifying one.
For myself, I would choose to disclose that I was a reviewer if it were relevant to what I was going to say. And I have a hard time coming up with a hypothetical in which it would be relevant. I would normally be talking about the paper itself, not the contents of the reviews. And I wouldn’t be able to talk about my own reviews without talking about (or implying something about) the authors’ responses or the editor’s evaluation, both of which were given in confidence and so are off-limits.
The only such circumstance I can think of is when I would wish to criticize a particular aspect of a paper that I felt some responsibility for as a reviewer. I would feel bound to share some of that responsibility if I criticized it in public.
Now I know many readers will think, “Wait a minute, isn’t that precisely what happened here?” Not quite. I’m not going to rehash arguments that have taken place elsewhere that turn on the meaning of individual words in the context of other words and perceived levels of power and influence. It was Eric’s call whether he felt responsible for the use of iridge; he did not regard himself as responsible for that choice, so there was no point in announcing himself as a reviewer.
There’s a basic problem here – in the public arena, people are supposed to unltimately trust climate science. Dishonesty in any form gives a basis ofr evaluating how trustworthy the individual scientists are, as well as the processes they tale part in. Once a paper is published, I see no point in keeping the criticisms confidential. I also see little point in keeping the reviewers’ names confidential, but I think there is a good argument that the comments match-up with reviewers may still be a point of confidentiality to prevent the reviewer from losing face due to silly comments (some will be more rushed than others, for instance).
As for what Eric did and did not feel responsible for, that is unknown and unknowable in any objective sense.
Anonymous reviews are kind of like anonymous blog postings. People will say the darndest things when they’re anonymous, that they wouldn’t even dream of saying if they weren’t
Sorry John. But “disingenuousness” is at the core of many of the problems in the climate research community today. To advocate for more is to miss the point entirely.
Steig wanted it both ways. Using the mask of confidentiality, he hammered on the paper in what is pretty apparently an attempt to hold it up, if not torpedo it entirely. Then, when the authors incorporated his suggestions, he attempted to use the fact of his confidentiality to play against those changes in public, in an effort to further discredit his critics.
I have no sympathy for him whatsoever. And I have no sympathy for a system that fosters that kind of attitude and approach to otherwise honest and genuine inquiry.
An interesting thought.
One of the primary reasons for anonymous peer review is for the reviewer to feel free to make honest criticism without fear of reprisal. If, however, that same reviewer chooses to later go on the record in public criticizing the paper, then one could easily argue that fear of reprisal would not have prevented honest criticism during review, as said reviewer is clearly not afraid of such reprisal.
In cases such as these, it would seem that the cloak of anonymity was not needed to begin with.
Ryan says: “One of the primary reasons for anonymous peer review is for the reviewer to feel free to make honest criticism “. Can Steig’s efforts be in any way classed as honest? Surely, he was pursuing an agenda.
We’re abstracting now.
If the reviewer doesn’t fear reprisal – as evidenced by publicly criticizing the paper – then there should be no need to maintain the pretense of not having been a reviewer.
There is a difference between a critique offered privately, in the capacity of a peer reviewer, and one offered publicly. If a critical peer review for a journal is attributed, there is the risk of reprisal without compensating benefit to the reviewer. If a public critique is attributed, the author of that critique may expect (though not necessarily correctly) that their public reputation as an expert will be enhanced.
Re: Ryan O’Donnell (Feb 17 13:40), yes, I think the fear of reprisal is the underpining of anonymity.
Steig had a diminished fear of reprisal from you. it’s not likely that you will ever be asked to referee.
his decision to reveal himself to you can be accounted for by considering these things. 1. it might put you
off and others off from speculating. 2. He might have felt some measure of compassion. 3. he didnt face reprisal.
There is an extensive literature on anonymous interactions in collaboration. As I recall, it does improve collaboration in many cases. Perhaps this literature should be consulted
Anonymity is useful until it’s complete. See all the literature on the Delphi (and derivatives) technique for reasons why.
Note that if comments are made anonymously then comments from the “Big Boss” and comments from a junior worker are given equal weighting and can be considered in the same way. This is not ‘fear of retribution’. I have seen the opposite effect many times in standards committees in which certain on-group members have their views considered and the rest of the participants learn that commenting is a waste of their time and effort.
There are indeed cases in which people are inhibitory from giving candid comments on proposals from someone with corporate power. In that case, the propose may elect anonymous reviews to enable candid comments. I have seen cases in which a senior executive will ask for reviews and comments before expressing any view either favourable or unfavourable on a proposal. This appears to me to be case in journal reviews in which an editor asks for advice on a paper without giving any indication of his own views. It is the editor’s decision to make with the reviewers just giving advice. This is the reason why I find the current brouhaha rather puzzling
http://portal.acm.org/citation.cfm?id=181524&dl=ACM&coll=DL&CFID=10454075&CFTOKEN=61514780
The article at the above URL seems to have great relevance to the current Steig-O’Donnell dispute. Anonymity of interactions is suggested to get over some the the diffculties we are seeing in the current dispute.
Abstract
Interesting though this is for other contexts I can’t see its relevance to O10, unless Broccoli, the editor, didn’t know who Reviewer A was. That’s surely not how the anonymity in peer review works, is it? (Genuine question, I’m that ignorant.) The problem here was created, as I see it, by a combination of Steig and Broccoli. If the editor hadn’t put so much weight on Reviewer A’s remarks then there would have been much less of a problem for the four authors of the paper.
I go with Steve McIntyre’s suggestion as a rule of thumb for future peer review – or for a total opening up of preprints and review. Either one will do. But disingenuousness has to be out for climate science, given the importance for all of us of it regaining the full confidence of the public.
The editor (Broccoli) of course knew that Steig was the reviewer. And he obviously was aware of the conflict of interest. I don’t think that the editor placed undue weight on Reviewer A’s comments. As he wrote after the first round of reviews in response to OLMC’s complaint of bias: “There is a difference between being asked to respond to the criticisms contained in a review and being required to agree with them or act on them. If you feel a particular criticism is unjustified you can respond accordingly.” In the second round, only reviewer A asked for a major revision, but OLMC agreed to do so rather than challenge this. After the third request of reviewer A for a major revision, I suspect Broccoli had had enough, and guided OLMC to respond with comments rather than revise again.
This is an important observation. Teh editor makes teh decision. Teh referees offer him advice which he can take or not as he pleases. I received on spurious review from a referee that was obviously threatened by the new results which, by implication, showed that past approaches were worse than wrong. They were pointless and useless. He “went to town” on my paper. I read only the first couple of paragraphs and realized what was going on. I did not respond to it. The editors did not take much notice of his review as well since the paper was published.
The use of anonymity in peer review in the case of publication reviews is that the reviewers can be candid and forthcoming without risk to their interpersonal relationships
There are three kinds of trust recognized as being important for group cooperation, These are a) interpersonal trust, b) competency trust and c) integrity trust. My opinion is that the current controversy is a good indication how the loss of interpersonal trust can harm the other kinds of trust and make collaboration very difficult. We are currently seeing how a loss of interpersonal trust causes diminishment of the others. It lowers the estimation of competency and integrity even if there is no real evidence to do this. Again this only reinforces teh diffciulty in creating trsut and collaboration.
The theory of iterative games has been studied to discover essential elements in building trust and cooperation. That is, in one example, players repetitively compete in a prisoner;s dilemma game. The players have to devices whether to cooperate or defect and they do this by trying to judge the other player through their past actions. If a player defects and causes losses thereby to the other payer he benefits in the short term but the group suffers in the long term due to lack of cooperation, However the defecting player must be punished since if one player acts as a patsy then again the long term benefit to the group is diminished. A major finding in this research is that the ability to forget a slight is essential. If a past defection is allowed to linger and fester in the strategy them teh group suffers.
I would recommend that the results of the theory of iterative games be used in climate science. past slights should be actively forgotten and cooperation always assumed. Unfortunately the AGW community appears to be one that will not take this lesson from the peer reviewed mathematical and sociological literature.
Axelrod’s book, Th Evolution of Cooperation, details the importance of forgetting to create trust and cooperation.
The TIT FOR TAT strategy is very effective. It forgets past slights almost at once and rewards cooperation immediately
steven – Hypothetically, would emailed death threats from anonymous third parties count as reprisal? And yes, I’m being disingenuous.
Re: John N-G (Feb 17 18:44), I think one aspect you under rate here is the facts that Ryan was a new comer. And that Steig had basically issued a challenge. That challenge (skeptics dont publish) has been a issue of contention for some time.
In anycase i think it interesting to ask Steig. why did you reveal yourself?
and to ask him how he would react if ryan were asked to review his next paper?
Clearly with the personal animus on the table now, you would not suggest it?
but there was personal animus on the table when steig took the role.
Ryan and company were already confident that Steig was A. Steig didn’t “reveal” himself. He simply confirmed what Ryan had already surmised and communicated. “Yeah, you’re onto me,” is quite a bit different than a revelation.
“In cases such as these, it would seem that the cloak of anonymity was not needed to begin with.”
It does appear to undermine the rationale for anonymity, though perhaps not completely so. One can imagine that circumstances might well change, for example, such that a given reviewers fear of reprisals might diminish over time (e.g. making tenure, or finding support from other respected minds).
In any event, it’s plain that an individual reviewer outing himself poses no threat to the institution of anonymity. Consequently, it seems quite reasonable to expect reviewers to do so, when necessary, due to other choices they make, to avoid disingenuousness.
Maybe John NG could tell us when Climate scientists should not be “disingenuous”.
I think the appalling thing about Climate science is that many people now assume that Climate scientists, and possibly all scientists are being “disingenous” all the time.
University policies and procedures apply to all staff.University administrators usually wish to apply the policies and procedures of the university consistently and fairly. Often such policies and procedures require scientists and other staff to behave honestly and with integrity in their dealings with colleagues and folk outside the university.To what lengths might one go in being “disingenuous” to avoid revealing that one is a reviewer and still comply with such a policy?
Also, implied into every employment contract under British law is the term that an employee will do nothing to bring his/her employer into disrepute. I can see dangers to any British scientist who wants to follow the opinion of Nielson- Gammon.In my view, if British law appied, Steig’s behaviour is worthy of a disciplinary investigation exploring whether he has brought his university into disrepute. One complaint might be enough. O’Donnell might be in the same boat.
I think it is OK for an author of a criticized paper to be a peer reviewer, PROVIDED that the editor doesn’t allow the kind of everything-but-the-kitchen-sink “review” by Steig et al. Authors of a paper being critiqued might have insights that other reviewers might not have. But both they and the editor have to be on good behavior if authors of a critiqued paper are to be peer reviewers.
I also think it is OK for a reviewer to pretend he or she isn’t a reviewer. It’s like a secret ballot; if you are relatively junior or with a funding conflict, you might bend your review if you were known to be a reviewer.
What went so wrong in this process were:
1. Steig trying through the review process to manipulate what the new paper said, so that he could later say the paper backed up Steig et al’s findings;
2. The editor allowing Steig et al to run riot with their 88 pages of comments — the editors should have detected such unsporting behaviour and quashed it, by removing Steig as a reviewer; and
3. Steig pushing O’Donnell et al to use a different statistical method, and then criticizing them for doing so.
Both (1) and (3) are so unethical that they should be enough to get Steig banned from….good scientific society?….for a long time.
John,
The 88 pages includes our responses – which were longer than the reviews. Steig did not write 88 pages of review.
I think it ok for an author of a criticized paper to be a peer reviewer provided that the identity of the reviewer is publicly disclosed. I think its fine for an independent reviewer to remain anonymous, but it is absolutely unacceptable for an individual with a clear conflict of interest to serve as a reviewer AND to not disclose the conflict. Conflicts of interest arise all the time in real life. A conflict of interest, in and of itself, is not a problem. It only becomes a problem when an individual fails to disclose the conflict prior to taking on activities which might be influences by the conflict.
Apparently they (the climate science community) see no conflict of interest in this matter.
From the management side, if I know that someone’s work is going to seriously question another’s work (or even extend it), I’m likely to have a sit down with the two sets of authors after the second paper is written. Only clarifications and corrections to misstatements of facts / interpretations are allowed to be made in the second paper – it is not rewritten (or a different paper all together produced). This is a courtesy to both, so everyone knows what’s going to happen next.
I think John has a divide by zero error in his moral calculus. I really see no justification for duplicity. One can always choose to be silent. No one is forcing anyone to engage in public or scientific discourse. What is essentially being said is that his need to express his opinion on the topic outweighs his responsibity to be truthful. The problem with this is that as exceptions to a general rule increase, the amount of faith I have in the accuracy of that rule decreases. Find enough exceptions to the rule of “Honesty is the best policy”, and I simply stop believing you.
Consider how this is handled in another area. In the realm of national secrets, the policy is completely clear. An ambiguous standard that is applied to some and not to others does not exist. If you talk about things you are not supposed to, you will go to prison. With prison as the consequence, do you suppose that standard instructions are to lie ? Nope, they council you to shut your mouth firmly and avoid comment. Even in these circumstances, you can remain truthful, you just need to shut up.
From my point of view deciding you need your two cents in a conversation is a really, really poor excuse for duplicity. I guess it does come down to the price that one puts on honesty. Some people value it highly, others have causes that are more important to them.
I think that the story may indeed be over, because Stieg himself said that he’s not going to bother to agree to reviewing any more papers from certain authors or certain spheres of thought.
The more such ‘qualified scientists’ of which JNG speaks that do this may force an unassailable editor to decide that there is just no capacity to credibly review a certain publication, and be forced to recommend instead that it be published elsewhere…perhaps the disparagable ‘gray literature’.
And then again, it may not be over. The statement above could just be Steig exercising his “scientific” licence for “disingenuity” – which may (or may not) be part of the “culture” of which John N-G spoke earlier.
I can’t help wondering if this licence to be “disingenuous” was part of the late Stephen Schneider’s rationale for the telling of ‘scary stories’. But I digress …
Even if the mores of the “culture” are not applicable for some reason, I’m not sure to what extent one should take Steig at virtual his word on this. After all, he did feign an exit stage left with his:
Only to return to make several further “contributions” to the same conversation. [links and details at http://hro001.wordpress.com/2011/02/13/of-dancing-with-warmist-wolves-and-paint-by-numbers-big-pictures/%5D
Then again, perhaps the above is merely a demonstration of ‘some people leave and never say goodbye, while others say goodbye and never leave’.
If this is indeed an accurate reflection of Steig’s position, then perhaps journals need to consider giving papers to reviewers with the author anonymous. This would certainly seem to decrease yet further the fear of reprisals for a “bad” review. Perhaps J N-G could post a reaction to this suggestion?
“The more such ‘qualified scientists’ of which JNG speaks that do this may force an unassailable editor to decide that there is just no capacity to credibly review a certain publication, and be forced to recommend instead that it be published elsewhere…perhaps the disparagable ‘gray literature’.”
I seem to remember that one of The Team was sort of hinting at this strategy of refusing to review papers from certain authors – can’t recall where or when I came across this.
So yes – this might happen.
But wouldn’t this be an own goal for The Team, and the editor rejecting the paper?
There are quite a number of other ‘qualified’ scientists, are there not? Or are we asked to accept that only The Team is ‘qualified’, especially when one of them confesses not to be a statistician?
I think we ought to include the editors in this debate on peer review.
There may have been some value in allowing Steig to comment on the first submission of the O’Donnell paper and rapidly provide information for editor and actual peer reviewers to consider when evaluating the paper. I don’t think Steig (or similarly-conflicted reviewers) deserved a “vote” on whether a paper is eventually accepted; but they have unique ability to provide insights into previous work and valuable experience that could illuminate major problems with a critic’s work. (Perhaps his critics’ approach had been tried by him and found to be flawed.) When considering Steig’s input, the editor could ask Steig whether he is willing to back up his assertions with data and code, when appropriate, and take a refusal into account when evaluating his input. Under normal circumstances, the editor and reviewers should be able to handle revisions without further assistance from Steig.
If Steig’s input had been limited to “facts” (not opinions about whether or how the material should be published), there would be no need for confidentiality in either direction.
If Steig were able to rapidly assure the editor that he was likely to recommend publication with minor revision, there might not be a need for a third reviewer.
Reviewers may need to be disingenuous.
I’m gonna need some help calibrating my generally accepted ethics practices meter to this new piece of information.
Given the autism-vaccine kerfuffle, the last round of global cooling alarmism, ozone holes and right guard underarm spray, alar and apples, bpa containers and drinking water, not to mention all the emails that were released involving all of the same parties involved in the current fooferall!!!
I’m not too certain this is very good guidance.
You are a half-step from the ends justify the means, in my book.
Clearly that is the opinion. This is post-normal science.
It seems that we are arguing over the quality of mattresses on a battleship. Marginally relevant, perhaps, but it is of greater concern that the guns fire and the hull doesn’t leak. Peer-review has been blackened by events of the past few years, and climate scientists would be better served by ensuring that the process retains its credibility, than by fretting about its comfort. If their position is that reviewers are sacred but editorial boards can be cajoled and manipulated, and dissenting opinion blackballed, then why should anyone take them seriously?
I think Steve’s suggestion about public debate is quite to the point and involves the question of having one’s cake and eating it. If one tried as hard as possible to block a paper and then wish to debate it in public, that is asking for trouble.
One of the backstory issues here of course is the long history of reviewer/editor misconduct on climate science papers, from check-kiting,to refusal by editors to enforce data archiving, to “going to town” on reviews, to softball reviews of “good guys”, to getting editors fired, to sharing submitted ms with non-reviewers, to hiding declines, to hiding verification statistics, to hiding code, to cherry-picking data and lying about it, to bullying editors, and in this case by close associates and co-authors of Steig. [all of the above either in climategate emails or documented here at CA] The decision to reveal Steig’s identity in spite of a gentleman’s agreement not to is very much mitigated by this history, in my opinion.
Mr. O’Donnell
I have first hand knowledge what it’s like to be a “whistleblower”. Let me sum it up for you…HELL ON EARTH. You have a great deal of courage sir, you would not have outed Steig otherwise. The kitchen sink is coming, and it will be directed at your reputation. You will experice this strange psychological effect, that will make you want to give in. I sense you’re tough enough to hang in there, at least I hope so. One thing I found during my research on whistleblowing was, that institutions that had a policy that encouraged whistleblowing, had fewer whistleblowing cases (huh, imagine that?). I believe that the science community should adopt a policy that encourages whistleblowing.
“Whistleblowing”? O’Donnell et al wrote a scientific paper that contributes to our understanding of Antarctic warming trends. How is this whistleblowing?
Revealing Steig as “Reviewer A”, is whistleblowing in my book. I for one, appreciates it.
I believe there are a number of people here who would disagree with your assessment of the “balance of power” in this affair, as well as who is “speaking Truth to Power”.
Attempting to silence dissent is not “whistleblowing”.
I agree with the principle that if a reviewer wishes to criticize a paper they have no need to remain anonymous as a reviewer. I got slapped on another blog when I made that point so it is obviously not a universal perspective.
This by the way is different than the question of whether the review comments themselves should remain confidential.
In this particular case, I think I’m siding more with John N-G, rather than Steve (Sorry, Steve).
Due to the obligations of an anonymous review, it is not unreasonable for a reviewer to pretend that they haven’t seen the paper in question. I have no problem with that, and that’s why I have no problem with Steig asking for a copy of the final paper. Indeed, even as a reviewer, he couldn’t be sure that the final published version would be the same as the last one he saw.
After the paper has been published, though, there is no reason a reviewer shouldn’t be able to comment on it in the same way that anyone else can comment on it. The only real advantage that a reviewer has is that they’ve seen the details of the paper for a longer than others, so they can be quicker to have their arguments and comments in place.
Commenting on parts of the paper that didn’t make the final published version are obviously a no-no, but I see absolutely no reason why a reviewer can’t comment on the final published paper in the same way as anyone else – the fact that they were a reviewer of earlier versions is irrelevant and I see no obligation to reveal that fact.
Please note that I’m not commenting on the exact situation here with the changes to iridge in OD10. I quite understand Ryan O’Donnell’s anger at what happened, but I equally acknowledge that John N-G’s explanation on his blog is just as reasonable. I’m personally marking this up as a gross miscommunication problem with some faults on all side, and without trying to assign who has the largest fault (though I do have a personal opinion).
Graeme–
In this particular case, when the paper was published, the reviewer announced to the world that he had read liked the paper. Then, after “having time” he write a post criticizing the paper– repeating the views he gave in his review.
In this context, anonymity is highly misleading.
lucia, I understand all of the first paragraph, but I don’t understand what you mean by the second paragraph. The views Steig gave after “having time” are still valid views on the published paper (I’m not saying they’re correct views, only valid), and the fact that he had given the same views in a ‘anonymous’ review is irrelevant. The views themselves are either correct or incorrect (with some grey in-between). The fact that Steig was ahead of the game in preparing those views is also irrelevant.
I honestly don’t see what is ‘highly misleading’.
Yes, there’s some misdirection because he didn’t want to publicly state he was a reviewer – something that Ryan original respected, and I suspect that Ryan didn’t have any problem with that initial misdirection. The problems have all arisen because of the criticism of iridge, which are also largely in the review comments so they’re not ‘new’ criticisms, and because of the perception that Steig insisted on the change to iridge (and that’s where I side with John N-G in that I suspect there’s been a massive communication breakdown as to what was going on).
In the particular case that a reviewer ‘sabotages’ a paper, I see no reason why anonymity should be preserved. But outside of that specific case, I see no reason why someone who wants to criticize a published paper should need to reveal if they were also reviewer for that paper. The problem then shifts to what constitutes ‘sabotage’….
So Steig was either disingenuous as a Reviewer, or as a scientist blogger. I still don’t understand why he had to be disingenuous at all. If he wasn’t honest with the review, why not? If he wasn’t honest with the blog posts, why not?
There have been a lot of posts on a number of sites working around this event. I posted this comment at WUWT, but it is worth repeating in this context too.
“.. I would just like to observe that Steig appears to have won in this whole event.
After O’Donnell10 was published I noticed his very quick response on the Air Vent where he congratulated the authors on their paper and remarked on how it had backed up his conclusion in S09 that there was significant warming in West Antartica.
That was strange in itself, as it was clear that the authors considered their paper as a rebuttal of S09.
As more information has come out it becomes clear that a slick trick has been pulled.
The original paper using TTLS was a direct attack on S09 clearly showing that the use of 3 PCs produced Chladni patterns making the results completely artificial.
Due to Eric’s efforts as reviewer A, this paper will not be published.
After 3 reviews and 2 major revisions, the authors attempting to work their way through the gating procedure had felt the need to surrender this message and simply run with a clear statistical technique that was without the artificial features.
The scope of the paper was still to demonstrate that a bad statistical method had been used in S09. It was commenting on appropriate statistical tools and not focussing on Antartic warming.
But Eric has managed to derail this message completely – both by his input as a reviewer and then comments suggesting it agreed with his and then by creating an explosion by attacking the paper “disingenuously” so that all the discussion is about the evil of revealing a reviewers identity.
Moreover, the bottom line on temperature, as RyanO keeps affirming, is that the warming pattern that comes out of better statistical methods is exactly the same as that which was commonly known before S09.
That is – that in all of Antartica, only the Peninsula was showing up significant warming.
The rest of Antartica was a neutral to cooling pattern, and S09 was completely wrong in challenging this common knowledge.
But having forced changes in the original paper so that a small edge of Western Antartic is stated as warming and creating a storm afterwards, Eric really can say “mission accomplished”, even if he cops a bit of flack for being tricky with his peer review.
The message is being lost in the noise.
Eric trumpets O10 as backing his conclusions on Antartic warming, while protesting that it is still an underestimate. ie S09 is still the better conclusion!
And now he can mock the new guys as playing dirty with false accusations and betrayal of confidence.
The only way justice can come out of this is if some key “climate scientists” actually try to understand why S09 is a really bad paper that got huge public exposure and make a big public expression to acknowledge this.
Great summary. My read of the chain of events is that ES realized that his math was flawed and he could not refute the documentation of such in O10, so he went to plan B. The changed paper ended up partially confirming Antarctic warming trend but was closer to ES results. The demonstration that ES methods was simply a mathmatical artifact was no longer in the paper. ES then publishes his response that O10 confirms warming, then throws in – but under estimates the rate.
See: http://www.realclimate.org/index.php/archives/2011/02/west-antarctica-still-warming-2/
How does ES know it is an underestimation? What method shows it? Should we trust his method, or believe him because he just knows it?
Thanks for the link.
I had not realised that Eric actually boasted that peer review had got rid of the criticism of S09.
Quote “O’Donnell et al. is the peer-reviewed outcome of a series of blog posts started two years ago, mostly aimed at criticizing the 2009 paper in Nature, of which I was the lead author. As one would expect of a peer-reviewed paper, those obviously unsupportable claims found in the original blog posts are absent.”
Gosh is that disingenuous or what??
He was the reviewer who made sure they were gone!
Except . . . he didn’t.
He leaves the impression that he did, but you might want to look here:
or here:
http://rankexploits.com/musings/2011/sensitivity-test-odonnells-idea/
and judge for yourself. For perspective, at the tAV link, I included what I told Gavin was my problem with the S09 reconstruction from March 2009. Those “unsupportable claims” in the blog posts were that the S09 reconstruction spread warming from the Peninsula to West Antarctica . . . and those did, indeed, feature quite prominently in the paper.
I too agree that there is no obligation to refrain from commenting publicly on a published paper that one has peer reviewed anonymously, for if there were such an obligation, refusal to comment on the published work could be taken as evidence that one was likely among the supposedly anonymous reviewers.
However, if an anonymous peer reviewer comments publicly on an article and in so doing contradicts their anonymous appraisal, and if the contradiction becomes public knowledge, then that reviewer’s public reputation may be affected in a way that will depend on how well they are able to rationalize the discrepancy in their anonymous and public assessments.
Concerning the obligations of authors, I would say that they are free to reveal both the contents and, if known, the authorship of a peer review, unless the journal to which they have submitted makes it a condition for the consideration of an article not to do so. No journal that I know of makes this stipulation.
I think perhaps this whole issue of revealing the parties involved, the reviews and authors’ versions in the peer-review process depends on the perspective of the observer. I think complete transparency is always the best policy, but the peer-review process is guided, not by the general public, but by the participants and particularly the editors and the publishing organizations. Once the process sees the light of day with examples like the one under discussion on the blogs, I would think that those of us who do not necessarily participate in the process but are interested in the end results will clamor for more transparency.
As a very practical matter, I am not at all certain what effects the sentiment of outsiders will have on the participants’ motivations for changing to more transparency. As long as most of the information exchanged in the process is of interest mainly to the participants and effects only their livelihoods, careers and peer esteem then it is doubtful to me that anything will get changed by the opinions of “outsiders”.
Quite ironic though to me is that the more a technical field gets populated with the advocate/scientist interested in the impact of their works on public policy the more those fields are susceptible to the opinions of outsiders.
I couldn’t agree more with this post. My feeling is that the editor was allowed to ask Steig to referee. Steig was allowed to accept. The editor should have kept a much tighter rein on him as referee, though in the end did the right thing by ignoring him. In normal circs, referees are considered to be entitled to be economical with the truth about whether they were referees. The reason for that is that it keeps tensions lower in the field and enables referees to be robuster than they might otherwise be. But if that referee then goes public… All the preceding kind of becomes irrelevant. This is not just a standard refereeing situation.
What has been missing through out the discussions of Steit09 and ODonnell et al 2010 is any response from JoC and/or the editor. I believe that we may safely assume that word of the controversy has reached them. The controversy has been going on for more than enough time for them to examine how O’Donnell et al 2010 was handled during the review process and to formulate a response. I trust that CA and WUWT would be pleased to post their input. Unfortunately, I think that the controversy over Steig’s role as a reviewer and the public disclosure of this has been diverting attention away from the other important issue. O”Donnell et al 2010 has demonstrated that the statistics of Steig09 were not not in keeping with the high standards that we are entitled to expect from Nature. Steig et al should do what is appropriate in these circumstances and withdraw their paper. If they cannot bring themselves to do so then Nature should do so for them.
Couldn’t agree more. We should have heard from JoC by now about the peer review questions and from Nature about how they’re going to put right a very misleading impression, including their cover graphic, when they published Steig. None of this particular firestorm should cause anyone to lose sight of that.
Errata for first line of my previous post, “Steig” not “Steit.”
Any system that has a built-in requirement for public duplicity does not seem from the outset as all that ethical to me. When dishonesty is seen as advantageous, even honourable, by a professional body it would appear to be giving mixed messages to its members and thus an invitation to trouble.
I don’t wonder that this custom as policy is not very clearly stated as it would hardly do to broadcast that it is considered perfectly acceptable to play pretend in public. Conceptually (not literally), the phrase, “honour among thieves'” comes to mind whereby protection of the group, its aims and its behaviour, is signalled as being more important than broader social and ethical considerations.
In any event, shouldn’t the first test of ethical behavior in professional dealings be, “Would what I say/do in private be personally discomforting or embarrassing to me if it were to become public and vice versa?” I would suggest if the answer is, yes, then the person is ethically conflicted.
No organization should expect this of its members at the same time as holding them to the high ethical standards demanded by peer review.
“shouldn’t the first test of ethical behavior in professional dealings be, “Would what I say/do in private be personally discomforting or embarrassing to me if it were to become public and vice versa?” I would suggest if the answer is, yes, then the person is ethically conflicted. ”
Depends what you mean by “personally discomforting or embarrassing.” If you anonymously state that a paper you have reviewed is rubbish for sincerely held reasons, you might, nevertheless, be severely discomforted to say that in public if the author of the paper in question is chair of the committee that determines whether your own research will be funded. But I don’t see this as an ethical conflict.
It would be awfully difficult for Steig as the lead author of the 09 paper not to comment on O’Donnell’s paper. Maybe a lead author should not be a reviewer unless he agrees to the release of his name and work.
JD
I see no problem with the author of a criticized paper acting as a reviewer of the paper in which he or she is criticized. That person, after all, is well qualified and highly motivated to point out any misconception or error upon which the critique of their work is based.
Where criticism of a paper is biased or illogical, this is rarely hard to spot by a disinterested third party with some competence in the area of discussion. The real problem resulting from bias arises where an editor mechanically accepts the advice of reviewers without themselves assessing both the paper and the reviews and then making their own independent judgment.
Too often, unfortunately, scientists accept appointment to editorial boards merely to beef-up their CV, without accepting the responsibility that goes with such appointments. In such cases, bad decisions will routinely be made, whoever the reviewers may be.
Why is this blog so obsessed with whether this or that climate researcher engaged in wrong-doing? We know how you feel about Steig. It’s time to discuss the science of climate rather than the foibles of the researcher who study it.
Ah, Michael, but the foibles of the researchers color the view of much of the science.
Since so much of the debate is about statistical work, why are they no prominent “warming” statisticians who support that viewpoint? Because the people who write the papers have never bothered to ask statisticians to vet their papers prior to publication. This is a foible of the researchers. But the papers they write set the agenda for the scientific investigation, and dissenters have to provide the proof that the “scientists” have failed.
Since the dissenters have to publish their proof in “peer-reviewed” literature for it to be accepted, the dissenters are at the mercy of the editors as to whether they get published. When the referees take the advocacy route displayed in the Climategate emails or in the current kerfluffle, it becomes almost impossible for science to actually advance, as it is getting built on a bad foundation.
So recommending publication (albeit with major revisions) and providing suggestions on how to improve a manuscript is now “advocacy”? Sorry, but from what I’ve read Stieg was simply acting thoroughly in his capacity as a reviewer. This wasn’t gatekeeping, nor does it demonstrate “an agenda”. It was peer review.
joe –
Steig never recommended publication. From his third (and final) review, “In summary, this manuscript needs to be revised again, and sent again to review, before it can be considered acceptable for publication in the Journal of Climate.”
joe
There is also the question as to whether the “suggestions on how to improve” the paper strengthened or weakened its criticism of S09.
Steig certainly appears to think (in his public comments) that using iridge instead of TTLS weakens the paper. And then there is also the complete removal from the revised paper of the section on Chladni patterns which means the Steig can now completely ignore this.
Michael
The issue is and has always been credibility.
Only a fool accepts that which is claimed by someone who doesn’t have credibility.
More succinctly, fool me once, shame on you. Fool me twice, shame on me.
Michael, if papers critical of one side of the climate debate have been suppressed through various methods (as suggested by Reviewer A’s methods and as shown in ClimateGate) then there is less science to discuss. The side that keeps the other side from publishing gets to claim superiority based on peer review numbers.
BillyBob, from what you are saying it seems to me that the real issue here is making sure all sides of the argument are being fairly heard and that one side is not be unfairly suppressed. I agree with that. But let’s get on with what those arguments are rather than spending too much time on who is making them.
It seems to mea that you are suggestig one side censor itself no matter what the other does. That would be convenient for one side.
Are reviewers allowed to comment on other reviewers comments?
Some journals allow reviewers to see comments of other reviewers. In such cases, reviewers are free to notify the editor if they consider another reviewer’s comments to be misconceived. In some cases, reviewers certainly do comment on another reviewer’s appraisal.
Right on the money Steve. Steig’s behaiviour at RC was deceptive and misleading. He should have declared his position or shut up.
It was all about his ego. He couldn’t stand the thought of someone critizing his work. He had to “put it down”.
In attempting to bolster his credibility he ended up looking like a complete fool.
If the chruch members want to build a church that accomodates all, then transparency must be its foundation.
To me it’s simple:
The advantages of anonymous review are outweighed by the disadvantages.
This is especially important when world changing decisions are based on and justified by these papers. The team provide ample evidence of the dangers of the current system, it’s simply too easy to game.
In this case Steig should never have been a reviewer. Let him reply in another paper and/or his blog.
We are in the era of celebrity, hero worship etc. As your status increase, so does your paycheck.
Being on the front cover of a fashion magazine will boost a models earnings.
Getting your article front cover of a “respected” science journal is a boost to self esteem. peer respect, and earnings potential.
When some non academia person calls you out, and suggests good reasons why you are wrong, you have two choices.
1 Admit your error
2 Attack, attack, attack.
It would appear that in climate science, attack is not the best form of defence. It keeps failing. Why?
Golf Charley: please submit an example or two of a non-academia person calling out a climate researcher with good reasons why the researcher was wrong.
MM05 is near the top of the left column of this page.
Steve,
Providing advice, commentary, or in any other way, documentation to two quotations yet competing intentions and where the party providing the advice, commentary, or in any other way, documentation has a vested interest in one of the competing quotation’s and intention, full disclosure of this equity bias must be declared at all times.
There is and can never be any such thing as an ‘Anonymous’ reviewer where there is a clear and unambiguous ‘Conflict of Interest’….end of story.
Michael Klein
Ryan O’ Donnell et al 010
Steve McIntyre et al quite a few
Steig et al
Jones et al
The Hockey Stick
“It is not for me to provide you with information, if you only want to find fault with it”
paraphrase of Dr. Phil Jones, as I can’t post links.
The antarctic is burning up with heat isn’t it?
Bedtime uk
A foundation of science is truth, or the search for it.
I’m simply gobsmacked to read “Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed”
This is untrue of the scientists I know. They are generally a better lot than those who indulge in the deceptions that Steve, Ryan & co are revealing.
The matter might be different if matters of extreme national security were involved, but this kid’s stuff of climate “science” is really dragging the rest of us down.
It is quite acceptable to use “I can not confirm nor deny”. It is not acceptable to lie. It is a pure lie to say in one forum “I’ll enjoy reading your paper” while in another forum it is known that the paper has already been read.
Call a spade a spade, a lie is a lie.
Geoff
For once I disagree with you (and Steve)
If reviews are to remain anonymous, then some minor ‘dissembling’ is inevitable e.g “I will give my views when I have read the paper…” etc.
The only people who will “neither confirm nor deny” are the actual reviewers – unless every scientist gives this answer every time they are asked about reviewing a paper, and doesn’t just say “no”.
apl
You can disagree all you like. But will we believe your words in future? Don’t drop below ther bar of decency.
Worth repeating:
Dr. Richard Horton, editor of the Lancet, on peer review:
“The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability — not the validity — of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.”
Sorry, here’s the link to Henneberg’s non-peer reviewed essay on peer review in naturalSCIENCE.
We tried, incidentally, with naturalSCIENCE.com to do as Henneberg advocated: publish what we thought interesting and then provide an open forum in which articles could be critiqued.
However, the effort was a near total failure for the reason that no one much less than a Nobel Prize winner felt is worthwhile to devote their time to writing for such a publication. Thus, we had contributions from on Nobel laureate, leading proponents and critics of the AGW hypothesis including James Hansen and Fred Singer, plus several Crafoord Prize winners, including an unsolicited article by Fred Hoyle, one of the greatest of the greats of 20th Century science.
Yet when we sent Fred Hoyle’s article (carefully edited for cylindricality by exclusion of all reference to Hoyle’s belief in panspermia) to, I think it was 16, qualified commentators, not one responded for publication.
When one prospective open-forum reviewer answered my request for comment with the words “why ever did you publish the article?” I received no reply to my response, which was: “why not?”
So, yes, I think an open review process would be a good thing, but it will not emerge unless we return to a world of science dominated by independents, people like Steve, who do not live in fear of the Dean, the research funding agencies and what some anonymous rival will say when their own work goes for peer review.
To All,
The proposal that “it is necessary for you to pretend to not be a reviewer. Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed” raises serious issues under potentially applicable U.S. Federal law.
If U.S. Federal funding is involved, then before following that proposal I strongly encourage a reviewer to seek guidance from an expert on the potential application of the Federal Policy on Research Misconduct, issued in 2000 by the U.S. Office of Science and Technology, Executive Office of the President, and subsequently adopted by all relevant U.S. Federal agencies.
As the text below makes clear, that Policy specifically covers “reviewing research.” The relevant parts of that Policy, including the definitions of “Research Misconduct” “fabrication” and “falsification,” are below. The full Policy was published in the Federal Register on December 6, 2000 at Vol. 65, No. 235, pages 76260-76264, and can be found in numerous places on the web.
I offer no opinion as to whether the proposed conduct, “to pretend to not be a reviewer,” would be “fabrication” or “falsification” within the meaning of those terms as defined below. I certainly think, though, that serious consequences may arise if a Federal agency concludes the reviewer’s conduct fell within the scope of one or both of those terms and Federal funds were involved. I therefore strongly encourage anyone who contemplates employing that approach to first obtain advice from experienced legal counsel and ethics specialists.
I hope this is useful. The operative text of the Policy follows. Footnotes are at the end.
FEDERAL POLICY ON RESEARCH MISCONDUCT[1]
I. Research[2] Misconduct Defined
Research misconduct is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results.
* Fabrication is making up data or results and recording or reporting them.
* Falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record.[3]
* Plagiarism is the appropriation of another person’s ideas, processes, results, or words without giving appropriate credit.
* Research misconduct does not include honest error or differences of opinion.
II. Findings of Research Misconduct
A finding of research misconduct requires that:
* There be a significant departure from accepted practices of the relevant research community; and
* The misconduct be committed intentionally, or knowingly, or recklessly; and
* The allegation be proven by a preponderance of evidence.
[1] No rights, privileges, benefits or obligations are created or abridged by issuance of this policy alone. The creation or abridgment of rights, privileges, benefits or obligations, if any, shall occur only upon implementation of this policy by the Federal agencies.
[2] Research, as used herein, includes all basic, applied, and demonstration research in all fields of science, engineering, and mathematics. This includes, but is not limited to, research in economics, education, linguistics, medicine, psychology, social sciences, statistics, and research involving human subjects or animals.
[3]The research record is the record of data or results that embody the facts resulting from scientific inquiry, and includes, but is not limited to, research proposals, laboratory records, both physical and electronic, progress reports, abstracts, theses, oral presentations, internal reports, and journal articles.
[4] The term “research institutions” is defined to include all organizations using Federal funds for research, including, for example, colleges and universities, intramural Federal research laboratories, Federally funded research and development centers, national user facilities, industrial laboratories, or other research institutes. Independent researchers and small research institutions are covered by this policy.
[I have omitted the sections of the Policy covering Responsibilities of Federal Agencies and Research Institutions [4], Guidelines for Fair and Timely Procedures, Agency Administrative Actions, and Roles of Other Organizations.]
“If U.S. Federal funding is involved, then before following that proposal I strongly encourage a reviewer to seek guidance from an expert on the potential application of the Federal Policy on …”
Does this explain the seeming explosion in, if not bad science exactly, then trivial science?
Does not the requirement before expressing an opinion on a technical question without first seeking expert guidance on the application of Federal Policy discourage what H.L. Mencken called “one of the most useful men the human race has yet produced: the scientific investigator.”
“What actually urges him on” said Mencken, “is not some brummagem idea of Service, but a boundless, almost pathological thirst to penetrate the unknown, to uncover the secret, to find out what has not been found out before. His prototype is not the liberator releasing slaves, the good Samaritan lifting up the fallen, but a dog sniffing tremendously at an infinite series of rat-holes.”
Re: mkantor (Feb 17 21:27), I think this is much less interesting as a legal case. The words you would probably want to hang your hat on is “manipulating the process” I suggested this when the case first broke. Upon reflection, however, I think the case is more interesting for the institutional and personal ethics questions it raises than the legal. Put another way I think this puts a problem to the journals that they might consider addressing in their own fashion. It also put questions to Steig and Ryan. I think I am out of bounds drawing ethical conclusions about either man’s actions. But I can put questions to them. I cant say what I would do if I were in either man’s shoes. I know what I would consider, but I havent been in their shoes and the questions are relatively novel.
I think mkantor’s post a good one.
I have worked as a trade union representative providing representation for university members. Usually, I was raising grievances, frequently about bullying, often about discrimination, or defending a person against disciplinary action.
If asked, I would advise a university client to disregard Nielson-Gammon’s opinion about the “need to be disingenuous”. To follow it might lead one to a breach of a university policy and potential breach of contract.
I would like to observe that some of the problems of behaviour that people have commented on in climate science might be cleared up by the use of university policies and procedures. I would recommend always to pursue a complaint about individual behaviour by a university member through the university procedures. If necessary, do it collectively.
To Alfred Burdett and Steven Mosher,
I am not referring to the possible application of the Federal Policy on Research Misconduct with respect to the dispute between O’Donnell et al and Steig. I am instead referring to the proposal by Neilsen-Gammon that a reviewer commenting publicly “pretend to not be a reviewer” and that “reviewers may need to be disingenuous when talking publicly about a paper they have reviewed.”
The Federal Policy does not require “before expressing an opinion on a technical question [that one] first seeking expert guidance on the application of Federal Policy.” It may require, though, that such an opinion not be expressed through pretense or disingenuously. That type of conduct may be “fabrication” or “falsification” constituting “research misconduct” when Federal funding is involved (similar policies, by the way, are adopted by most research institutions for research matters not federally funded).
My simple recommendation is that, if Federal funding is involved, then before a reviewer follows Neilsen-Gammon’s suggestion to “pretend” or be “disingenuous,” that reviewer should seek competent legal and ethical advice. I would, of course, have similar concerns about “pretending” or being “disingenuous” even if the matter had nothing to do with federal funding, climate science, research or reviews of papers(although I might instead phrase my comment so as to urge that the person “first talk to a wiser head,” rather than employing the legalistic phrases I included in my initial comment). In the context of federally funded research, however, there is the added dimension of regulkatory requirements and legal sanctions to be carefully considered.
I hope this clarifies my earlier comment.
MK
(Contribution, slightly modified from original, as it is relevant in present discussion)
The issue is not merely whether or not Steig ought to have been a reviewer to O’Donnell et al. It is his criticism of O’Donnell et al’s results arising from a method he had a hand in, being incorporated in their paper. O’Donnell et al challenged/questioned Steig et al. For good or bad, Ryan asked Steig whether he was the reviewer and Steig affirmed it.
Imagine a situation where this asking does not take place. Imagine a fresh paper, that inadvertently challenges an existing paper of a prominent researcher. Journal editors are likely to send such papers to prominent members in the same area, and this means such papers can land on the desk of those who wrote the original paper. Such scientists, acting as anonymous reviewers can, and do extract significant changes in the paper under review, delay its publication, get their own rebuttals lined up, perform more experiments, blunt the paper’s criticism by insisting on changes in language and tone.
They then publish their own papers providing solutions to the resulting impasse, keeping their own nose ahead.
The authors sometimes never know that this has transpired, sometimes for years.
These things happen all the time in science. I am speaking for first-hand experience, having observed such phenomena. Lots of scientists just shrug their shoulders and accept all this as a part of the politics of academic publishing and the cut-throat of competition.
Eric Steig did something along similar lines – (he reserved the juiciest criticisms for himself). He did nothing ‘wrong’, but he made two tactical errors.
Firstly, Steig could have confined himself to a criticism of O’Donnell et al’s results in a more abstract sense, than declaring Steig et al’s results were superior, having himself had a hand in the methods of the former paper. I think he is well within his rights to point out limitations arising from use of this ‘new’ method (iRidge), as a scientist. But he could not have done it without coming clean that he was a reviewer of the said paper himself. Not doing so resulted in Steig’s criticism of using the new method being in the public domain, but the fact that was not unfavorable to its use, in the private domain (in his reviewer comments).
The second tactical error, of course was that Steig chose to air this criticism, publicly, in a blog. Why would he expect that his ‘adversary’ in the Antarctic temperature field not return the favor in kind – i.e., make details available publicly as well, that support his paper’s position?
Steve,
Is it fair to point out that mpaul, harold, ian, billybob, john, glacierman, sierra117, patrickm, jd ohio, rayg, j, DGH et al, are anonymously commenting on Steig as an anonymous reviewer?
Why do we prefer anonymity? I, for one, have no fear of “retribution.”
Authors, reviewers, journal, ande the process all enjoy benefits of anonymous critique. Here are some heretofore (on this blog) unexplored examples…
– The anonymous scientist reviewer neither endorses nor condemns a paper. He/she isn’t interested in being associated with a paper or a Journal but feels an obligation to participate in and contribute to the scientific process.
– The entry level author of an innovative paper prefers his/her article to be considered on its merits as opposed to being publicly associated with or criticized by a high profile reviewer. Would Einstein have welcomed Newton’s critique?
– A journal can diligently empanel highly qualified reviewers from the universe of possible experts without being criticized for selection bias.
Is the “partially reimbursed, voluntarily anonymous” process perfect? No human enterprise is flawless. That begs – is the proposed open model better, worse or only different?
dgh, the difference is that I have no conflict of interest and I have no power in determining what Steve publishes. Its a significant difference.
Steve’s post isn’t about Eric’s “conflict,” it’s about the peer review practice of protecting the identities of reviewers. He suggests that the practice should be modified if a reviewer criticizes a paper publicly.
Anonymity in and of itself is not the problem. We can have great discussions in the blogosphere without knowing each other’s identities. Indeed, anonymity allows us to avoid discounting comments because “that guy is sponsored by big oil.” Our comments must stand on their merits alone.
Acts of bad faith and the supposed need to make disingenuous comments are the issues at hand.
In this space we are snipped when our comments cross the lines established by the moderator. The editor must set and enforce similar standards for his journal. It might not be so satisfying for O’et al, to know if ES were quietly sanctioned by the editor. But they would know that the editor stood by them.
As for scientists making disingenuous public comments, it would be better to say nothing at all.
Comments here are indeed anonymous.
That means that many of us are not using our names to establish credibility, or to insert our views into the process.
Eric Steig’s name was used to establish his credibility and insert his views into the review process.
My name means nothing. It’s not part of any process of evaluating my words.
Eric Steig’s name was part of the process.
Do you see the difference?
dgh, Your anonymity analogy is not complete here. You are not electing to reveal your name publicly, but you have to some extent revealed ways to contact you, at least by the administrators of this site, in your very act of posting. To post a reply, you have to provide a valid email address. Steve usually does nothing with this, but he does know how to contact you other than replying to you here. The screen name you choose is up to you (I do use my real name and last initial, for example), revealing as much about you as you wish to reveal. Some who have blogs of their own also allow others access to their thoughts by the links provided by listing a website. All this is voluntary on the part of the posters. And if we do not wish to be known, at least by the administrative staff of this site, then the only way to do that is to not post.
In contrast, a reviewer/referee accepts a task of evaluating a paper. He can reveal himself to the person who wrote the paper in some instances, but many Journals have policies requiring anonymity, which the referee is obligated to uphold. This means his commentary should also not give away his identity, which Steig’s failed to do. The referee also can not elect to not submit a comment in order to remain unknown. His examination of the paper should be to determine its validity and worthiness, not to blunt its impact if its conclusions are accurate.
This whole confrontation looks like a ploy by ES to distract from discussion of the actual papers. How can he lose? His team will close ranks and protect him regardless. They are happy to discuss “ethics”. There are fewer numbers in “ethics”.
OH wait….
That would be nice in a perfect world. Unfortunately in the real world it’s different. Economics are more important, no wonder with the megabucks going around in climate science.
I compare it with the financial industry where distinguished academics and their universities were drafted by the banking world to compose cheerleading research that supported reckless deregulation. And they happily obliged after being healthy rewarded, financially.
Follow the money…
Reference to such research?
So the processes of science presume honesty accept when they don’t.
In some sense, I agree with John. This is partly about the priority of various rules. I still think that the institutional rules within science, including anonymity of the reviewers, should be more important than the public’s right to see into every detailed story.
In fact, I actively think that the public shouldn’t intervene into such matters. The global warming hysteria was partly created by the non-scientist’s declared interest over the detailed advances in science. This is really how the discipline got politicized. It’s populist and popular to say that the public has the right to influence everything, but the very point of science is that the public can’t have this right because the public, by definition, doesn’t respect the highest standards of the scientific method.
Lubos,
I think you are answering a different question than Steve asked. The question Steve asked is whether – in dealing with the public – it should be ethical for a scientist to be disingenuous. He is not talking about the more general case of how much the public should involve itself in the inner workings of science.
I do not think that the public has any intrinsic right to interfere with the institutional rules within science carte blanche, and I don’t think that is what Steve is getting at. He is merely proposing that – while some disingenuity may be acceptable within the scientific community to protect the institutional rules – when the discourse is no longer between scientists but between scientists and the public that different rules should apply.
Consider this from a pragmatic standpoint. The public is generally not fully aware of what these institutional rules are, and even were they to be aware, they may not agree. If a scientist wishes to accomplish the goal of simultaneously convincing the public of something while maintaining the public’s trust, then he must necessarily adapt to the public’s sense of ethics – lest he be misunderstood. He cannot force the public to adapt to his (as much as he might like to).
This is a separate question from whether the public, in the absence of direct communication from the scientist, has any right to tell the scientist what his institutional rules ought to be. I agree with you and Dr. N-G on that one. I do not believe the public to have that right.
However, I do believe the public has the right to demand a separate set of rules in the case where government institutions adopt policy based on scientific recommendations. I think the public has every right to declare the peer review process – and the institutional rules governing that process – to be insufficient for the purpose of evaluating scientific evidence for policy. The scientists need not adopt these new rules, but the government institutions do.
One facet of the disingenuousness that is quite common is to pretend that peer review is much more than it is. The public is often led to believe that peer review is an extensive audit of a study that verifies its conclusions. As we all have seen, this is not the purpose of peer review and for a number of reason exceeds its capabilities.
This is one way in which the public is led to believe things of material importance that are not true. This is being disingenuous in a way that is harmful to the public interest
“This is one way in which the public is led to believe things of material importance that are not true. This is being disingenuous in a way that is harmful to the public interest”
I think this is a very important point. When scientists send out a press release concerning a new publication they don’t usually release a disclaimer with it stating that peer review does not imply that the findings have been “audited”. I think in some, (more likely many), cases scientists are counting on the fact that the public is not really familiar with what “peer review” implies.
IN a new blog post by Nielsen-Gammon referring to Ryan O’Donnell’s experience, entitled (with the tin ear so common in climate sicence) Welcome to Peer Review, You Ignorant Little Twerp , John N-G implies that journal peer review is, in fact, a sort of audit:
Readers of Phil Jones’ peer reviews in the Climategate documents know that Phil Jones – honored by the community for his achievements in peer review – did nothing of the sort. In fact, it is the blogs that have gone through these analyses with a fine tooth claim.
Who were the reviewers that went over Steig et. al 2009 with a fine tooth comb?
Maybe that comb had some teeth missing.
I’m afraid that passing readers will see Steve’s imprecise phrasing and think I’m calling Ryan that. Far from it.
Patrick M- In the tin-ear post that Steve links to, I also say:
I agree 100% with Ryan’s remarks here – Ryan shows once again the “cultural” differences between academics and non-academics.
Ryan is saying that there is a better way of establishing the truth in the matter of a scientific question than peer-review. However, I think that to assert this proposition is to make the false assumption that peer-review is a mechanism for establishing scientific truth.
Self-evidently this is not the case. Most peer-reviewed papers contain multiple minor errors, and many are based on false assumptions, unsound inferences, or untrustworthy methods — statistical methods in the biological sciences, for example, being more often than not applied without reference to the underlying mathematical assumptions. What is more, in any active field of research, the literature is rife with contradictions, which means that even to the community of scientists, the truth is more or less profoundly uncertain.
Indeed, any experienced scientists surely realizes that the scientific literature is mostly bunk. Moreover, there are an astonishingly large number of known examples of important papers rejected for publication by peer reviewers of at least one journal.
So no, the process of peer-review does not reveal incontrovertible truth as a basis for public policy on science-related issues.
But likewise, peer-review provides no basis for establishing a productive course for future research. For both scientists and non-scientists, the evaluation of scientific evidence comes down to a matter of logical analysis and judgment and the acceptance of uncertainty.
Thus, in matters of public policy, it must be hoped that governments have competent science advisers and ministers able to grasp the essentials of the scientific debate.
Alfred,
Without meaning to be rude, you misunderstand the entire thrust of my comment. I don’t think peer review establishes scientific truth, nor do I think it has any such intent. The intent is to help prevent obvious errors from being published and provide a way to grossly sort wheat from chaff, lest our heads explode from information overload. Perhaps Bill Gates would refer to it as “Google” vs. “Bing”. Whether it actually accomplishes this is a different topic.
However, when scientists talk to the public, the public does not realize this. The public must take the words of the scientist on faith, because the public has neither the knowledge nor the time to independently evaluate what the scientist is saying. While disingenuity between scientists may not impede scientific progress, as scientists understand the rules of the interaction and scientists understand that most published literature is wrong, this does not carry over to the public.
If scientists wish to both communicate effectively and maintain the public trust, the rules regarding the interaction are different. You seem to be arguing that (a) the rules associated with peer review work well for peer review, (b) scientists understand the limitations of the system, (c) the public has no intrinsic right to change this system, and conclude (d) that scientists should be able to communicate to the public using the rules and ethics associated with that system.
The problem is, (a), (b), and (c) do not necessitate (d). In fact, given the differences in opinion between lay people (like myself) and academics (like Dr. N-G), it is immediately apparent even within this thread that (d) does not follow. You may wish it to be this way, but reality opposes you, and I see no easy path for you to alter the situation.
The public has no choice but to take scientists on faith. Same with policymakers. Advisors don’t support your cause, as they are simply middlemen, and the policymakers are still taking the advisors’ words on faith. The public does not understand a need for any kind of disingenuity when a scientist communicates to the public. Evidence of such disingenuity will make the public less likely to take future statements from scientists on faith.
You may not like this, and you may not see an easy way to fix this, but this is the reality. The public places its faith in scientists. Even the appearance of disingenuity can shake that faith.
Effective communication requires adapting your presentation to your audience, not adapting your audience to your presentation.
That is my point.
“Without meaning to be rude, you misunderstand the entire thrust of my comment.”
Ryan, I wouldn’t have thought you intended to be rude — merely that you failed to make yourself clear!
What you did say was:
“I think the public has every right to declare the peer review process – and the institutional rules governing that process – to be insufficient for the purpose of evaluating scientific evidence for policy. The scientists need not adopt these new rules, but the government institutions do.”
And my comment, which I stand by, was in response to that.
But to elaborate on it, if the public gets the idea that scientists are not using the best possible rules for “evaluating scientific evidence for policy” or any other purpose, they are surely going to begin questioning the value of putting their hard earned tax dollars into scientific research.
The fact is that what is called the “peer review process”, which is in fact, or should be, much more than simply a poll of two of three more or less qualified scientific reviewers, is about the best method we have for “for the purpose of evaluating scientific evidence for policy” or any other purpose. In particular cases it may fail badly, but considering the peer reviewed literature as a whole, it provides the best source of scientific information that we have.
And if you disagree, what do you suggest as a better basis for for determining whether, say, smoking causes cancer than by considering the peer reviewed paper by Doll and Hill (1954. “The mortality of doctors in relation to their smoking habits”. BMJ 328 (7455): 1529), and related publications?
Of course reviewers may be dishonest, editors may be bent, but that is just as unacceptable to the scientific community as to the public at large.
If I may make an aside, not much to do with this thread, it is this:
There are two kinds of papers that usually get a hard time during peer review: those that should never have been written, and which go from editorial office to editorial office until they reach an editor sufficiently careless or sufficiently desperate for copy that it is accepted; and those that say something particularly inconsistent with prevailing assumptions.
The reason for the difficulty experienced with the latter type of paper is that the scientific community tends to be both conservative and deferential to authority.
Your paper challenged a Nature cover story. It was hardly to be expected, therefore, that it would get an easy ride. So what is encouraging about your experience is that despite a tough evaluation, your paper was published.
To digress even further and probably unacceptably, I once wrote, at the behest of a co-worker, a paper that, on completion, seemed to me to comprise an entirely pointless collection of data. But it was well polished and it went through the system like a dose of salts — without, of course, having the slightest scientific impact.
On another occasion, I wrote a short note describing a method developed through several years of hard and somewhat ingenious work. It was rejected by one journal on the grounds that the method was not new, could not possibly work and was, anyway, already widely used — or words to that effect. From another journal, after waiting 9 months for the assessment, I received a further rejection. However, in that case I rebutted the critical review by an old geezer who seemed to believe I had encroached on his private domain, and supported it not with 88 pages but over 100 pages of documentation. The editor, a fine man at U of T, phoned me to say that I’d better divide the work and submit it as two separate papers, which I did, and which I was glad to find were promptly accepted.
At the time I was indignant about the effort required. But having since, as a publisher, read literally thousands of reviews, I would say that my experience and yours is not at all unusual.
Ryan,
Accepting that most published research from academia is wrong leads to a simple proposition – that the publication was made too early, before the hypothesis was adequately tested. If, sometimes, an author is floating an idea, he will use early timing and hope for useful review suggestions to take the work down fertile roads. Unfortunately, peer reviewers seldom take the time to help, amid correcting grammar and spelling. This apart, you are questioning the early publication of poorly based speculation in general, with the S09 example, which seems to be a habit in climate work.
If the author is seeking to inform the public, for reasons such as attracting funding, he will likey use generalisations and this means an even higher error rate, simply to make the topic comprehensible. I have no naswer to this except honesty.
As to anonymity, communicators would recommend that a facial image and short c.v. of at least one author goes out with the release of the paper, especially the type of paper meant for the public. The public does not have an expectation to alter the rules; it does have an expectation that a scientific statement is not tainted by an agenda that could harm the public. That is the cause of most mistrust I have found.
I make these comments with past experience wherein the release of a commercial paper (such as one describing a new ore discovery of considerable scientific and economic importance) has to be very precise and very correct, because laws cover those topics. The outcome of a disingenuous paper could well be a time locked in a small room making big rocks into little rocks. One knows that a false statement of an ore resource will be discovered when the mine fails to find the ore. Climate work needs an equivalent frightener, but that is a different subject.
“Accepting that most published research from academia is wrong…”
On what possible basis can you make such a statement? Where’s the evidence?
Do a google search for “most published research is wrong” and get 15 million hits.
“Do a google search for “most published research is wrong” and get 15 million hits.”
Funny, but wrong.
You get about 1200 hits (if you use the quote marks), including an article in the Economist (how much of what the Economist tells you is wrong? LOL) entitled:
“One group of researchers thinks headline-grabbing scientific reports are the most likely to turn out to be wrong”
There we read:
“Dr Ioannidis based his earlier argument about incorrect research partly on a study of 49 papers…”
Forty-nine papers out of two million published annually? LOL, again.
And note, the articles were “in leading journals that had been cited by more than 1,000 other scientists…”, i.e., journals such as Nature, which as I pointed out on another thread, can hardly be considered a peer reviewed journal: they pick flashy papers from the multitude submitted and do a peer review merely to cover their hind end.
So yes, perhaps the main thesis of a large proportion of papers in Nature and some other high profile journals are refuted eventually, but that’s not a generalizable conclusion.
Furthermore, such claims are essentially meaningless unless you define what you mean by “wrong.” Most published papers contain errors: probably an average of dozens, but it would be astonishing to assert that most of the basic data published — the evidence that underlies the explicatory thesis advanced — is wrong.
So if the authors’ interpretation of their results is proved incorrect, does that make the paper wrong? I don’t think so. It merely makes the authors’ hypothesis wrong.
What’s more, if hypotheses advanced in high profile journals were not frequently proved wrong, then science would not be advancing would it?
In response to Jeff Alberts:
“It would seem, then, that the Steig09 paper didn’t need rebutting, since it “can hardly be considered peer reviewed”.
You can take note of, or ignore, whatever you please. Because something has been peer-reviewed, it does not mean that it is true. Equally, because something has not been peer-reviewed it does not mean that it is false.
Science cannot establish absolute certainty about the real world, it is merely the best organized effort to understand the real world that man has as yet conceived.
Part of that effort involves a process of record-keeping which is largely, though by no means exclusively, the function of the journals.
What the journals record are observations and inferences of scientists that are deemed to be reasonable and interesting by qualified reviewers and editors.
Because it is expected that authors of journal articles will relate their work to that of others, e.g., with an introductory literature review, the scientific literature amounts to a rather carefully conducted conversation within the community of qualified participants. Ideas are advanced, supported or rejected and sometimes rejected and then revived, in the assumption that over the course of time the general matrix of ideas within which science advances approximates more and more closely to reality. But about that, no one can be absolutely sure.
Dig here
http://www.newscientist.com/article/dn7915
For a detailed response: see my reply to freezedried.
As in the case of the Economist, New Scientist fails to state explicitly what they mean by “wrong,” but it is clear from the discussion that they do not mean the basic data. So again, we’re talking about erroneous hypotheses or statistical inferences, and I don’t think that mistaken inferences or a subsequently falsified hypothesis make a paper wrong, for by that standard you’d have to say that Newton’s Principia was wrong because Newton failed to anticipate the theory of relativity.
More likely, the paper asserting that most papers are wrong was, by it’s own definition, wrong, although even so, the data on which it arrived at that wrong conclusion were most likely correct, at least more or less correct.
I agree with Artifex Feb 17, 2011 at 2:26 PM
There is neither need nor justification for disingenuousness. If asked directly you should say “I never make any comment about the identities of reviewers”.
Once disingenuousness is accepted, or worse encouraged, the camel of dishonesty has its nose inside the tent of science.
John Nielsen-Gammon’s claim of an expectation by scientists to be disingenuous wrt reviewer status is very close to an implied duty to be disingenuous. I can’t imagine that academics are formally trained in how to be most effectively disingenuous, but some people might have no natural talent for it. Do the grad students with the least talent in this area make for poorer candidates as future scientists? I sure hope not.
Apparently bore hole results http://europa.agu.org/?view=article&uri=/journals/gl/gl0902/2008GL036369/2008GL036369.xml&t=gl,barrett
reinforce Steig, according to The Carbon Brief http://www.carbonbrief.org/blog/2011/02/steig-et-al .
I am wondering if Jeff ID, O’Donnel, et al have compared the bore hole results to their result. Would their result correlate to the bore hole result better than Steig’s?
Steve: Nic requested the borehole data – it was refused. BTW there are major issues in the inversion of borehole results pertaining (interestingly) to regularization. See some back posts on this.
4 Trackbacks
[…] N-G: “Reviewers may need to be disingenuous” John Nielsen-Gammon writes as follows in a comment in the preceding post: If you are a reviewer and wish to remain […] […]
[…] Source: https://climateaudit.org/2011/02/17/n-g-reviewers-may-need-to-be-disingenuous/ […]
[…] Climate Audit by Steve McIntyre Skip to content Hockey Stick StudiesStatistics and RContact Steve McProxy DataCA blog setupFAQ 2005Station DataHigh-Resolution Ocean SedimentsSubscribe to CAEconometric ReferencesBlog Rules and Road MapGridded DataTip JarAboutCA Assistant « N-G: “Reviewers may need to be disingenuous” […]
[…] John Nielsen-Gammon has articulated a doctrine of what might be termed justified disingenuousness as applying to climate scientists acting as reviewers. I criticized this doctrine in yesterday’s post. […]