Over the past week, there has been considerable controversy over the outing of Eric Steig as Reviewer A, together with the publication of the voluminous review comments and review responses for O’Donnell et al 2010. John Nielsen-Gammon has been a sharp critic of Ryan’s decision (see here and elsewhere.)
The history of the affair is quite lengthy and will not be reviewed in this post. (See here for example.) Two very distinct issues arise in regard to the Steig situation:
1. do originating authors have a duty of confidentiality under journal policies or otherwise to maintain review comments received from journals in confidence? And, if so, where is this duty set out and what are the limits of the obligation?
2. did O’Donnell have a duty of confidentiality arising out of his correspondence with Eric Steig in December? And, if so, did Steig’s conduct breach whatever understanding he had with O’Donnell sufficiently to justify O’Donnell’s backing off his December promise to Steig? Potential examples of such conduct would be blog comments that suggested to third party readers that he was not Reviewer A (e.g. the comments that gave rise to Lucia characterizing Steig as the Rod Blagojevich of Science and Steig the Shameless) or by re-litigating issues that he had raised as Reviewer A in his Feb 1 realclimate post without disclosing that he had been Reviewer A and that he had previously raised these issues as Reviewer A (and that he had been unsuccessful in convincing the “editor and knuckleheaded reviewers” of his point.)
Over and above these issues, the validity anonymous peer review as a component in a quality control process is something that, to say the least, puzzles people from an engineering or commercial background, where signed reviews are typically mandatory.
I would like to focus on journal policy for a couple of reasons. First, at this point, none of us (including me) know the full backstory of the correspondence between Ryan and Steig; none of us (including me) are in possession of this correspondence. But more importantly, there are a lot of particular incidents in the Steig-O’Donnell correspondence that are not likely to recur in other situations, whereas journal policies govern every article and lessons learned about journal policies may have applicability in other cases.
Duties of confidentiality arise all the time in business and legal situations and are not necessarily symmetric (as academics too often assume). For example, if a lawyer gives legal advice to a client, the lawyer has a duty of confidentiality to the client not to disclose the content of the advice. The obligation is not symmetric. While the client has the right to keep the advice confidentiality, he has no obligation to the lawyer to keep the advice confidential.
In ordinary business, contracts frequently contain confidentiality clauses. They are common in the mineral exploration business and one of the most important mining cases (Lac-Corona) turned on a confidentiality agreement. Anyone doing mineral exploration business has to have some understanding of these clauses and they are something that I understand. Confidentiality clauses, amusingly, were involved in the CRU FOI incident (see CA posts for the long backstory).
I’ve wondered for some time about whether authors have a duty to the journal to keep review comments confidential and how this duty, if any, arises. In early December just after the announcement of the acceptance of O’Donnell et al 2010, Jeff Id forwarded me an email from John Nielsen-Gammon. This led to a short correspondence with Nielsen-Gammon between December 5 and 8, in which I asked him whether policies of the American Meteorological Society required submitting authors to maintain confidentiality on review comments. I was already then contemplating placing the review record online and, before doing so, wanted to determine whether this would breach any journal policies. (Ryan was not a party to this correspondence.) AMS policy for reviewers is online here.
Nielsen-Gammon volunteered to ask the responsible people at AMS and came back with an answer that may be surprising to many academics: that there was no policy nor formal objection on the part of AMS to making review comments public, providing that the publication redacted the name of the reviewer (if that had been provided in the review comment, as sometimes happens). N-G’s letter began as follows (as per his blog):
Jeff & Steve (with copies to AMS publications leadership) –
What I told you about making reviews publicly available is correct. There’s no AMS policy against, nor any formal objection to, an author making the contents of anonymous reviews and responses public. If a reviewer provides his or her name, or if there is other information that makes it possible to discern the identity of the reviewer, such information should be redacted unless the reviewer grants permission.
In our particular case, none of the reviewers had signed or included other identifying information in their reviews; accordingly, within the AMS policy, there was nothing within the reviews that needed redaction under the AMS policy. So far, so good.
This paragraph was followed by another paragraph – the second paragraph – below, and it is this paragraph that has occasioned N-G’s criticism of recent actions:
What I told you about making reviews publicly available is correct. There’s no AMS policy against, nor any formal objection to, an author making the contents of anonymous reviews and responses public. If a reviewer provides his or her name, or if there is other information that makes it possible to discern the identity of the reviewer, such information should be redacted unless the reviewer grants permission.
In the context of this, I would think that publishing an anonymous review and speculating as to the identity of the reviewer would be unethical. The author, if making the review public, has a duty to preserve the anonymity of the reviewer.
Note that the first paragraph is a synopsis of AMS policy as understood by John N-G. As someone who’s perhaps over-used to reading interpretation bulletins, I understood his second paragraph (which begins “in the context of this”) as a sort of interpretation bulletin of AMS policy, providing his interpretation of the last sentence of the first paragraph, a sentence which discusses the redaction of reviewer names (if they were included.) I understood N-G to be saying that authors, in possession of redacted information, could not evade the intent of the redaction by concurrently publishing speculations as to the identity of the author.
Contrary to Nielsen-Gammon’s apparent belief, this is not what happened in the case at hand.
The reviewers did not name themselves within the four corners of the O’Donnell review comments. The review comments did not require redaction. Nor did Ryan (or anyone else) use redacted information to speculate on the identity of Reviewer A.
Ryan’s post in controversy did not, in fact, speculate on the identity of Reviewer A. Ryan positively identified Reviewer A, who, by that time, had identified himself to Ryan. Ryan’s knowledge of Reviewer A’s identity did not depend on redacted information in the review comments or from information in the review comments, but from Steig voluntarily identifying himself as Reviewer A in then friendly correspondence with Ryan.
In my opinion, Neilsen-Gammon has failed to show any breach of journal policy. On the contrary, if anything, the controversy has shown very clearly that there is no AMS journal policy prohibiting authors from publishing review comments subject to the redaction requirements discussed above.
As noted above, I recognize that there are a number of issues arising out of the correspondence between Steig and O’Donnell ( see point (2) at the start of my post.) However, in today’s post, I do not have time to survey these issues and ask readers to defer discussion of these matters until I review them, which I hope to do in the relatively near future, and to restrict their comments to issues directly pertaining to journal policy and any breach of AMS journal policy actually occurred.
Climate Audit 
297 Comments
Making the reviewer’s comments public would be a breach of the ethics policy of some journals, for example http://jeb.biologists.org/journal/publishing_ethics.dtl#Confidentiality . Other journals may not have explicit guidance on this, relying instead on authors following established norms and practice.
If O’Donnell made a promise of confidentiality then he is honour-bound to follow it.
Steve: I asked readers to defer discussion of the Steig-O’Donnell correspondence until the circumstances are reviewed. The issue is whether Steig’s conduct voided whatever understanding that he had with O’Donnell in early December, as O’Donnell believes. But please wait until the issues are reviewed.
Re: richard telford (Feb 15 16:12), It’s not entirely clear that Ryan is honor bound. I’ll raise a case that I had to consult on recently. The conversation went like this.
A: Can I tell you something in confidence?
B: Yes of course.
A: You know your patient X?
B: Yes.
A: I slept with her.
B: You realize she’s schizophrenic
A: Yes I feel terrible, don’t turn me in.
Now, is B “honor bound” to A? Certainly, B gave A their word. In fact, when B brought this case to me her main concern was that she not be seen as a “rat”. Her concern was for herself, and not for the mentally ill
patient who had been violated. So, B faced a dilemma. In the end B did the right thing and turned the matter over to the authorities who then decided what to do. In my eyes her decision to break the promise increased her personal honor.
To get to the bottom of these cases one wants to ask the following questions.
Dr. Steig; why did you fell it necessary to divulge your identity? Doesnt that act put you in the dubious position of being able to publicly disagree with things you said privately? Did you think of that?
Would it be ethical for Ryan to ask you to comment on reviewer A? That is, if Ryan and only Ryan knew you were reviewer A, which would be more ethical:
1. Reveal you are reviewer A and have it all out in the open.
2. Keep the fact secret and drive you to either agree with or repudiate ‘reviewer A’?
Do your public denials of being reviewer A put you and Ryan in an untenable ethical situation?
that is, if he sees you saying you are not reviewer A in public and privately saying you are, how does he decide which is true?
And we would have questions for Ryan as well. You gave your word to Steig. Did you expect him to publish public comments at odds with his private comments? What higher value do you appeal to to justify your actions?
Without both people willing to submit to questioning you really can’t get to the ethical bottom of this.
To understand that Ryan is not ethically bound to keep his promise you merely need to note this.
“Can you ever break a promise?” is an open question. our moral intuitions on matters like this differ sharply.
However, if you want to hold that all promises should always be kept, we can probably go through the climategate mails yet again.
In the end, worst case. Ryan broke his word. What’s the sanction? Well, people who want to share stuff with him that they shouldnt be sharing will likely not disclose such information to him. if you have some dirt, especially dirt on yourself, pick a different priest.
I think it’s more like self-defense against initiation of force.
Steig tried to damage O’Donnell’s reputation by publicly attacking O’Donnell’s public actions that were undertaken due to covert actions of Steig. O’Donnell’s repudiation of his prior agreement with Steig to keep Steigs involvement secret is just pure self-defense. It’s justified.
I noted that obligations of confidentiality can agree by agreement e.g. an implied agreement from submitting to a journal with an express policy stating that review comments not be published. The journal linked by Richard Telford has such a policy, linked here:
If we were discussing reviews obtained from JEB, then this policy would be relevant. But we’re not.
We’re dealing here with an AMS journal, where Nielsen-Gammon informed us, after talking with responsible AMS officials:
It’s really logically simple. Steig revealed his name because he saw a possible gain, unspecified.
From that point, the formalism changed. Ryan was no longer bound by the quoted portions of the AMS policy because confidentiality no longer existed.
The general questions for publishers should also include, “Should an anonymous reviewer have a duty NOT to disclose his/her identity to anyone?”
BTW, I agree with the principles of Steven Mosher’s conversational example.
Since Ryan asked Steig if he was Reviewer A, Steig would either have to lie and say no, refuse to answer (which would be interpreted as an affirmative), or answer truthfully (which is what he did). Steig didn’t volunteer the information, but once he did respond affirmatively, he really should have taken care in what was said afterwards.
Not necessarily. He can answer (as I was trained to do about clasified info): ‘I can neither confirm or deny any speculation about that’. If that answer is stated as a matter of policy, it reveals nothing.
Sorry, I messed up closing tags on https://climateaudit.org/2011/02/15/are-review-comments-confidential/#comment-255351: para’s 3, 4, 5 as well as 7, 8 & 9 in above block are my words, not Ryan’s 😦
In this context, if you are not Reviewer A, why would you even think of caveating your response with “I can neither confirm or deny any speculation about that”. No-one says that in normal discourse. It’s like saying “I refuse to answer on the grounds that my answer may serve to incriminate me”.
Did you try to shoot the President? I can neither confirm nor deny…
Did you wuss out and not try to shoot the President? I can neither confirm nor deny…
Did you eat my last Rollo? I can neither confirm nor deny…
It’s like admitting guilt with “without prejudice” at the top of the page.
A lawyer may say that’s not admitting anything, everyone else says that’s bulls**t.
I’m not sure that Ryan actually “asked” Steig. Rather, he presented his surmisal (as well as his reasons). Excerpt from portion of E-mail that Ryan had posted at Lucia’s (Comment #68428):
[Ryan Dec. 6:]
Thanks for laying this out. I’ve thought it really curious that some people asserted that Steig could volunteer the information and then demand confidentiality.
What are the policies that cover the obligations of the reviewers? Much is being made of what RO’s obligations are/were, but what about ES?
I do not think being an anonymous reviewer means you can actively use that status to shape public opinion behind a cloke of protection. Reviewer status does not give one a free pass on their ethical obligations.
Thanks to Steve’s link below, I have at least a partial answer to my questions. http://www.ametsoc.org/pubs/authorsguide/authorseditorsreviewersguide.pdf
Any more input would be appreciated.
The following items jump out at me:
“5. A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.”
“7. A reviewer should treat a manuscript sent for review as a confidential document. It should neither be shown to nor discussed with others except, in special cases, to persons from whom specific advice may be sought; in that event, the identities of those consulted should be disclosed to the editor.”
And of particular interest is this regarding Editors Obligations:
“2. An editor must protect the confidentiality of all reviewers unless the reviewer reveals their identity to the author.”
and
“9. Editors should avoid situations of real or perceived conflicts of interest.”
I agree with Steve’s comments about confidentiality in the business realm as well. I didn’t have any problem reviewing papers authored out of NIST when NIST people were organizing it. Everyone knew who I was, I knew who they were, everyone expected things to be kept professional and it was.
It’s interesting to read the discussion of actual AMS policies. Eli wrote a post in which he posted some speculation about RyanO’s sharing the identity of the reviewer with you “immediately”. I thought he was suggesting I clear that up– and I you know I contacted you, jeff and ryan to clear that up. But it turned out Eli also wanted me to comment on JNG’s interpretation of AMS policy in context of Ryan publishing the reviews and naming Eric.
My view on all of this is: If JNG is describing AMS policy, then that’s AMS policy. AMS will then, presumably pursue any issues and do whatever they do in the event of breaches of policy. JNG is more likely to know AMS policy than I and it appears he did discuss it with highly placed people in AMS.
As I had no independent knowledge of AMS policy, and everyone knows Ryan published– it’s no secret. RyanO certainly doesn’t seem the type to conceal anything!
I was curious enough to spend a small amount of time at the AMS web site trying to hunt down any specific document discussing the precise obligations vis-a-vis confidentiality. They may well be there, but I couldn’t find them easily. (Can’t say I tried too hard.)
Is there a link and a check box from the pages used by authors to submit their articles? If someone has a link to any write up of the confidentiality agreements binding authors and reviewers, that would be interesting for us all to read.
Lucia,
I was unable to find anything specifically with regards to that myself when I looked (as a result of Luis Diaz’ concerns that AMS had a policy against releasing reviews). The full AMS authors’ guide is apparently being rewritten – so all that is up are the summaries and a page that says when the revised guide is finished it will be posted.
However, I do have the old AMS guide, which was online at the time we began the submission process. I have uploaded it to the same directory where the reviews are (AMS_authguide.pdf). I was unable to find anything in that guide concerning these issues.
Also relevant is the AMS policy for reviewers online here.
Under the “reviewers” section of the AMS policy
5. A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.
It seems like most of the implied culpability on the part of RO arises not from the actual AMS policy, but instead what N-G further feels it should also say (regardless as to whether that charge is landable).
This may seem Clintonian, but it looks to me like there are plenty of outs on semantics alone. The AMS just might be revising their policies in the near future.
Salamano–
Well, traditions and cultural expectations do exist. But there are difficulties adjudicating breeches when the breech is of a tradition or cultural expectation. This is particularly so people from other cultures or traditions breech and there has been no clear communication about the traditions.
It seems to me that when SteveMc emailed JNG with a question, JNG tried to determine a) the written policy and b) infer the cultural expectations. He communicated those. That’s really all anyone could expect of him.
Is there not a movement afoot to get RO into some trouble with the AMS because of a possible breach of ‘inferred cultural expectations’..?
I could have sworn I saw many folks taking up their torches and pitchforks.
Being in trouble with JNG’s expectations isn’t necessarily criminal, is it?
I suspect the journal policy WRT disclosure of reviewer identities will be more explicitly defined in the near future.
From JNG’s site, there is a later post Steig this is not complicated, in which I posted this:
Steve: I understand why you might think that Phil Jones’ conduct was unethical but I am dumbfounded that anyone would take the position that I am not entitled to speculate that he was the reviewer in question. In the material received from the journal, Jones did not identify himself. Nor was any redacted material from the review comments used in my post. Get real.
>In the context of this, I would think that publishing an anonymous review and speculating as to the identity of the reviewer would be unethical.
I’m having trouble squaring that with your statement
I am dumbfounded that anyone would take the position that I am not entitled to speculate that he was the reviewer in question.
JNG appears to have taken that position, though he declined to say so explicitly when I asked.
It’s clear to everyone but yourself that JNG refused your offer to take a position; equally clear is that his previous remarks have nothing to do with Steve’s situation. Rather than trying to get him to “appear” to say what you want, how about you read what he actually said?
For instance:
“If a reviewer provides his or her name, or if there is other information that makes it possible to discern the identity of the reviewer, such information should be redacted unless the reviewer grants permission.
IN THE CONTEXT OF THIS (..emph added..), I would think that publishing an anonymous review and speculating as to the identity of the reviewer would be unethical.”
Its interesting that breach of policy is brought up with an obstructionist reviewer getting outed. It’s so unlike when someone is required to archive their data with the journal, that request to comply with policy seems to get ignored more often.
I guess that’s the message then? Don’t mess with the bank of “SME” reviewers, but don’t call anyone out to archive data…
It may also be interesting to note that Steig’s revelation of authorship wasn’t one. This is from a 4/26 letter the group sent back after the first review:
my bold.
I disagree with your interpretation here. Just because there is no material redacted, doesn’t mean that you can hypothesize about the identity of a reviewer. The guidelines are built on the idea of reviewer confidentiality.
In the current context, Eric Steig identified himself as Reviewer A. If he had merely signed his name to the reviews, then this info has to be redacted, but because he relayed it personally, that doesn’t mean it is OK to publish it.
Steve: once again, the journal policy was that authors were not entitled to engage in speculation using information on author identity that they had received from the journal e.g. named review comments.
In the Steig case, Steig told Ryan that he was Reviewer A outside the journal review process without any prior agreement from Ryan to maintain confidentiality. The subsequent dispute is between Steig and Ryan as whether a promise to the “Rod Blagojevich of Science” is binding under the circumstances.
Take a look at the AMS guidelines for reviewers and then consider Steig’s behaviour instead of O’Donnels’. The guidelines also require reviewer impartiality – even the “appearance” of a conflict interest should be avoided by reviewers. In particular look at items 5 and 6. Steig was, without any question whatsoever, in violation of Item 5. Six is especially interesting in the light of the so-called “Climategate” emails. Item 2, under Editor’s Responsibilities with respect to reviewer confidentiality explicitly absolves editors of responsibility to protect reviewer confidentiality if the reviewer discloses their own identity. Steig revealed himself to Ryan O’Donnel. In short, the AMS no longer has a dog in the dispute and lean back and wash their hands of it. The real issue is the ethics of Steig baiting Ryan in the assumption that Ryan could not fully respond having “given his word” vs. Ryan fully outing Steig and his reviewer ethics. In the 19th century this would have lead to a “pistols for two and coffee for one” situation.
MikeN:
If I sign my name to a review , I expect it to get publicly shared. (Never hurts to get permission though.)
JNG appears to have some “cultural expectations” that are somewhat at odds with “standard practice” in any field that I’ve worked in.
To all:
I eliminated the huge indent on nested comments (a minor variation of the site theme, found while browing on an ipod 🙂 )
Please yell if there is a problem on some browser somewhere. Apparently you may not immediately see the change.
Excellent!
Thanx.
Huzzah, Huzzah! Now that we know to look for the small indent, there should be no problem.
This is a really great improvement!
I use Firefox and NoSquint to make text easily readable.
{Cntrl +} is my big friend.
But it also tends to shrink the width of the central column, so the big indents led to one or two words per line on multi-indented threads.
This new skinny indent is fantastic!
BIG THANKS to Mr. Pete!
RR
Requiring that anonymous reviews be kept confidential by an author makes no sense whatsoever and could be counterproductive to doing good research..
On numerous occasions, an author might wish to consult other academics who are not current co-authors (either individually or in a seminar) on how best to solve scientific issues raised by a reviewer. In such cases, allowing the consultee to actually read the review (as opposed to paraphrasing the content) could be necessary to ensure they would properly understand what was being said, particularly if the content was lengthy.
I notice that in the link given by richard telford for The Journal of Experimental Biology, it says
I would sincerely hope that the word published would not preclude the use of a review in the manner that I just mentioned above.
Seems to me The Journal of Experimental Biology needs to move rapidly into the 21stC else it becomes yet another casualty of the digital age.
As Wikileaks has convincingly shown transparency is everything.
“As Wikileaks has convincingly shown transparency is everything”
I think you’ll find that was Pilkington Glass Dave
Boro,
Unfortunately, I think Pilkington Glass went bust several years ago so there is no more transparency from them.
EMBO J (a prominent Biology journal) and several others are moving to a system in which all reviewer comments and author responses will be available online once the paper is accepted.
IMO this is a small, important step in improving peer review, which is currently in an abysmal state.
If a reviewer is acting in good faith, there is no reason that the review comments should not be freely available.
Cark,
I agree completely.
I am currently preparing an article on this issue and will publish it quite shortly.
There are advantages pro and con (are there not always in every situation?)
But the issue of one sided control of publishing needs a radical overhaul.
Open access is part of the solution.
There is a vital need to return to good practce in all scientific endevours.
What if 1) a reviewer himself puts his review on, say, his own website after the paper has been published, and 2) an author puts the review on his website?
Steve: Different situations entirely. As I said in my post, duties of confidentiality are not necessarily symmetric. Most journals specify that a reviewer has an obligation to maintain confidentiality. My question pertained not to the reviewer obligation (which seems to exist), but to the author obligation.
Case 1 should be forbidden. The review may well contain a discussion of material which the author may have decided during the review process to postpone to another paper. Disclosure of this material would then compromise the rights of the author.
Case 2 differs in that an anonymous reviewer should not be disclosing items which are of a similar nature. A reviewer who has identified themselves would have greater privileges in the suppression of reviews.
Re Roman’s Case 1:
Ironically, Climate Audit’s genesis arises in part because of breach of confidentiality by Mann in 2004 that Nature ignored and which no climate scientist has ever spoken against. Our 2004 submission to Nature was provided to Mann for the exclusive purpose of submitting a response to Nature and for no other purpose. Mann’s Reply to our article was published online by Stephen Schneider at his website while our article was still under review. We complained in no uncertain terms to Nature to no avail. Mann then gloated over the rejection of our article – a rejection that, in large part, originated from Nature soliciting a third review from Phil Jones, after the initial reviews were positive (revise-and-resubmit). Mann gloated over the rejection, again a breach of confidentiality.
The contents of Mann’s response to our Nature submission were published at realclimate in December 2004, in what appeared to be a preemptive strike against our pending GRL article, where there had been a temporary “leak” in Team control of climate journals – a leak later repaired to Team satisfaction according to Climategate emails.
To my knowledge, no climate scientist or other academic has ever spoken out against Mann’s preemptive use of material sent to him by Nature for the exclusive purpose of submitting a reply to Nature.
I was referring to situations where the paper was already published.
To SteveMc: The “leak” incident is of major relevance now. I would love to read more about it. Especially, updates or speculations on how your work was sabotaged and by whom. Cheers.
I usually deal with Non-Disclosure Agreements (NDAs). Under NDA, you’re generally free to publish covered material once the discloser has made it public.
For Case 1, the reviewer would generally be clear to publish any review comments on the portion of the original paper that made it to publication. If some major portion was removed, that should still be covered by NDA.
If there’s a question, talk to the discloser and/or a lawyer.
Steve, with all due respect, I don’t think your analysis of N-G’s statement bears up. Specifically, he said:
The fact that Steig’s name did not show up in the four corners of the document does not mean that it can now be made public at will. You seem to be focusing on the fact that no redaction of the review was required, and saying that, because of that, all bets are off because N-G’s statement no longer applies. There are several problems with that conclusion.
First, N-G’s statement is not a binary true-false computer program decision node that negates everything that comes after the false result, but is rather a general description of policy, and I don’t see in the paragraphs you quoted any reason to assume that is what N-G meant. Second, and more directly, N-G’s point sppears to be that if the name shows up *or* can be ascertained by other means, then the name should not be disclosed publicly (yes, he uses the term “redacted” because he is focusing on the question he was asked, namely, can the review be made public). It doesn’t work to say, well there was no name in the four corners of the review, therefore no redaction required, therefore no limit on disclosure. Third, several have noted that there is other information in the review that might make one suspect that Steig was Reviewer A. According to N-G, (depending on exactly how obvious such information is) such information should have been redacted. Finally, what N-G *does not* say in his reply is that if the name is not included in the review but you find out some other way then you can go ahead and publicly disclose the name. That is one possible (I think erroneous) interpretation, but N-G clearly is not the one saying it.
The more salient question is whether Ryan’s correspondence and/or Steig’s public identification of himself as Reviewer A override the general obligation not to disclose the name that N-G outlined. As you said, we’ll wait for a further post on that point.
Steve: the other information only caused us to hypothesize that Steig was Reviewer A; sceptics could argue that the information was insufficient to determine that Steig was Reviewer A. It could, for example, have been another coauthor – Rutherford, Schneider, Mann – or an associate – – Gavin Schmidt or someone we hadn’t thought of.
Steve:
Keeping this thread on topic is going to be a challenge.
I see OIMC2010 as a special class of articles that specifically criticize an existing article. Journal policies need to coherently and reasonably cover this type of situation including granting of anonymity to reviewers with obvious CoIs. More generally they need to lay out the recourse authors have when a reviewer essentially goes public with the substance of their anonymous reviews without acknowledging that they were in fact a reviewer. It seems to me that the previously anonymous reviewer has now acted in a way that they can no longer expect anonymity. My understanding is that even lawyers, as officers of the court, cannot maintain the confidentiality of their clients if the lawyer knows that the client is ongoingly engaged in criminal activities including offering perjured testimony.
Let us not forget that this whole issue was triggered by Steig’s post. Steig poked a hornet’s nest and has been stung for his temerity. Tough cookies.
Steve: yep, that is the issue, though academics don’t want to discuss it. If Steig had forthrightly disclosed that he was Reviewer A, rather than making coy comments at blogs suggesting the opposite, then Lucia wouldn’t have called him the Rod Blagojevich of Science.
Not to mention making what appear to me to be coy statements in blog posts at RC.
I think that’s OLMC2010 — Ryan O’Donnell (Ryan O), Nic Lewis (NicL), Steve McIntyre and Jeff Condon (“Jeff Id”)
Absolutely – my apologies to Nic and Jeff
From the ICMJE, whose rules are binding on many journals and regarded as good practice by many others.
“Reviewer comments should not be published or otherwise publicized without permission of the reviewer, author, and editor.”
http://www.icmje.org/ethical_5privacy.html
To answer RomanM, this would not as far as I can see preclude showing the review, to third parties when asking for advice as this would neither be publication or publicizing the review.
Steve: once again, Nielsen-Gammon told us that AMS journals had a different policy after consulting with responsible officials.
You asked “Are Review Comments Confidential?” I have shown there is a substantial body of opinion that they are.
With regard to AMS, it would appear that they didn’t have a different policy, but rather they didn’t have a policy.
“There’s no AMS policy against, nor any formal objection to, an author making the contents of anonymous reviews and responses public”
Many journals have rather little in the way of written policy, depending on the community to follow established norms. I am sure that you would have been the first to complain if they had developed a policy of confidentiality is response to your request.
Steve: once again, the reason why I asked was to determine whether there was a policy or objection to making the review comments public. I was told that there wasn’t. It was open to say something different, but they didn’t.
I didn’t realise that the question you posed at the top of this page was meant to be rhetorical.
I took the question as directed at the Journal of Climate/AMS policies.
Re: richard telford (Feb 15 18:19), I took it to be an open question and it is. Had it been rhetorical only idiots would have answered it.
Richard Telford observes that ICMJE rules are “binding on many journals and regarded as good practice by many others”, citing their policy on privacy in connection with review comments.
http://www.icmje.org/ethical_5privacy.html
Richard omitted an analysis of their conflicts policy http://www.icmje.org/ethical_4conflicts.html which states that conflict exists when a reviewer has a potential bias. Its policy, like AMS, states that reviewers are bound to “disclose to editors any conflicts of interest that could bias their opinions of the manuscript”. But it goes on to say that reviewers “should recuse themselves from reviewing specific manuscripts if the potential for bias exists”.
If Richard’s view is that it is relevant to assess the events under ICMJE policies (and, to the extent that this represents good practices, this may well be a useful exercise), then one shouldn’t pick and choose among ICMJE policies. The controversy over the confidentiality of reviews would never have arisen if Steig had adhered to ICMJE conflict policy.
Question;
Do the purported comments posted by Steig at the AirVent after each of his reviews, constitute a breach of some policy or other?
I can’t find them, but they seem to be an important piece of the puzzle here.
I regret asking this on multiple threads, but it seems to be highly germanium to our porpoises of clarity…
RR
Ryan O has clarified this matter over at Lucia’s;
http://rankexploits.com/musings/2011/central-england-temperature/#comment-69881
Steig’s temporal pattern of posting (1:1 with revisions of O’10) was deemed to be a signal of his awareness of the paper, at least by the 4th instance.
I don’t detect a coherent theme to Steig’s posts, although I might fail that test also. No interest in interacting with replies to his post, even a prompt (within 2 minutes) apologetic reply from JeffId. Why bother to make these posts?
Anyway, thanks for the interesting, free-ranging discussions. I’ll tolerate the trolls to have an open exchange forum.
RR
Are you imagining that the editor at AMS would be so fantastically inept not to realise that Steig had a potential conflict of interest in reviewing O’Donnell et al? And Steig could hardly recluse himself after being specifically asked by an editor well aware of his situation. Indeed he should no, as his review could be very valuable if considered with caution by the editor.
It is not an uncommon practice to for the editor ask an author of a paper being criticised to review the criticism because they know the material better than most other people and have an incentive to do a thorough review. This review would often be to supplement one or two regular reviews, and the editor would be discount it. I have a manuscript currently being reviewed by the author whose paper I criticise. He is not delighted, and I fully expect him to do his worst, but there is no-one whom I would rather have as a reviewer – provided the editor treats his review with due caution – as if there are any faults in the manuscript he will be sure to find them.
The ICMJE rules on conflicts of interest are to deal with conflicts that the editor is not aware of.
I think you are confusing 2 different concepts here.
1. Appearance of Impropriety
2. Impropriety
The first is to not get into a situation that you could even be questioned that you did something wrong. The second allows the question, but you better have a good answer.
In the ICMJE rules, we can see 4 reviewer situations.
1. stated conflict of interest, no potential bias (looks like a conflict but can still be objective)
2. stated conflict of interest, potential bias (looks like a conflict but may not be objective)
3. no stated conflict, no potential bias (does not look like a conflict and can be objective)
4. no stated conflict, potential bias (does not look like a conflict but cannot be objective)
Quite clearly the ICMJE says the reviewer should disclose any and all conflicts of interest HOWEVER if the reviewer themselves thinks they have the potential to be bias, they should not review.
So the ICMJE reviewer rules say you can be a reviewer if you have 1 or 3. You cannot be a reviewer if you have 2 or 4. It is not as you state to “deal with conflicts that the editor is not aware of” (ie. only #4) when the ICMJE rules says “Reviewers must therefore also be asked to state explicitly whether conflicts do or do not exist.”
Lets look at that again… If the reviewer thinks they have the mere potential to be bias, then they should not agree to be a reviewer. There is nothing in the ICMJE rules that allows the editor to overcome the potential bias because according to the ICMJE rules, it is not up to the editor. That is it. You could have stated every conflict you have, (which is what the rules say you must do) if you have the potential to actually be bias, you should not review.
The rules of ICMJE are to avoid the appearance of impropriety to as large an extent as possible, far more than AMS it appears.
So richard, if you want to argue that the ICMJE rules should apply, then you need to overcome why Steig did not follow the obvious rules and recuse himself because of the potential for bias. You cannot argue that Ryan is wrong based on ICMJE rules but then say Steig is ok based on AMS rules.
If you want to make this all about Ryan, then the question would be is it ok to violate a ICMJE confidentiality rule to out a reviewer that broke a conflict of interest rule.
Under GRL procedures (when implemented properly), Steig would have been permitted the right to respond and both the comment and response would be sent to referees.
When a paper (as O’Donnell) is not a simple comment, but a major independent work, that approach seems not to get used. Instead, the odd procedure of conflicted reviewers.
John N-G says that responsibility for sorting out the mess is on the hands of the editor. This is fine if you’re on the Team, but when the editor is Stephen Schneider (say), as for Wahl and AMmann, the hearing doesn’t seem very fair.
OK let’s establish the context.
The actual discussion that advances the science is about which statistical method best advances our understanding of the Antarctic instrument record.
This discussion is ill defined between following policy (a legalistic approach), ethics (assessing the net benefit of action) and the notion of proper conduct (an emotional response). These seem to be obscuring the substantive scientific discussion at the moment.
It’s true that ICME policy is very strict in the discipline it imposes, however this does not relate to the best way of discussing the science, it relates to the overriding obligation to protect Doctor/Patient privelege irrespective of any impediment to the discussion of science.
In the journal actually in question no such issue arises and consequently there is no requirement placed on either the reviewer or reviewee to keep their identity or indeed their remarks confidential. indeed the bulk of the reviewer policy concerns the need to protect the intellectual
property of the author. The principal requirement regarding reviewer confidentiality is placed on the editor who shouldn’t reveal reviewer identity until the reviewer has done so themselves.
With regard to the ethics of the situation this can only be assessed with respect to the net result, i.e is the ,science or indeed the world better off the way things have panned out than if an alternative course of action had been taken. If I were being ironic I would note that however someone feels about the ethics of being “outed”; under the right circumstances it would be perfectly ethical to murder them in their beds, feed the body through a bark shredder and feed the resulting slurry to herd of starving pigs.
This leaves us with “proper conduct”, “rightness” and “honour” which each person must decide for themselves. My review experience does not extend to academic papers , only to engineering and software projects,management and system reviews, public procurements, and National and International
standards commitees. In all these cases I have been known to both the editor and the reviewee unless there was a procedural or legal obstacle precluding it.
In my view the “honour” requirements would be the following. bear in mind this is the emotional part of the discussion not the logical one.
You should always declare yourself unless there is an overarching requirement otherwise, its good manners and, well, the honourable thing to do.
You shouldn’t put anything in a review that you wouldn’t say face to face or you wouldn’t be willing to defend in a public forum.
You shouldn’t find yourself being, what’s a polite phrase, imprecise, about what you stated in your reviews and the position you are currently defending. That would really put you outside the bounds of honourable conduct. It would also demonstrate that any moral contract assumed had not been
made in good faith and could be deemed dissolved.
I’ve seen reviews taped to office doors. These were reviews that were so poorly thought out that the authors wanted to share them. If teh reviwer is anonymous then what difference can it make.
Thinking about that, I have also seen the answers to examination questions posted in the main hallway of a physics building. The professor thought that the answers were so bad as to be humourous. I wondered at the time if the student’s friends would recognize his handwriting, writing style etc. I thought the professor’s action were despicable, juvenile and all too typical of academic arrogance.
Perhaps an anonymous reviewer could be recognized from his writing style, references, interests etc.
Another element in the chronology was the discussion at Dotearth in early January. The 88 pages of review correspondence was contested by Tom Crowley, who more or less challenged me to show the correspondence or withdraw the statement. This led to a number of emails between Crowley, Ryan and myself, which I’ll discuss some time.
Nobody at dotearth- including Revkin – said that confidentiality should prevent me from being required to produce the review comments.
I might add that the dotearth challenge had pretty much made up my mind to disclose the review comments and probably would have done so in early January had I not been in Thailand and was then unable to access the server. By the time that I got back, I was thinking of other things.
I’m struggling to see why the AMS policy is relevant here at all, given the facts as presented. If Steig provided information outside the review process without requesting confidentiality, it’s not confidential. It may have been an unintentional slip on his part, but if so, it’s his mistake to live with.
Recall also the Obsborn, Jones, Mann episode.
osborn is a reviewer. He lets Jones know he is reviewing a paper ( is that a violation)
Osborn does not want to discuss it with Jones because osborn wants to do things “squeaky clean”
Jones takes that information and asks mann if he is a reviewer. a simple yes or no. because he knows
that mann will say the things that need to be said.
Under point 2 in the main post, the quote should be “editor or knuckleheaded reviewers” at the end of the paragraph.
Click to access authorseditorsreviewersguide.pdf
“7. A reviewer should treat a manuscript sent for review as a confidential document. It should neither be shown to nor discussed with others except, in special cases, to persons from whom specific advice may be sought; in that event, the identities of those consulted should be disclosed to the editor.”
I wonder if Eric Steig disclosed to the editor the identities of those with whom he consulted. O’Donnel et al noted that the Review A comments appeared to have been written by more than one person….. based upon both writing style and content.
Eric has stated that he wrote the reviews in their entirety. I do not see any reason to doubt him on this. I think this line of speculation is a dead end.
Has it not been established that:
1) Steig was Reviewer A, and that;
2) He implied, and tried to give the impression, in public that he was not Reviewer A?
If that has been established, that is gross – snip – alongside which arguing about journal requirements seems minor.
One could argue that Steig/Reviewer A was a victim of the illogical policy of assigning or granting anonymity to reviewers who are unavoidably intimate parties to the article and all subsequent discussions. As I have said repeatedly there is no logical/practical reason for granting anonymity to reviewers who have CoIs.
Personally, I think secret reviews and reviewers causes more problems than it solves, but that said, if I had been Ryan I would not have “outed” Eric. I would have simply referred to him as “Reviewer A” and forced Eric to out himself when he subsequently tried to clarify Reviewer A’s statements. Pitting Eric against himself in public would have been quite entertaining.
If review comments were confidential, what are we to make of all the papers that contain an acknowledgment: ‘The authors wish to thank the anonymous referee who carefully reviewed the paper and pointed out…’? Are such papers in contempt of this supposed confidentiality? (Of course not!).
I’ve been involved in peer review from both sides for over 30 years. Having “been brought up” under the then prevailing culture of confidentiality, I did raise an eyebrow when I saw the “outing” of Steig as Reviewer A.
However, in the New World Order, it is becoming clear that the quaint idea of confidential peer review to “move the science forward” has outlived its usefulness.
Steve Mc’s recounting of Mann’s gloating, “Crowley’s challenge”, the climategate e-mails (“go to town”, “redefine peer review”,” don’t tell anyone I sent you the manuscript” etc., etc., etc.) reminds me that we’re not in Kansas anymore.
And it was, after all, Steig’s resurrection of the debate started during the “confidential” peer review process in his Feb 1 post at Real Climate that set this all off again.
All this leads me to conclude that the outrage shown by the academic poobahs about all this is sort of like someone expending a lot of energy trying to protect the latest buggy whip patent in 1905.
Times are changing. Get used to it.
Science is a full contact sport.
Q: “What’s the difference between a modeler and a terrorist?”
A: “You can negotiate with a terrorist”
I think review confidentiality is a product of old “cottage industry” type journals. The journals are looking for volunteers to do work that will not be financially remunerative. One incentive to get unpaid volunteers (or poorly compensated reviewers) is to tell the reviewers that their involvement will be anonymous. Otherwise, the headaches involved in doing reviews might not be worth the hassle.
Climate science reviews are different because huge public policy and spending choices can be determined by climate science studies. Because the stakes are so large, at the very least the reviews should be made public. Also, any person who is reviewing a paper that comments on his (the reviewer’s previous paper) should be publicly identified. It is a close question to me whether other reviewers should be publicly identified in my opinion.
JD
[Snip]
[RomanM: I think most everyone here has had enough of your high school level games. If you insist on continued trolling, you may well find your permanent way into the round file of electrons.]
Boro Nut,
Time to look forward, not backward. Those buggy whips worked real well for a while too.
Besides, rayon, jet engines, and the electric light bulb were invented before the current version of peer review took hold, and most of the advances in semiconductors have been due to manufacturing innovation and not peer reviewed research.
Other than all that, your comment is as spot on as all your previous ones.
Awww Roman, now you’ll have to snip mine too.
Rats.
[RomanM: C’mon, you know better than to feed the trolls…]
The guidelines and bounds presented above (J N-G’s comments and the guidelines of AMS) have limited utility within the scope required by our host.
Are review comments confidential?
Apparently not.
Nonetheless, does an author have an obligation or an interest to protect the interests of the journal and society? Must the journal and society reciprocate? At the least a handshake partnership prevails.
I see no offense on the part of the journal in this instance nor one been alleged. The conflicted peer review has been well debated; at best it’s a push. More likely it’s a necessary weakness of the process.
Accordingly one wonders how the powers that be at AMS feel about this kerfuffle. They might be compelled in this case by one side or the other. Who knows? But certainly this was not their preferred venue or outcome.
Their response, public or private, will reveal the actual bounds.
I keep reading the AMS guidelines for authors/editors/reviewers linked above. I have to admit to being absolutely stunned that the guidelines for authors are absolutely silent on any confidentiality requirements on authors.
It’s not like the people writing this totally overlooked discussions of confidentiality. They explain the requirements for editors and reviewers. But for authors? Nothing!
That makes sense if you think about it. The authors do not know who the reviewers are so how can it be their ethical responsiblity to maintain some perceived right of anonymity? It is the Reviewers responsiblity and if they out themselves, I do not see how it is anyone’s problem but theirs.
Also, consider this on Editor Resonsibilities from the AMS guidelines:
“2. An editor must protect the confidentiality of all reviewers unless the reviewer reveals their identity to the author.”
That infers to me that if a reviewer reveals themself to an author, well, it is on them from that point on.
Precisely, once the reviewer reveals him or her self, the editor and the AMS no longer have a responsibility in the matter. In addition, under the MAS list of reviewer responsibilities we find:
“5. A reviewer should be sensitive even to the appearance of a conflict of interest when the manuscript under review is closely related to the reviewer’s work in progress or published. If in doubt, the reviewer should indicate the potential conflict promptly to the editor.”
Steig had no business reviewing the O’Donnel et al. paper. He should have waited for publication and then responded. Further, one is lead to wonder just how Steig was even on a list of potential reviewers for the paper. Also, did Steig reveal his conflict, and whether he did or not, how did the editor miss this conflict of interest?
That should read “… under the AMS list …”
““knuckleheaded editor and reviewers””
Odd. I’d thought it was the reviewers ES considered knuckleheads”
Silly me.
Sorry, but I can’t help a slightly O/T but related observation here.
The most “common” standard/practice in peer academic review seems to be the maintainenance of confidentiality for the Reviewer(s) – even to the point of an injunction against publishing reviews, as seems to be the case with some journals,
So, it’s somewhat ironic to recall that SOP in the course of an IPCC “gold standard” peer review is to offer no protection to the Reviewers (except for Gov’t Reps) – whose names are always in plain sight during course of “responding” – while at the same time shielding the authors (who, in effect, act as final arbiters) from any responsibility for their choices, by virtue of the smokescreen of “Chapter Team” (cf Muir Russell’s verbal gymnastics).
YMMV, but I think there’s something wrong with this picture.
>if there is other information that makes it possible to discern the identity of the reviewer, such information should be redacted unless the reviewer grants permission.
You are speculating based on the content of the review. Thus the review contains information that makes it possible to discern the identity of the reviewer.
Thus you have speculated on a reviewer’s identity using information that should have been redacted. I’m referring more to the Jones review as it is a cleaner example.
I should probably say the possible Jones review, as your evidence was less than convincing.
Once upon a time, in a galaxcy far far away. (Somewhere in the Midwest, flyover country)…a distinguished professor at a Nuclear Engineering department of a State University threw two patents on the desks of two of his graduate students. One was filed by Philo T. Farnsworth, the other by his associate, Robert Hirsch, shortly after Farnsworth’s untimely death, in 1969. The patents covered the Farnsworth Fusor.
This distinguished professor said to his students, “I’ve always wondered if this think really worked…we’ve got $20K in our discretionary funds, would you like to give it a try?”
Well, just under $20 K later, and 9 months, a 2′ diameter SS ball (hollow, with various electrodes inside) was filled with D2, 4000 Volts applied, and a “blue/white” fusion star appeared in the center of the device.
IMMEDIATELY it was shut down as the 270 KeV Bremstrallen, and the 10,000,000,000 5 MeV neutrons per second, were quite the radiological hazard, and it took a couple weeks to put enough moderator and shielding around the device to make it safe for operation without frying grad students.
One of the grad students took on the device as his Phd study. He characterized the fusion reaction, doing detailed neutron flux measurements for a year. Wrote up a thesis, submitted it for his Phd, and was awared his Phd in Nuclear Engineering on the basis of this work.
The distinguished professor suggested that a nice, 8 page paper should be submitted to “Science”, which was done quickly…and a couple months later the paper was returned, with a few annotations from the “peer reviewer”..and a copy of a 1972 paper written by some other “peers” or “experts”, which PROVED that the “Farnsworth Fusor” can’t acheive any fusion reactions.
After waiting a few weeks, the distinguished professor, was “ticked off” enough to take a stab at who the reviewer was, and he decided to call him.
He said something like this (I’m told, I’m not he..but over a nice “brew” in front of a warm fire, on a cold night in a midwestern city far from the center of the galaxcy…this story was related to me..) “Fred (made up name), did you enjoy reviewing the paper by B.J. and myself?”
“Ah yes Ralph, I did, in fact review that paper…” (Trailing off, I’m told, as distinguished professor was not supposed to do this according to SOME IDIOTS CONCOCTED PROTOCOL WHICH SHOULD IN THE FUTURE BE IGNORED…!!!!!!)
Distinguished professor then said, “I’m just curious, if the F.F. (which we call the IEC Device) doesn’t produce real fusions, where do the Xrays and 5 MeV Neutrons come from when we turn it on?”
At this point, Fred addressed the D. P. by name, and said curtly, “I’m really busy right now, and don’t have time to talk…” And hung up on the D.P.
Now the D. P. has published in several journals, NONE of course as prestigious as “Science”, and there has been a HOST of activity with people building F.F. devices for research and even HOBBY interest.
But Gosh Darn…some now dead “experts” in 1972, who worked for OAK RIDGE by golly, showed the F.F. couldn’t work.
And if it was good enough for them, it was good enough for the ANNONOMOUS “peer reviewer”, and the poor D.P. should have known better than to persue the matter and should have given up right then and there.
The moral of the story…Peer Review is a method of re-inforcing ORTHADOXY and who knows HOW MUCH REAL PROGRESS HAS BEEN BLOCKED since “peer review” has been elevated to a DOGMA status?
I hope that S.M. and others can continue to DESTROY the Peer Review process and re-open science to the true exchange of ideas it should be, without “gatekeepers” blocking what their minds are too small to contain.
AMS policies also state:
That reads clearly to me. What is the context? Where is that in the list of policies?
Re: Jeremy (Feb 16 09:10),
The context seems to be the document describing AMS policies that was posted at the time O’Donnell was submitted:
Click to access authorseditorsreviewersguide.pdf
I haven’t seem anything else– though maybe eventually something else will surface.
In an era of Wikileaks and Climategate it is touchingly quaint to have assorted partis pris complain of confidentiality breaches.
Moreover, with the increasing scope of FOI requests, it is hardly obvious that any scientist who obtains government funding has much claim to privacy in respect of his professional communications.
It is, ladies and gentlemen, a new world with new and emerging rules. Like ’em or hate ’em, the fact is that the rules mean your material, however casual, is likely to be published.
Should you wish to obstruct a position or a paper you must know that your obstruction will out.
As to Steig; he was much better off taking the high road and accepting the correction so wonderfully adminstered by O’Donnell et al . His method overestimated overall warming. Stuff happens. Then it gets cleaned up. Steig’s initial welcoming of the paper was a lot smarter than his subsequent whinging. At the moment he has lost his basic credibility by being unable to answer the subtance of the paper whilst running blog to blog yelling “Liar, liar, pants on fire.” It is unseemly and, more importantly, utterly unresponsive to the O’Donnell critique.
Bearing in mind all the facts, the issue of disclosure is clearly not a matter dependent upon journal policy, but one that (in my opinion) has more to do with _expectation of privacy_ and questions of _bona/mala fides_. The saying is that one cannot have a “confidential murder”; but in civil matters, how can one have an expectation of confidentiality when one publicly contradicts one’s own private statements? “Public interest” is the clear and overriding objection to any complaint; nobody can be expected to treat as “confidential” any information that is publicly contradicted: that would be repugnant, as effectively binding any party to silence if they received, confidentially, any information which a person wished (perhaps premeditively) to contradict.
Why remain anonymous in the first place?
Why reveal your identity in the second place?
Conscience or mind games?
A professional looking at a fellow professionals work, in the UK, is duty bound to advise that they are reviewing another professionals work. Also they must show that they have made all efforts to do so. The only get out is that if the Client refuses.
Also the reviewer should make all efforts not to un-necessarily injure the reputation of their fellow professional.
The two statements by the editor are logically incompatible in my view.
If the journal policy allows the anonymous review to be published ie put into the public domain, then once it is in the public domain any individual may speculate all they like as to the identity of the reviewer.
No policy can prevent this, or apply any further sanction at this point. Even if the restriction of not speculating on the identity still applied to the author, what would be the point? Plenty of others can speculate to their heart’s content once the review comments are in the public domain.
1. do originating authors have a duty of confidentiality under journal policies or otherwise to maintain review comments received from journals in confidence?
No. Usually there is an unwritten understanding that reviews are confidential, but there are several examples of reviews that are regarded as unfair being made public (see Tom Gray’s comment).
2. did O’Donnell have a duty of confidentiality arising out of his correspondence with Eric Steig in December?
No. Steig messed up in several ways and fell into a hole of his own digging.
It was unwise to admit to being the reviewer – it would have been better not to say anything or to say that reviewers should remain anonymous. I only very occasonally admit to being a reviewer, and then only when I am writng a positive review.
He messed up further by pretending that he wasn’t the reviewer in public (for example saying that he looked forward to seeing the paper and asking for a copy – I think this was on Jeff’s blog). Of course saying one thing in public and a quite different thing in private is a common Team procedure, as shown by climategate.
Even more unwise was his Feb 1 post at RC which led directly to Ryan’s understandable response.
I understand this may be snipped;
here is an example of me some years ago publishing reviewers comments in an egregious situation.
nobody complained to me about this, but nor did it achieve much, other than helping me feel better and leaving a record.
http://clarkfrancis.com/rafts_debacle/index.html
In short I withdrew under duress from authorship on a paper that I had been instrumental in starting. They proceeded with the mistakes anyway, and when I tried to publish a straightforward (but significant work) replication that demonstrated the now published paper was based on, at best, magic pudding modelling, I got snowed.
Personally, while the conversation about journal policy and the nature of the traditions is useful, I reckon it’s simple: the peer-review process needs a shake-up, and more open review is an obvious way forward.
If someone started a site that collected and ordered everyones peer-review stories, I have other crackers to share, but need a reason to put the work into presenting them.
John Nielsen-Gammon sent me the following email observing that his attempt to post the comment here had been unsuccessful. His comment is not in the “pending” file. I have no idea why he was unable to post a comment; it was nothing to do with me. Anyway here is his comment:
Re: Steve McIntyre (Feb 16 10:29),
I appreciate Dr Nielsen-Gammon commenting on this issue.
If he is merely expressing his personal opinion (as he says),I do not understand why he would expect his personal opinion to be binding on anyone else.
As Dr N-G observed, I have speculated on the identity of reviewers at CA from time to time. On those earlier occasions, to my recollection, neither Dr N-G (nor anyone else) suggested that such speculations might be a violation of ethical principles.
I would also be interesting in also considering the situation e.g. Climate of the Past where a journal publishes open anonymous reviews. Does Dr N-G think that it is unethical to speculate on the identity of reviewers in that situation?
A related situation. What if AMS and the individual reviewers had all given specific consent to the publication of the reviews? Does Dr N-G think that it is unethical to speculate on the identity of reviewers in that situation?
Dr. Nielsen-Gammon is not expecting his personal opinion to be binding. He is explaining to you, what the generally held ethical standards in the scientific community are. Whether or not you agree with them. Dr. Nielsen-Gammon correctly lays out to you, how the (initially incomplete (1 20100121 Antarctica SI.pdf), possibly still incomplete (4th round?)) publication of the reviews by you and your co-authors are seen by the average member of the scientific community. By your actions you have lost a lot face with them, whether you like it or not, whether you care or not, whether you try and lawyer your way out of it or not.
bluegrue, Dr. Nielsen-Gammon has said that the second sentence is his own personal opinion.
However, he then goes a bridge too far. He also claims that his own personal opinion “outlines the bounds of ethical behavior” within the peer review process.
You repeat the error, by claiming that what Dr. Nielsen-Gammons gives as his own personal opinion represents what the “generally held ethical standards in the scientific community are”.
Now, as far as I know (and always subject to correction):
• Neither you nor Dr. N-G has been elected by scientists to rule on what the “bounds of ethical behavior” or the “generally held ethical standards” might be.
• Neither one of you has expertise or publications on the questions of ethical science.
• Neither one of you has polled the world’s scientists or journal editors or authors about what the ethical standards actually are.
• Neither one of you has provided a scrap of evidence or a single citation to back up your claim that you have the inside information on global ethical standards for peer review.
Given those facts, I’m surprised that both you and he are setting yourselves up as the global arbiters of ethical science … although with what I know of climate science, I suppose I shouldn’t be surprised.
w.
I am a researcher in solid state physics. Take my word for it or leave it, I don’t care. I describe the peer review system as I have seen it in action and I describe the ethics as I see them held by my colleagues and myself. Take my word for it or leave it, I don’t care. I am not an arbiter but a witness. Take my word for it or leave it, I don’t care.
bluegrue said:
Posted Feb 16, 2011 at 2:42 PM | Permalink | Reply | Edit
No, you are not “describ[ing] the peer review system as you have seen it”. Instead of giving us your own opinion, you have claimed that Dr. N-G knows “what the generally held ethical standards in the scientific community are”.
Now, Dr. N-G may be correct or not. In a scientific discussion, that question is usually settled by facts. You know, observations. Polls of scientists. Information about the real world. Research papers by Dr. N-G or others that explore the question.
It is not settled by you saying that you agree with Dr. N-G, and we can “take [your] word for it or leave it”. You have provided nothing to back up your word other than your further word that, like I and many others here, you have actually seen peer review “in action” … that’s your evidence? The fact that you have actually observed peer review in its native habitat backs up your claim about Dr. N-G?
Given your excellent options (“take [your] word for it or leave it”), I’ll go with Choice B, thanks.
w.
You are welcome.
Re: bluegrue (Feb 16 13:08), And yet Dr. Mann did not lose face by the actions he took WRT McIntyre in 2004.
How does that work?
old boys club.
“By your actions you have lost a lot face with them”
Who cares. By my reckoning that still leaves him one whole face.
LOL
It appears that his ethical standard only applies in one direction and only to someone who is not an insider. I still await an opinion as to what the ethical obligations of Reviewer A was/are, and if he stepped outside Dr. N-G’s self proclaimed boundaries.
Re: Steve McIntyre (Feb 16 10:46), This whole thing leads to another funny situation.
Would be ethical to post reviewer A, and post Steigs RC post, and speculate that Steig was NOT reviewer A?
What Dr. N-G and others fail to address is the specifics of this case. Its the specifics that create an ethical novelty
Any comment on ethical behaviour of Reviewer A, or just the Author?
Actually, I haven’t commented on the ethical behavior of the author. I’ve tried to explain the policies and expectations within my scientific community. The author is not from that community, and arrived with a different set of ethics and expectations which were no less legitimate.
I have commented below on the ethical behavior of the reviewer.
John N-G Said: Actually, I haven’t commented on the ethical behavior of the author.
The below is your statement as posted above at: https://climateaudit.org/2011/02/15/are-review-comments-confidential/#comment-255409
“Precisely because the issue is not discussed in the AMS’s guidelines, I thought it would be useful to outline the bounds of ethical behavior for the benefit of those not so familiar with the process. Announcing the identity of Reviewer A fell well outside the bounds that I outlined.”
You do not consider the above to be a statement on the ethical behavior of the Author?
Re: Steve McIntyre (Feb 16 10:29), Well yes…. and then the situation changed…
Precisely because the issue is not discussed in the AMS’s guidelines, I thought it would be useful to outline the bounds of ethical behavior for the benefit of those not so familiar with the process. Announcing the identity of Reviewer A fell well outside the bounds that I outlined.
So I see that this advice was followed till the situation changed significantly. So the comments are now clarified for what had occurred in early December last, but seem to have no bearing on what occurred in January and February of 2011. So I do not see these comments as helpful in unraveling this situation.
So by implication Steve is unethical. I guess I am too since I would have reacted in a similar fashion to the treatment given
I guess for me to feel the situation has been addressed he would have to address the issue of having Steig review a paper which points out the flaws in his work. The he would have to address the issues of Steigs public comments. Since that has not happened,then there is still no resolution.
I am used to open review so I guess I come down on that side — fwiw. Then anybody I am criticizing can review to their hearts content. If their review of my work is flawed they will pay in the end, and that is all that I ask. …and I think the payment is about to be extracted from Steig.
Gee, seems like some of us were spot on.
Steve, I think you’ve gone a bridge too far in trying to split hairs on this one.
Re: Eric Anderson (Feb 16 13:18), Maybe. Maybe not!
Eric aaid: “Steve, I think you’ve gone a bridge too far in trying to split hairs on this one.”
When someone says to me “Let me tell you today what I really meant two months ago…” and there has been a lot of water under that “bridge too far” then I never know what to think. I certainly have my suspicions though.
Re: Steve McIntyre (Feb 16 10:29),
Anonymity should not be used as both sword and a shield. I think the intent is for anonymity to be a shield for the reviewer as has been well commented on by others. However, when a reviewer attempts to use his or her anonymity as a sword (as I would submit Dr. Steig did), then he or she should lose the benefit of anonymity as a shield.
JNG said “Steve asks whether any breach of AMS journal policy actually occurred. Certainly there was no breach of any published policy. The issue of publicizing reviews is not touched upon in the AMS’s guidelines for authors, editors, and reviewers. Likewise, it would not violate AMS journal policy to visit the home of the author of an unfavorable review and let the air out of the tires of his or her car. That issue, too, is not touched upon in the guidelines.”
This is a fun premise…but let’s go a bit further. If such a thing actually occurred…letting air out of tires…there are other remedies to pursue against the errant author in your story; statutory remedies, for example, that are laid out in writing.
Let’s take your fun premise about what’s not in “the guidelines” a bit further…just for fun. So…we all know that there is more to the story here. Let’s flesh it out. Actually, the rest of the story is…Reviewer (let’s call him “A”) had just come from Author’s house where he had stolen Author’s chickens and put them in the trunk of his car before returning to his own home. Author saw him in the act and followed him to his old homestead where Author confronted A. He asked A if he had stolen the chickens and if they were in the trunk of his car. A admitted that he had in fact stolen the chickens upon Author’s promise that he would not make the indiscretion public. A then said he was headed to the farmers market to sell the chickens. Author, thinking quickly, let the air out of the car’s tires so A could not get away until the sheriff arrived. A was promptly arrested. At the hearing, A and all his family complained loudly that it was just not fair that A was arrested. After all, Author promised not to tell. It is Author, they protested, who should be indicted for letting the air out of the tires.
JNG…do you still agree with A and all his kin. It is Author who is the low down varmint. After all, he promised. Is that right?
A new euphemism for reviewing a paper: “stealing chickens”. I like it.
Substitute “helped build a stronger chicken coop” for “stealing chickens” and I think you’ll have a better analogy, though.
Well, there we are…the answer to your question number 1:
“Certainly there was no breach of any published policy. The issue of publicizing reviews is not touched upon in the AMS’s guidelines for authors, editors, and reviewers.”
Dr. N-G’s statement is consistent with your observation:
“In my opinion, Neilsen-Gammon has failed to show any breach of journal policy.”
Dr. N-G then goes on to state that in his opinion you acted unethically, but that’s not the question you were asking for this particular topic. That would apply to question 2), and you had asked for discussion of that topic to be delayed until you discussed it in another post.
Since he, and others, have brought it up, however…
What Dr. N-G seems to be saying is that you should feel ethically bound by inexplicit customary behavior. His example is poor; there does not need to be a policy by the paper of not letting air out of tires because it is already addressed by law. A better example might be: the Policies do not explicitly address trapping Steig in an elevator and “letting wind”.
I would not agree that the authors are ethically bound by inexplicit customary behavior, especially and ironically if people like Dr. N-G consider the authors to be outsiders, (as implied in Dr. N-G’s response) but I think the question is really moot.
The overall question of whether outing Steig was unethical can’t be discussed as if there were only one ethical imperative at a time. It’s easy to see that if someone says “Promise me you won’t leave this closet for the next hour”, and you agree, and then he says “I’ll be right outside this door slaughtering your family”, then although it would be unethical to break a promise, it would be even more unethical not to.
I think of the idea that ethics should be considered in a vacuum as “Batman Ethics”, because during the early part of Batman’s career, this was a plot device. The Joker might extract a promise from Batman to not untie himself, then start filling the room with water, and Batman would have to try to figure out a way to save himself without untieing himself.
Even Comic books have abandoned this simplification as comics grew up.
I think the simplest statement of the actual ethics of keeping a promise is that you have an ethical responsibility to keep a promise, or implied promise of confidentiality, unless it causes some harm to do so. If there is harm, then greater ethical responsibilities kick in, and the situation must be considered as a whole.
In this case, if Steig used his anonymous reviewer position to influence the paper, then criticized the paper based on the implementation of his recommendation, he has caused harm in a number of ways. He has damaged the ethical position of the journal, done damage to the peer review process, damaged the authors of the paper, and, given that the AGW folks insist that science is to be done through the Peer Review process, he has damaged the science itself.
In my opinion, it is actually the Journal that had the responsibility to out Steig, but given that the Journal didn’t, I think Ryan did the only ethical thing to be done. It would have been an ethical crime for Steig to get away with abusing the science.
RDCII- I disagree with your characterization of my statements and agree with your criticism of that characterization.
John N-G…
I read further of your comments yesterday after I posted this one, once you had worked past the environmental issues preventing you from freely posting, and I agree that I misunderstood your intent in the last paragraph.
Given that my response was somewhat critical of you based on my misunderstanding, your response here was beautifully succint and quite gracious. Thanks.
“For example, if a lawyer gives legal advice to a client, the lawyer has a duty of confidentiality to the client not to disclose the content of the advice. The obligation is not symmetric. While the client has the right to keep the advice confidentiality, he has no obligation to the lawyer to keep the advice confidential.”
Steve, all the explanation that is needed is contained in your example (above). The reason that the lawyer cannot disclose publically or privately any advice given to a client rotates on a fact of property law, in that, the advice given was/is paid for by the client therefore becoming the property of the client. The client, on the other hand can divulge the advice simply because he/she/them (the client) own the property rights to the advice once payment is made.
Unless Ryan sold the paper to the journal, thereby transferring the (a) property right, then Ryan would be bound to some sort of agreement or confidentiality, as opposed to, if Ryan did not sell the paper then Ryan retains legally, ethically and morally all the property rights associated with his paper and reviewer names and comments form part of the paper which is also Ryan’s property.
Simple really.
After peer review and before publication the author usually has to transfer the copyright of the paper to the journal. He may retain the right, to post the paper on his homepage for download in formated (rare) or unformated form, but even that right depends on the journal.
The notion, that the author of the paper could gain the copyright on the verbatim comments of the reviewers is bizarre.
Property AND things forming “Part” of the property. Bizarre indeed, try again.
The obligation for an attorney to maintain the confidentiality of his advice to his clients doesn’t really have much to do with property law. If that were the rationale, the bounds of the obligation would be rather different than they are. For example, if that were the rationale, then it would be improper for me to re-use the results of legal research that I performed for one client on behalf of another client.
The real reason for the confidentiality is that it is necessary to permit clients to get good advice, because only under confidentiality can the client be comfortable giving potentially embarrassing (or worse) information to their attorney. It’s a bit like the reason journalists protect their confidential sources. Not surprisingly, judges (who are, after all, just experienced lawyers with more power) have elevated this interest to the status of a legal privilege.
In the case of peer review, it’s not clear to me what purpose the anonymity of reviewers is supposed to be serving. The idea that it will free them to give the most honest review seems dubious, though not completely unreasonable. As the present case shows, transparency seems a more likely motivation for honesty in the context of science, since, by it’s nature, science is supposed to be about reproducible facts. By contrast, the need for the reviewers to protect the confidentiality of the subject matter they’re reviewing seems far more crucial, so the authors need not fear losing rights. (The fact that the goal is publication doesn’t eliminate this concern, since the timing of publication can be important.)
“For example, if that were the rationale, then it would be improper for me to re-use the results of legal research that I performed for one client on behalf of another client.”
That is not so. Two different legal practitioners acting for two different parties may come to the same conclusion. Your opinion just fell over. Accidental overlap in advice given to different parties and upon paid commission still belongs to the individual parties as separate entities. There is no breach.
You’re confusing the content with the work product. The property right obviously cannot exist in the legal “answer”, because, by the act of doing the legal research, I am not creating that answer–I’m simply extracting it and writing it down in a form that is more useful to my client.
To further illustrate the difference, property rights in a commissioned work can revert to the creator for breach of contract–for example, if the patron doesn’t pay. That does NOT happen with the legal privilege. If my client doesn’t pay me for the work I do, I am still obligated to maintain the confidentiality.
According to their 2010 copyright policy, the AMS requires the author to transfer copyright to the AMS as a condition of publication (waivable for some materials). The transfer becomes effective when the paper is accepted.
At least by the definition of copyright in the UK, copyright includes paternity rights, so this would mean that AMS has the right to be recognised as the author of the paper.
Despite the nearly 6 hour lapse, bluegrue’s comment at 1.37pm (and QBeamus’) weren’t visible until I’d posted my comment, although I’d refreshed just before posting.
The comments and recent comments list have been behaving strangely in the last couple of days (sometimes showing older comments).
Interesting theory. Without a contract and the passage of funds ‘Copyright’ is not worth the paper it is written on.
Here’s a comment at RC from Steig on March 24, 2010 that bears re-interpretation now knowing that Steig was Reviewer A. (Steig’s review had been submitted three weeks earlier on March 5.) At the time, Steig knew that I was a coauthor of a comprehensive analysis of the Antarctic record but nonetheless alleged that I was “unwilling” to do the “real work”. Santer’s claim was also untrue – Ross and I had twice submitted a comment on Santer et al 2008, both rejected (one rejection was promptly reported by Peter Thorne to Phil Jones before I reported it anywhere).
Eric Steig says:
Steig’s claim as to availability of data was also untrue. It took a prolonged effort to get the AVHRR data. This was resolved only after Ryan made a materials complaint to Nature in August 2009.
Professor John W. Nielsen-Gammon:
Since you have outlined the bounds of ethical behavior, what is your opinion on this issue?
Re: glacierman (Feb 16 11:14),
Yes, I’d like to know that bit of information as well. I have an opinion with respect to Dr. Nielsen-Gammon’s assertion of ‘unethical behavior’ for the facts presented to date. It is not the same opinion that Dr. Nielsen-Gammon proposes.
Dr. Nielsen-Gammon’s imputation of misconduct against O’Donnell/McIntyre remains in the nature of a snobbish observation that the new boy wore a lounge suit at dinner.
An unwritten accommodation agreed amongst a self serving clique for their mutual comfort does not amount to an ethic.
Why do you always have to drag the kids into our squabbles.
You can keep em Bob. I’m going back to my mothers. It’s over.
Steve, you are focusing on whether or not it was legal to disclose the reviews and to disclose Steig as a reviewer. I think you are missing a major point. It is at a minimum customary, to keep the review process confidential for all parties involved. Sorry if I should have missed this, but have you ever asked yourself, why it is handled this way?
Of course, I’ve wondered why. The custom in other fields e.g. engineering, business is that parties have to sign their reviews. If climate science is to be applied towards public policy, it seems entirely reasonable to me that more formal systems of review e.g. signed reviews, might well provide a more secure chain of quality control and due diligence.
In some ways, it seems to me that the primary resistance on the part of academic climate scientists to making reviews public is that the public will be unimpressed by the quality of the reviews – they are not due diligence as the business and engineering public understands it.
I think that you are confounding engineering audits with the anonymous peer review of papers. Anonymous peer review is commonly used to assess engineering papers for publication.
An engineering design or a consultant’s audit of a design woukld be signed and stamped by a certified professional engineer. The professional engineer opens himself to legal (criminal and civil) liabilty with this signature. A profesional engineer isn’t just some grad student writing a paper about his thesis
I agree with you that the review of results for use in public policy should be much more than the anonymous peer review used to vet papers for publication. However the use of literature surveys based on anonymous peer review is the method chosen for teh IPCC. I think that this gets into a topic that you inidcated would not be appropriate for discussion on this thread at least.
The review process in scientific literature is a means to weed out the worst manuscripts and – where necessarry/possible – to improve the paper before publication. The reviewers invest the time, so that the community saves way more time, that it would otherwise spend on weeding out the nonsense themselves. This is quite different from an audit or review in engineering, where it is the job of the auditor/reviewer to ensure under threat of penalties, that every i is dotted and every t is crossed. In addition, it tends to improve your manuscript, if a third party gives you feedback (I for one prefer long replies over disinterested I-hunted-for-a-missing-comma reviews). The assured anonymity (be it explicitely endorsed by the journal or simply implied by custom) allows to ask blunt, hard questions, as well as uniformed questions without any party involved losing face in public.
Of course, people can try to game the system in order to delay/inhibit publication, if a reviewer tries to do so and the editor plays along. Just as you can game the system to push a shoddy paper. You seem to think, gaming the system was indeed the case with you and Steig. I disagree, I read it as the honest attempt to improve a manuscript and so do many others. Perhaps you can try to reread the reviews from my point of view; you’re of course free to come to your own conclusions.
If Reviewer A was actually a disinterested party, you would have a point, bluegrue. Reviewer A, Steig, was not. He had a clear stake in the outcome. He argued points he openly admits he is not qualified to argue so any claim that he was trying to “improve” the paper are specious at best. Whether he was trying to make the results look more like his, or simply delay and/or prevent publication, does not matter, an improvement was clearly not his goal.
Mark
It does not matter, whether or not Reviewer A is disinterested. You don’t even want disinterested for this kind of peer review. That’s the whole point. Who else, but the target of an attempted rebuttal, is better equipped to point out weaknesses in said rebuttal? You hand in the manuscript, you get back the reviews. It is then your job to incorporate those changes, that you think to be valid, and argue against the ones, that you do not think to have merit, and explain to the editor, why you are not going to go along with the criticism. The editor is the one person to judge, whose arguments on which point are better, the author’s or the reviewer’s. If necessary, lather, rinse, repeat one or two times. If you are still unable to convince the editor of your point of view, it should give you a pause.
And I do disagree, Mark, I do not see intent to either delay or inhibit. Reviewer A asked the editor to insist, that the authors plot the results that the authors themselves identify as the most likely. What is wrong with that? Should they plot the third best? Or the worst?
P.S.: If you want to see hostile reviews, see e.g. here:
http://onlinelibrary.wiley.com/doi/10.1111/j.1462-2920.2010.02394.x/full
bluegrue,
In what specific way do you believe the paper to have been made “better” ? I don’t want some hand wave “well other eyes looked at the paper”. What specific items were required to be removed or added by Steig that you believe made the paper better. I see some interesting and on target material chopped simply because O(10) had to deal with the asymmetry of the review process. It essentially came down to Steig’s personal opinion (and in many cases his opinion being flat out inaccurate) vs O’Donnell et al in a case where Steig had to present no backing evidence. In some cases value was cut just because it was too difficult to deal with Reviewer A. In this specific case, it sure seems to me that the loss of technical work backed up in the paper both in the paper and in pervious blog articles here is a net minus.
Arifex – In all fairness, I believe that ODonnel himself said that the process resulted in a ‘better paper’. Now I’m not sure what aspects he was refering to and if it had anything to do with Reviewer A, but he did acknowledge that.
Ryan and Nic preferred iridge to TTLS relatively early on and wanted to publish an iridge paper as “future work”. They accelerated this in response to Steig’s Second Review comment – which Steig now more or less disowns. This hardly counts as a process ‘working”.
If one considers the literature trail on this matter, Steig’s participation in the review process seriously damaged the research record. The literature trail needed something like the First Version, which placed TTLS results and Chladni patterns on the record, as well as the iridge results. Had the TTLS results been placed on the record, with a follow-up using iridge (as was planned), that would have been a much more coherent literature trail than what we now see, with a jump to iridge and Steig arguing against the method on blogs.
The literature trail needed something like the First Version, which placed TTLS results and Chladni patterns on the record, as well as the iridge results. Had the TTLS results been placed on the record, with a follow-up using iridge (as was planned), that would have been a much more coherent literature trail than what we now see, with a jump to iridge and Steig arguing against the method on blogs.
This is by far the most important, overlooked point in the last days of various tangential arguments.
I’ve only had a look at some of Reviewer A’s requests and the replies by Ryan et.al., I don’t feel qualified to pass judgement on the benefits of one method vs. the other. One improvemnet would be “The RO10 text does not describe the relative importance of the modifications proposed.” Apart from that, the question discussed here, was whether or not Steig tried to be obstructive. I see no evidence of that. He does drive the authors to focus on the results for the trends in the Antarctic and showing not only the differences, but the similarities between O10 and SO9, concerns echoed by Reviewer D.
bluegrue says:
“He does drive the authors to focus on the results for the trends in the Antarctic”
… and IMHO this is one of the factors that actually a minus. I thought that the Chladni patterns were both interesting and relevant. The message I took from the initial paper was: “The algorithms used by SO(9) are subject to numerical artifacts, this is understood in other fields and we can clearly demonstrate these errors”. The question “Is the Antarctic warming ?” is really a sideshow to this paper. Pushing the paper in this direction did it no favors, though certain reviewers may feel more comfortable with questions such as “Is the Antarctic warming a little or a lot ?” rather than “Is my method actually solid ?” (which yields a far more uncomfortable answer)
Success for this paper is when future reviewers ask: “O(10) showed these methods have artifacts, can you demonstrate your method is free of these problems” rather than “So how does your warming compare to O(10)”. I feel trying to push it in the direction it was pushed weakened what I felt was it’s primary message to allow the face saving maneuver of arguing results instead of arguing algorithm.
IMHO, that’s a matter of perspective. What are you interested in, the method per se or the results? Given the focus of Journal of Climate the editor and reviewers, in the interest of the readership, naturally put the focus on the results of the analysis and the best known algorithm to tease information out of the data available. Whether or not SO9 was suboptimal or even just lucky is of secondary concern. The readers are interested in what is happening in the cryosphere, so warming in an important area is of major interest. Don’t get me wrong, though, the people in the field will note that Steig’s analysis has been superseeded, even if it’s just part of a paper and not a whole paper dedicated to that matter.
You say that the field will note that Stieg’s analysis has been superseeded, care to show where the field have actually responded to that in the past?
You will just get others using a slightly similar method to Steig who get similar results and they will claim O10 is wrong.
If you change a paper from saying why something is wrong and should not be used ever, to a paper that simply says this new method is better, people will use the method devised by their Team member over the “outsider” even though the “outsider” method is slightly better.
If the old method was valid and the new method is better, then ok. If the old method was flawed and should never be used by anyone, then that should be the point of the paper.
It’s really rather simple. Under anonymity Steig insisted upon a change to the paper, which the authors believe they were forced to adopt, and which at a later point Steig publicly attacked the paper for. You’re making the argument that the authors being reviewed are obligated to preserve anonymity even if the reviewer should disclose himself; if this is the case, then there must be an equal responsibility on the part of the reviewer not to make a burden of this anonymity, by planting bombs secretly and setting them off publicly. Given the lack of enforcement or recourse, Ryan pursued the only option open to him: a full disclosure. If the “culture” takes no precautions for or actions against the sort of actions taken by Steig, then the culture has some work to do in justifying that it be defended in the first place.
Perhaps an appreciation of the fragile nature of confidentiality, will convince future reviewers to use greater discretion.
Furthermore, it is generally conceded that in matters of importance, the appearance of propriety takes precedence over hurt feelings or the risk of looking “uninformed”. It may do to discuss under anonymity obscure matters with little immediate importance, but in a matter with grave political consequences, in which there is already a perception of cliques and conflicts of interest, many of us believe that it would be better for dialogue to be held in the open. The stakes are too high to trust in the objectivity of the editors who gate keep the science. You seem to say that climate scientists should not be asked to sacrifice their comfort; to which I can only assume that you forget the much greater sacrifices being demanded of the general public, on their account.
If he’s not qualified to comment on the statistical issues then he’s not qualified to render an opinion that will “improve” the paper regarding statistical issues. There’s not much left for his comment other than obfuscation.
Disagree all you want but that’s the simple truth. He should have rightfully stated “I’m not qualified” and left it at that. He did not.
Mark
Utter nonsense. The whole point of “disinterested” is someone that does not care about the outcome. The author of the paper being questioned has a clear conflict of interest and is absolutely the last person you’d want as a reviewer. His opinion is compromised not only by his position (his efforts are being tarnished) as well as his lack of understanding (he’s not an expert.)
There’s nothing wrong with offering the author the right of reply, an opinion published along with the rebuttal (a rebuttal of the rebuttal so to speak,) which is done quite often. But to say that someone with a stake in the outcome is the same person you want as an anonymous reviewer is silly if not disingenuous.
Mark
I’m not sure I follow you. This was a stats paper, with a limited number of review points.
colloquially phrased :
Have they buggered the data by poor or biased selection
Have they buggered the process by a bad choice of method
Have they buggered the results by poor execution of the method
have they buggered the conclusion because its not logically founded on the results.
Does it read like it was translated from Japanese to English by a Frenchman who spoke neither like the instructions in the Tamiya model kits I used to build as a kid….
Surely the discussion that advances science is the discussion between O’Donnel and Steig AFTER the publication of a lucidly written paper free from assorted buggerations NOT the attempt to prevent publication so that no discussion ensues at all?
Or am I missing some crucial point here….?
Hey I’ve got a good idea. If Steig never helped you in any way but was in fact simply obstructive, why not just recant your tainted published version of the paper and stand by your superb unadulterated first draft? It’s the only honorable thing to do under the circumstances.
Hey Roman….
Boro Nut–
Have you read the AMS policies on changing manuscripts after they are accepted? O’Donnell unilaterally reverting the manuscript back to v1 really would violate AMS policies: as in the written ones.
Authors are not allowed to get an “accept” and then send in whichever the heck version they like even if the authors thought the first version was better.
Isn’t it a bit late to start worrying about silly old fuddy-duddy policies? Why not just keep asking the same question over and over again until you get the answer you want, then do what you want anyway when you don’t. Nobody will ever notice. It will work.
“Please Miss, can we grass on Steig?”
“Under no circumstances, no”
“It was Steig”
“Dear god, what did I just tell you not to do?”
“Steig made me do it”
Over in the States, after the Patriot Act was passed and some people complained about invasion of privacy and loss of the protections of due process, the counter argument was “If you haven’t done anything wrong/illegal, then why do you care if the government reads your mail/listens to your phone calls/spies on you?”
If scientists engaged in peer review all perform honorably and honestly in their pursuit of scientific truth, then why should they care if the reviews are published or not? All too often non-scientists view this secrecy as indicative of sneaky underhanded dealings. Place the entire process under the clear light of public disclosure.
I consider the “nothing to hide” argument to be one of the most stupid arguments you can make. How free are you, once every step you take makes you think “what will my neighbors think of me?”, “am I in line with Joe Six-Pack?” Will they frown, that I like author X? Will they accept, that I collect gummibears? And then of course there is The Live of the Others. Highly recommended.
So why the confidentiality in peer review? Because it saves face and time. Both on part of the author and the reviewer. Obviously the authors can guff up and if the reviewer points it out the red faces are not seen by the public. Peer review takes a lot of time on part of the reviewer, already. It would take even more time (unpaid for, and even more importantly, taking away time from their own research), if they had to make 100% sure that everything they write is correct and that they don’t make a fool of themselves by asking the wrong questions. This time-consuming pressure is lifted by the confidentiality of the process. You can write short comments without spending hours to hone it down to perfection. You can ask questions that betray that you are not really aquainted with a subject (even these “stupid” questions can be valuable feedback to the authors).
The scientific community usually works quite well. Research is too complex for frauds to survive long. Dishonesty wastes too much of your valuable time. If you are caught dabling in dishonesty or fraud, say bye-bye to your career. Scientific papers don’t have to be the ultimate truth and explanation; they are stepstones towards that goal. They should be honest and competent, otherwise they waste the most precious resource of a researcher: time. Timewasters will find themselves to be ignored.
Peer review has worked for more than a century very well. Be careful, that you do not break a successful system, by blindly calling for “public disclosure”. Why do you think, you know better than the thousands of scientists working in the system?
“It would take even more time … if they had to make 100% sure that everything they write is correct.”
Yes, this seems to be the general understanding amongst climate scientists. Much better that we operate under a peer review process that employs reviewers utterly ignorant of the matter at hand – say, statistics – under the supervision of an editor who almost assuredly has no expertise in the matter whatsoever, and rely for their education upon the authorial team – who, unlike the reviewers, can afford to “take away time from their own research”.
If peer review truly is capable of turning gross ignorance into creditable science, then I agree with you that we should all do our best to preserve this divine institution.
Isn’t the peer review system really all about keeping ‘established’ scientists in prominent positions long past their sell by date?
I’m confused. If dishonesty and time-wasting are to be avoided, doe this not make confidentiality bad and transparency good?
I was talking about how peer review works, not how it can be gamed, if the editor allows it. Assuming an honest author and an honest reviewer, a confidential atmosphere, where you can voice frank and harsh criticism without having to give thoughts to repercussions, does speed up the process. The author does not have to decipher what the reviewer might have meant, the reviewer does not have to weigh every word as if a life dependet on it. The author should be clever enough to spot gaffs, if any, in the reasoning of the reviewer and point them out to the editor.
A dishonest reviewer can try to stall a paper, that’s where in good journals the editor steps in.
I simply do not see, how publishing the exchanges will enhance the process, especially if you remove anonymity, as well. And keep in mind, peer review is just the first step to get your results accepted by the scientific community. Bad papers sometimes elicit a direct reply. Most of the time, bad or uninteresting papers are simply being ignored, as nobody wants to waste his/her time writing a refutation.
Working in the commercial world I find the whole debate startling as to the lack of professionalism of the academic world. If there is an NDA or confidentiality agreement, then I comply with it. If not, then I am proud not to be anonymous as I am accountable for my views and for any errors. For this reason anonymous contributions to consultations are discounted, as are those that are unnecessarily marked as confidential. Why really would a reviewer want to be anonymous? Do academics routinely go round threatening physical violence to those who disagree with them?
Maybe not physical violence, but obstructing/reducing/destroying ones career is always on the table.
glacierman
But hey ho, that is life. Renewing a career is a great objective as it means a greater contribution because one learns from the cock up! This thread makes academics sound as precious as bureaucrats – but then they both depend on nice politicians diverting my earnings as a taxpayer to them. So why am I surprised!
This seems fairly simple from the policy document. Authors have no obligation to confidentiality. The editor was bound under Editors_2 but was released when Steig admitted his identity to the author. Steig was obliged to declare a conflict of interest under Reviewers 5, and realistically should have recused himself unless he was the only specialist capable of reviewing the submitted material.
Is anybody else not directly involved with the scientific community suffering from ethics overload? I, and I suspect many others, don’t actually care if Steig or O’Donnell or both or neither acted unethically. It seems to me that the key signal has been lost in ethics noise!
What remains crucial is whether the Steig et al 2009 paper uses methods which are statistically unsound as is claimed demonstrated by O’Donnell et al 2010.
O’Donnell concludes that Steig’s methodology has unacceptable sensitivity, i.e. it always produces warming in the Western Antarctic, whatever the sign of artificial trend adjustments to the chosen peninsula station data, a frankly risible result.
One or other of these papers is worthless, I would find it constructive if just a fraction of the mental energy expended on the ethics debate had been directed towards validating the statistics!
Is O’Donnell’s statistical claim justified?
“One or other of these papers is worthless, I would find it constructive if just a fraction of the mental energy expended on the ethics debate had been directed towards validating the statistics!”
Thats all that “really” matters.
In many areas of science and engineering, where the main question is ‘what is the truth’, and the idea of ‘replication of results’ is not foreign, peer review works. In these areas careers rise and fall based on the ability to get closer to the truth, not the ability to toe the party line.
The climatological world is apparently different, as set out in the climategate emails, and subsequently copiously demonstrated by Steig. Climatology has ‘redefined’ peer-review and ‘replication’.
It is absurd to suggest that authors treat review comments as highly confidential, the comments are intended to improve a ‘publication’, i.e. something intended for public consumption. It is absurd to suggest that speculation as to a reviewer’s identity is somehow contrary to the peer-review system. If a reviewer is endlessly saying ‘this work would be better if it was exactly like Steig et al’, the reviewer’s identity is obvious. The only reason to suggest that anonymity is being requested is to obscure the gaming of the system.
Something of an external perspective of all these shenanigans can be gleaned from Graham Stringer’s remark on being told that Briffa couldn’t even reproduce his own results. ‘That just isn’t science. It’s literature’, the exasperated MP (with a chemistry degree) exclaimed.
Why is replication of results relevant to a discussion of peer review? Because it allows others to assess the ‘truth’, ‘validity’, etc. of a given paper. Hence, unsurprisingly, while diligently gaming the peer-review system, the team have also been hard at work controlling access to raw data and algorithms.
Whatever the team and the Steig apologists are, they are not scientists.
>The world would be upside-down if it is wrong to say “Maybe it’s so-and-so” but okay to say “It is definitely so-and-so.”
There’s a reason I was focusing on the possible Jones review, as it takes away the Steig caused it escape hatch. JNG is clearly saying that that was unethical in his view, and it was this that caused him to add the extra clarification.
As to your followup questions, I’m a bit surprised they didn’t occur to you to ask JNG in the first place.
Re: MikeN (Feb 16 15:27),
Nielsen_Gammon expressed his opinion, but I still don’t understand why he expects his opinion to have the force of an ethical duty.
If it is an ethical duty, I would expect to see it in a Code of Conduct somewhere. No one has pointed me to such a code of conduct.
Let’s return to the open review situation and see whether his opinion makes sense there. Let’s consider the example of Reviewer 2 of Burger and Cubasch 2006 (CPD) and let’s suppose that this review was being discussed on a blog and some commenters speculated on the identity of Reviewer A. What code of conduct is being violated here?
Some journals (as readers have pointed out) have policies that require authors to treat review comments as confidential.
If an author submits to such a journal, I think that it is entirely reasonable for a journal to take the position that, by doing so, the author has agreed to comply with the published policies of the journal. If the journal has a policy requiring the author to maintain reviews as confidential, then that is an implied term of the submission agreement between the author and the journal.
A journal could also establish a policy prohibiting authors from speculating as to the identity of reviewers. To my knowledge, neither AMS nor anyone else has done so. If there was such a policy, that would apply to both “private” emails and public websites, though the former would be harder to enforce.
A Climategate incident involving confidentiality of reviews here:
Phil Jones had been a reviewer of McKitrick and Michaels 2004. He sent the editor’s decision letter and his review (and possibly others) to Mann. Mann thanked him and then asked Jones whether he could present the reviews to the chair of his department to bolster his personal situation.
Jones suggested that he present only from the editor’s decision letter:
There doesn’t seem to be any dispute that reviewers are obliged to maintain confidentiality of review comments. Did any of the climate scientists who now object to the outing of Steig as Reviewer A speak out against Jones sending Mann a confidential review for Mann’s use?
It’s well within the “bounds of ethical behavior” if you are on the Team.
Steve- I thought you did a decent enough job of criticizing it yourself.
Two things:
If John N-G is attempting to admonish SM, he has it wrong.
By the same token, I feel that interpreting JNG’s opinion to be some form of an edict or a commandment, to be parsed and pored over, is not necessary and probably not a good idea as well.
What JNG is saying is a simple thing – authors do not talk about reviewers – and this is a widely understood, unspoken rule in academic publishing. There is no need for any guidelines or policies to understand the value and the origin of such a code, and the reason for such a rule not being a part of any written-up policy is simply that it is considered a given, which everyone understands.
There are many aspects of scientific and academic peer-review practice that are archaic and 19th-century, in the sense of their un-writtenness. This does not make them backward, or inferior or outdated. Nor is it necessary to codify, lay out guidelines to choke and sclerose such practices.
The issue is very simple and unfortunately, one that involves honor, and therefore very difficult to speak about. Steig broke an unwritten code – he exploited his status as reviewer, not so much in causing a certain method to be used even as he did so, but in arguig on his blog that its consequent results favor his own paper. Would the arguement have been a problem at all, had Steig not been a reviewer of O’Donnell et al 2010? Hardly. But having given O’Donnell et al the impression of their having passed the stringent of tests – a review by the original author – Steig’s RC post was an attempt to destroy that very legitimacy. No wonder Ryan was incensed and he broke the other unwritten rule, in retaliation, so to speak.
Scientists do stuff like this, and as a comity of peers, the affected party decides to clench his/her teeth and take it saying “your turn will come buddy”. Ryan is an amateur scientist who derives no career support from his publishing. If he were to be a professional scientist, would Steig have extended the same discourtesy? Probably not. By the same token, Ryan would have taken the icepick up his back and kept mum, as well, which is what JNG wants him to do.
So, in my book, what Ryan did does not need any defending, because I do not judge these actions ‘by the book’, but by a code, which Steig fails first, and initiates the whole thing. But that is just ‘my book’. I don’t see value in the approach of a ‘technical’ defense of Ryan’s actions.
Many scientists and academicians exploit the unwrittenness and honor-bound nature of peer-review to gain advantage. It is just that the points where the rules are bent, are different between the academic world and the business world. People pull the exact trick that Steig pulled, all the time. You fight tooth and nail, backstab, loudly complain and use every dirty trick in the book, to publish and popularize a scientific position in the literature. And then you attend conferences looking like a saint giving talks about the very same ‘state of the literature’. You then justify your initial actions under the umbrella excuse of ‘fighting for the science’. The word ‘scientist’ and ‘honor’ have the exact same relationship as, ‘business’ and ‘honor’.
If JNG and AMS are now ‘changing’ their policies and ‘updating’ them, reflexively, to protect their reviewers, that would be a retrogressive step.
1. I’m not.
2. I agree.
I’m not Ryan, I don’t wear his shoes. I can only venture a guess about what I would have done, in the context of my own understanding of the ethics involved. As I’ve said before,
Dr N-G,
Thank your for your response!
So is the AMS updating its policies at all in light of this present affair, or is it that, that is a separate thing altogether?
I’m not aware that they are, but I wouldn’t normally be in the loop. If I were they, I’d wait until the dust settles.
Nielsen-Gammon says that it is unethical to inquire into the identity of a peer reviewer. Another Climategate incident presumably referring to von Storch et al 2004. Jones asks Mann if he is a reviewer as Jones wants to make sure that his comments get taken “on board”. Some journal policies require reviewers not to disclose the ongoing reviews to others. Osborn appears to have broken this by telling Jones that he is a reviewer, but is too scrupulous to let Jones influence his review.
http://www.eastangliaemails.com/emails.php?eid=402&filename=1077829152.txt
Wow, it sure is a good thing we had such thorough investigations into these incidents that cleared these scientists of any wrongdoing, or someone might start to think they were rigging the peer review process.
Yes I agree. If there’s one thing this blog lacks it’s that we don’t try to redefine the peer review process nearly enough.
That and the metaphysical concept of shame, obviously.
Steve – Policies differ from journal to journal. For the AMS, the present policy (which is consistent with my recollection of instructions to reviewers in past years, is
From the email you quote, I don’t know the journal in question, nor whether Osborn disclosed the information to the editor.
Ok–but you’ve also given opinions about what’s kosher based on cultural expectations, as well. Do the cases of Team conduct pass the smell test under that standard?
Some do, some don’t. And no, I don’t have time to write a book about it, nor investigate particular cases in enough detail that I’m comfortable passing judgment. Some of Steve’s criticisms of material in the Climategate emails seem convincing.
Nielsen-Gammon emailed me as follows, as he has still been unable to navigate the mysterious process of inserting a comment here.
Nielsen-Gammon says:
Unfortunately, this is not the cultural face of climate science today. The culture that climate science presents to the public through realclimate and the like is an attack culture in which critics are reviled as infidels, heretics, tools of the fossil fuel industry, etc.
Hardly anyone from the broader climate science community – Judy Curry and Peter Webster are notable exceptions – has publicly “frowned on” the realclimate culture. On the contrary. Most have endorsed it or acquiesced in it.
Allowing reviewers the privelege of remaining anonymous would make more sense if the authors were allowed the same privilege (pre-publication).
Post-publication, authors are judged by the scientific community on the quality of their work.
Why should reviewers comments not be judged the same?
Brilliant, insightful or useful review comments would then reflect favouably on the reviewer.
The pressure is on authors – science would benefit from having the same pressure on reviewers.
Brilliant, insightful or useful comments do reflect favourably on the reviewer, because the journal editor knows who wrote them. The editors are the judges, juries and executioners of papers, the reviewers are just the prosecutors. But when the reviewer happens to be on the stand one day, havíng the judge’s respect and good opinion may be useful. Sure this (otherwise admirable) system is open to abuse, but to the best of my knowledge only in ‘climate science’ has such abuse become endemic.
Personally I prefer journals with a double-blind policy, where the names of both authors and reviewers are confidential. My papers don’t get dismissed out of hand just because I work for a competing institution, and I feel no pressure to avoid getting on the wrong side of possibly ‘important’ authors of papers I review.
It’s any author’s choice: submit to a journal who’s editorial policies you like.
‘….because the journal editor knows who wrote them.’
How does this help the wider scientific community?
If the editor becomes the arbiter for what is or is not ‘best’ science I think we have a ‘Houston’ moment.
And there’s Briffa’s ‘…Confidentially I now need a hard and if required extensive case for rejecting…’. Seems like there is an extensive case to be made that the team have been gaming the peer review system, and caught red-handed doing it.
I wonder if ethical-Mann et al will comment on the complex world of team peer review principles?
Before getting drawn into too many other hypotheticals, let me simply list the considerations as I see them, based on some of the purposes of anonymous peer review. There may be others, and, as ethicists know, one’s choice in any specific situation may be affected by other, possibly competing, ethical considerations.
(1) What is the reasonable expectation of the reviewer for privacy or anonymity?
(2) Would the reviewer be subject to personal criticism or retribution for the content of his or her review?
(3) Would the expectation of anonymity be diluted, such that future reviewers would be less willing to undertake a review?
(4) Might future reviewers alter the content of their reviews out of a heightened concern that their identity and review contents would be made public?
Steve- John, I think that you’ve missed many of the essential fact elements of the case at hand e.g. public commentary by the reviewer both during and after the review process.
I’m so proud of myself! I just successfully posted a comment on Climate Audit! To do so, I had to change my name, email, and web site, while leaving the text of a previously vanished comment intact. Please now bear with me while I try to isolate the problem. With this post, I’ve changed my name.
With this post, I’ve changed my email address.
Yet, if I list my web site as http://atmo.tamu.edu/profile/JNielsen-Gammon , which is my professional home page, my post vanishes into the aether. Well, now that that’s settled, I’m happy to participate in the conversation.
Congratulations on poster-hood!
Really good to see that your first instinct is to do ‘sensitivity’ tests to isolate cause and effect.
Not enough of that in certain ‘scientific’ fields, IMHO.
It certainly doesn’t fly in engineering.
RR
Oops, the cross-post made my clumsy greeting less positive than I intended.
I salute your efforts and candor, Sir.
RR
Greetings John N-G. I suggest another consideration: Might a confidential reviewer abuse her position as such to publicly undermine an author?
(5) Should/Could there be any safeguards to protect this freedom-to-be-frank from being a cover-up for evil deeds?
[even Gavin Schmidt himself has conceded that there have been a few editor/reviewer gaffes where they ‘lost sight of the ball’, yet carrying negative impacts for the author(s), their work, and all with zero recompense (and no ‘punishment’ for the perpetrators].
Committing to be better at it next time seems a little weak at this point.
The editor is the primary safeguard against people who write “evil” reviews.
So it would seem that editors and the reviewers they select can indeed conspire against a paper. It also seems that this opportunity may never be removed, save for the resignation of a board (like was hoped for/seen in the climategate emails [note, the team was actually trying to end a journal for actually publishing something more, not refusing to publish]).
If this is the ‘best solution’ available (the trust of the editor), it is always true that an author can simply publish somewhere else in order to see daylight.
This brings up perhaps a fair point to ask though… Why do we have a characterization of ‘gray literature’ even within peer-review? It may be OT, but it seems there may be ethical fairness problems afoot when it comes to scientists trying to publish against editors who prejudicially dismiss their contributions, and then disparage the only places they may be able to go.
Most rejected papers are rejected because they’re not worth publishing. Some papers are rejected even though they’re worth publishing because something went wrong in the review process. I’ve had a paper rejected because of what I thought was a knuckle-headed reviewer whom I refused to cave in to. In such cases, the normal course of action is to submit it to a different journal, explaining the circumstances. If the paper gets rejected twice, it’s probably a problem with the me and the paper rather than the reviewers.
Scientists judge the quality of a journal by the quality of the papers within them. It doesn’t take them long to develop a sense of which journals have the highest standards for accuracy.
So there’s not a rule against a reviewer demanding a method be used, then turn around publicly, and criticize the method to discredit the author? How can that be?
Apparantly not. And more to the point, there doesn’t seem to be a rule against an author falsely accusing a reviewer of demanding that a method be used, only to defame him regardless either, as this sorry saga demonstrates.
Gord, Keith – It would certainly be improper, but that’s not my reading of the present situation.
Re: John N-G (Feb 16 16:56),
John, your ‘take’ on the linked bog appears to have missed (or assumed no significance to?) some important points.
Eric did not just request request an insistence on “something” more likely as you wrote on your blog. He made several statements indicating his strong preference… Statements clear enough that both the authors and editor understood him to want the iridge work to be done and placed in the paper. This perspective brings consistency to his reviews #2 and #3 (you too said you had the “impression” he was recommending iridge, at least in his #3 reply, and you caught the inconsistency with his later beyond-review statement.)
The interpretation of the strength of his review recommendation seems pretty important.
Two pertinent quotes not noted in your analysis on the other blog:
1) Steig literally suggested dropping TTLS. See the paragraph in question in Review #2 (page 4, para 4) and go to the middle of the paragraph:
“Perhaps, as the authors suggest, kgnd should not be used at all, but the results from the ‘iridge’ infilling should be used instead.” (note the pea/thimble: Steig says the authors suggested this switch. They didn’t. They wanted to use iridge for future work!)
2) Steig explicitly urged not leaving the iridge “future work” out of this paper; at least to me the final sentence of the paragraph is quite clear that he wanted them to do the “future work” now and incorporate iridge into the current paper.
John, did you miss these, or did you think they were inconsequential? Or…???
Thank you by the way for continuing to shed light on this topic. Very revealing for the ethos of at least one peer review community!
MrPete – I think your point #1 is implied in my summary. I interpret the text behind your point #2 a bit differently. Steig did not “[want] them to do the ‘future work’ now” but seems to believe that this work has already been done (since the results are quoted in draft #2). He also stated that the reason provided was “not a compelling reason” for focusing on iridge but left open the possibility that there was another, better reason for not doing so. (Being familiar with the language of reviews, this is obvious to me, but I recognize that it may not have been obvious to O’Donnell et al.)
If the editor understood him to want the iridge work to be done and placed in the paper, I missed it. Where is this in the Editor’s correspondence?
Dr. John N-G:
Here is a quote from the response to the last review by reviewer A to the Editor:
“Not only did we comply with the request to replace the TTLS / TSVD results with ridge regression (making this whole discussion moot insofar as our results and conclusions are concerned), we demonstrated that performing all of the requested additional cross-validation that was not present in S09 (including withholding the manned Byrd station) yielded the same results as before.”
It looks like the authors believed that replacing TTLS with iridge was made by reviewer A. The editor must have accepted this response as the paper was published after this response was accepted.
Here is a correspondence sent to all reviwers notifying them of the change made as requested:
General Note to All Reviewers
“Based on a request from one of the reviewers, we have agreed to incorporate our “most likely” reconstructions into the main text. These reconstructions do not infill the ground station data using TTLS; instead, they utilize ridge regression.”
Again, this was accepted by the editor and no reviewers challenged that this change was made at the request of one of the reviewers.
I seems everyone believed that the change was made as requested by Reviewer A, except Reviewer A, and his supporters.
Re: John N-G (Feb 17 12:06),
You ask,
Quite simply, the editor requested “further major revision.” Other than the TTLS-to-iRidge switch, I haven’t noticed any other requested major revision to the document, just minor fixups here and there. Do you?
To make progress on this subthread easier, later today (?) when I get a chance, I’ll create a difference-doc for Rev 2-to-3 similar to what I produced for 3-to-4. (It is not difficult with the tools I have; but my time is quite limited these days; my apologies!)
This entire episode is a great example of how communication relies not only on what is explicitly said, but also on the back channel so to speak — further understanding of the people, the overall topic, the community ethos and more. Various parties in this whole episode easily made and continue to make assumptions. This is a challenge even when everyone is 100% friendly and cooperative! I can tell stories of complete disconnect even when the (friendly) parties involved met for an entire week to ensure complete mutual understanding of a project, its terminology, etc etc. [BTW, this fact is used by security experts to easily obfuscate their communication.]
Another point. While John N-G does not presume that editors would be adverse in interest, some of us (definitely me) were extremely wary that the editor might be part of shall-we-say a Team for reasons that shouldn’t require a whole lot of explication. Editor Broccoli should have been aware of this.
Up to that point, the editor had consistently sided with Reviewer A (who we strongly believed to be part of the Team). Moreover, he blew off our earlier complaint about Reviewer A.
Mr. Pete,
I, too, agree with the assessment that the TTLS-to-iRidge switch was the only “major” revision in play. Perhaps others would disagree based on their read of the reviews and Dr. Broccoli’s intermediate decision, but at the time, all four of us were in agreement that the editor was fully expecting us to comply. Eric may not have meant for that to happen (knowing that would require reading his mind), but those were the events that his review set in motion.
MrPete – Your summary claimed that the authors and editor had the same understanding of whether iridge was required. The editor’s decision letter requesting “further major revision” says no such thing. Reviewer A recommended major revision and the editor agreed with the recommendation. That’s all.
It was going to be a major revision whether iridge or station-overlap or whatever was selected as the most likely results. Yes, all the authors interpreted this as a demand for iridge. But as you note in your comment, your assumptions are not necessarily the editor’s assumptions.
Re: John N-G (Feb 17 13:47),
We’ve already established that both the authors and reviewer A were looking at a TTLS-to-iRidge switch. I can’t find where reviewer A (the one asking for the revision) suggested any other candidate for the “pool.” Do you? Certainly the authors never suggested another candidate.
So, why do you assume there are more than these two candidates? This logic seems a bit unusual, but then perhaps this is another aspect of peer review practice with which I’m unfamiliar?
Retaining TTLS would not be a major revision, iRidge would.
So while in theory there could have been any number of options on the table, in reality there was only one “major revision” option.
Perhaps we are uncovering another subtlety of the peer review ethos? I’m honestly trying to stretch my imagination here.
Perhaps, even if there is no other logical meaning to a phrase, if a euphemism is used rather than the direct term, in the peer review world we must give the author the benefit of the doubt: when they used the euphemism, they were asking for an “out” so no one could ever say they meant what was obviously implied by their words.?
“Do you want strawberry or chocolate?”
“I prefer the red one.”
“Here you go, a nice strawberry shake…”
In my world: he asked for strawberry
In peer review: we can’t assume; must give him benefit of the doubt.
Perhaps a different observer might think he was asking for pomegranate? (smile)
Mr. Pete,
We used both iRidge and direct RLS, so theoretically, either could have been available. However, the verification stats associated with direct RLS were worse than iRidge (a point that was not at the time known to the reviewers or editor), so the perceived need to shift to a different method left iRidge as the only option. The comments made by Eric seemed to indicate that he had a preference for iRidge . . . though whether that preference was due merely to not understanding anything at all about direct RLS, I don’t know.
Re: Ryan O’Donnell (Feb 17 15:35),
What you share here tells me that from your position as one who was actually doing the work, you knew of more than one alternative to TTLS, viable or not. Yet you didn’t suggest extra alternatives; you didn’t place it “on the table” for consideration. Neither did Eric.
Thus, it would truly be coming from left field for the editor to imagine such a thing.
In essence, it’s as if you were serving the ice cream. As the manager, YOU know sometimes there are a few more flavors in the deep freeze in back. But they are not available today, not on the flavor board, and nobody has offered them to the customer.
So when the customer says “I want the red one” you are the only one in the room that knows you could in theory give him a red flavor other than the strawberry you offered.
The observer over in the corner doesn’t know any of that. All he sees is chocolate and strawberry, all he hears is the customer asking for the “red one.”
Great, now I’m hungry for some dessert. 🙂
Or Raspberry.
For what it is worth, comparisons of rev 2 and 3 are now online. Due to the significant amount of differences, two versions are in place.
This one shows the differences from Rev 2 perspective.
This one shows from a Rev 3 perspective.
Dr N-G, (please forgive for the appellation)
The real issue here is not strictly confined to one of journal policies and anonymous peer-review but in how it relates to instances where the academic debate weaves in and out of the forum of peer-reviewed literature and onto to author-controlled discussion forums such as Realclimate (and CA).
Quite so. Steve McIntyre has asked me to comment on Steig’s public attempt to not appear to be one of the reviewers. As I just noted above, I suspect I would have been angry at Steig for the reasons I list, but NOT for pretending not to have reviewed the manuscript.
If you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse, it is necessary for you to pretend to not be a reviewer. Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed.
John
What is your view once the reviewer has voluntarily revealed his identity to the author without pre-condition. Does the author still have a duty of confidentiality?
If I wanted to reveal the reviewer’s name publicly, I would ask his or her permission to do so. Points 1 through 4 here all apply. The duty remains not just to the reviewer, but to the review process. I would want to make absolutely certain that the reviewer was okay with his or her name being public.
I imagine there is good form in handling things like this (misappropriation of the review process, with the editor going along) in-house through the journal. Unfortunately for Steig, that wasn’t the forum he was using. I anticipate the journal will tell authors that they can’t do something like this, to save the editor further embarassments in the future.
I handled confidential information all the time and also reviewed. If someone asks for confidential information, at most I would tell them who would need to approve the disclosure. You don’t have to say you don’t know anything about it, just say who reviewed what paper is confidential.
^ This. Exactly.
You don’t see that as a potential conflict of interest? RC is a public relations web page set up specifically to shape public opinion. You are fine with doing reviews that you know are going to be debated publicly, then publishing stuff on a PR website to provide talking points to the media when you were involved in shaping the direction of the paper during review?
You do not see this as a conflict? Or you trust ES to be unbiased at one time, but then free to work the PR angle later?
So, for you, dishonest behaviour is less important than “the potential harm to the reviewer and the potential chilling of the review process.”
I suspect many University administrators might disagree with you. University policies and procedures often require scientists and others to behave with integrity. Honest behaviour is often given as an example of what is required from all employees to to meet the standards set by the policy. Are you suggesting that scientists should breach University policy to protect the reviwere and review process?
I think University administrators would wish to apply policy and procedures consistently and uniformly. Disciplinary procedures are recognised as having a normative influence on employee behaviour.Do you believe that Steig and any other University employee who behaves dishonestly should be subject to a disciplinary process?
Further, implied into every employment contract under British law is the obligation that an employee will do nothing to bring his/her employer into disrepute. do you consider that, were steig employed under a British contract he has, by his behaviour in the episode under discussion here, breached the terms of his contract? Or are you unable to say?
Quite so. Steve McIntyre has asked me to comment on Steig’s public attempt to not appear to be one of the reviewers. As I just noted above, I suspect I would have been angry at Steig for the reasons I list, but NOT for pretending not to have reviewed the manuscript.
If you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse, it is necessary for you to pretend to not be a reviewer. Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed.
…rats, WordPress didn’t like my new web site this time.
Or maybe I was just a tad too impatient the first time.
John N-G–
I could accept that Eric would not reveal himself as reviewer if that is all he did. But what I cannot accept as legitimate is this series of events:
1) Eric reviews the manuscript anonymously, describing it as flawed. The first manuscript suffers from serious shortcomings”, its “discussion is misleading
and confusing, and at least one critical aspect of the methodology is flawed”. The 2nd manuscript “retains several important flaw”. Manuscript 3 is specifically characterized with “because the manuscript remains flawed in a very basic way”
2) After he has obtained the final paper– which is nearly indistinguishable from manuscript 3, he not only fosters the impression the paper was new to him, but tells his audience he likes it. Specifically “P.S. For those actually interested, yes, I’ll have more to say about O’Donnell et al., but overall, I like it.–eric”
3) After some time, he begins a post presenting the criticisms he had in the review, but reminds readers that “At the end of my post last month on the history of Antarctic science I noted that I had an initial, generally favorable opinion of the paper by O’Donnell et al. in the Journal of Climate.”
Eric then proceeds to explain the serious flaws that he always saw in the paper.
Eric could very easily have retained anonymity and limited his “P.S.” to “P.S. For those actually interested, yes, I’ll have more to say about O’Donnell et al.,” without adding the — I think– highly misleading “but overall, I like it.” The later statement was entirely unnecessary to the goal of remaining anonymous and appears to have been deeply misleading.
It seems to me, that if a reviewer wishes a cloak of anonymity to permit harsh reviews of papers they think flawed, they should not later also use that same cloak to tell people they like the paper before waiting to revert to presenting the criticism they tried to advance during the peer review process.
Lucia –
1) His reviews became more and more favorable with each iteration. The third review closes with “it definitely has the potential to be a solid and oft-cited contribution” and “I do not think the manuscript will require more than minor re-writing.” Were I saying it, or receiving it, I would consider this to be very solid praise.
2) He must foster the impression that the paper was new to him to preserve his anonymity. Saying “Overall, I like it” is to me (and perhaps to Steig as well) entirely consistent and similar in meaning to what I quoted in item 1.
So, suppose you are an anonymous reviewer of a paper everybody wants you to comment on after it comes out, and you want to remain anonymous. What is the most ethical course of action? You must react publicly pretty much as you would were you seeing the paper for the first time.
I also don’t see that Steig’s public opinion of the paper changed very much between review 3, to “Overall, I like it”, to “in my view O’Donnell et al. is a perfectly acceptable addition to the literature.” If anything, the final statement is the strongest praise, because he is acquiescing to the judgment of the editor that version 3 didn’t require major revisions.
JNG–
I’d say less unfavorable. I don’t think they ever even reached “favorable”. All were described as flawed– the rev 3 “remains flawed in a very basic way and will need to be re-reviewd in any case.”
This is only true if we assumes the authors acquiese to his recommendation to change the method so as to change the results in a rather substantive way. It is true that night not require much rewriting in the of changing many sentences but they were being required to change what they considered to be the reported results in a way that represented a substantive change.
Fair enough. You think it’s entirely consistent.
But I don’t see “I like it” as consistent with recommending what amounts to a large substantive change– particularly if the author considers the change required to correct an important flaw.
Of course we can differ on this– but I just don’t see this as consistent with “I like it” but this is largely owing to my not accepting your view on 1.
.
As I said: It’s one thing to pretend the paper is new to you. It’s another to say you like a paper you described during review and continue to find as “flawed in a very basic way” The ethical course requires you to not write comments and blog posts saying you like. One may remain just as silent on whether or not one likes the paper as on the question of whether or not you were the reviewer.
Maybe. Or not. It may merely be an unwillingness to publicly criticize the editor even if he disagrees with him.
Obviously, Steig has no choice but to acquiesce to publication of the paper– it has been published. So, such acquiescence is rather meaningless.
In any case, he goes on to explain precisely why the paper he claimed to iike initially is flawed.
Merely not claiming to like initially would have made a big difference to my evaluation of Eric’s behavior. It need not make a difference to yours– but it does to mine.
“So, suppose you are an anonymous reviewer of a paper everybody wants you to comment on after it comes out, and you want to remain anonymous. What is the most ethical course of action? You must react publicly pretty much as you would were you seeing the paper for the first time.”
Firstly, as I understand it, your rationale for confidentiality is to protect the reviewers–so other potential reviewers will be willing to take on the job, and to give negative reviews. Under that rationale, you can argue it’s ok for a reviewer to deceive to preserve anonymity, but you can’t argue that there’s any ethical obligation to do so.
In fact, it doesn’t even follow from your policy that it’s ethical to deceive others to preserve anonymity. One might well imagine that the interest in confidentiality would justify certain sins of omission, but that the interest in honesty ethically obligates reviewers who value that anonymity to maintain a certain circumspection in their public comments, in order to avoid being put in a position where the two intersts conflict.
I just don’t see the conflict. Steig waited until after the paper was published to comment extensively on it, and his RealClimate post was based exclusively on the paper itself rather than any privileged information from the peer-review process.
I know that others interpret the post differently.
QBeamus,
I don’t see any problem with Eric wanting to raise criticisms after the paper came out, nor any problem with him wanting to raise the same criticisms he brought up in review – if he felt that those were insufficiently addressed in the review process. I don’t see any problem with him raising new criticisms, either. Being a reviewer on a paper does not obligate Eric to forever hold his peace if he truly disagrees with the paper.
What has irritated Lucia about Eric’s behavior is not quite the same as what irritated [to put it mildly] me about his behavior. I was willing to accept some of the deflection that he had done. I didn’t necessarily enjoy the deflection . . . but, by the same token, I doubt he enjoyed a bunch of amateurs posting endlessly about his paper in a not-so-complimentary fashion.
Insofar as the arguments that he posed in his RC post, there are some that are legitimate (discussions about warming at Byrd based on instrumental data and boreholes). I strongly disliked the fact that those arguments were used to imply our method was wrong – as his arguments are that other data (instead of the data used by both us and S09) show more warming. That makes no statement on the method. Rather, it implies that if Byrd really is warming at a high rate, that warming is not captured by the data sets used by both us and S09. Very different things. So I did resent the way the post was written. Again, I could have dealt with that. Eric feels strongly that the true temperature history of West Antarctica differs from our reconstruction, and I do not begrudge him that, nor do I begrudge him arguing that. What I disliked was the implication that S09’s method was “better” for West Antarctica (a conclusion not supported by the argument) and ours was “worse” (another conclusion not supported by the argument).
What set me off about Eric’s post (and Dr. N-G and I have different opinions in this regard) was Eric’s explicit statements that iRidge was the source of the differences. I know that he cannot know this, because I know that the statement is not true. I have done the calculations. I can (and will) show that the claim is definitively false. It is not possible for Eric to have checked his claim mathematically (unless he did it incorrectly), because if he had, he would have seen that the statement was unsupportable. When I contrast the absolute conclusion of his post with what was written in the review, I find them to be irreconcilable. I know Dr. N-G may disagree (as may others), but I do not see any other way to interpret this. Some of the proposed interpretations to attempt to reconcile the two disparate statements truly stretch incredulity. Occam says otherwise.
This is why I asked the hypotheticals that I did. If someone is not 100% certain, then they should not give the impression that they are. I cannot reconcile Eric’s self-admitted unfamiliarity with iRidge (as evidenced by his erroneous explanation of the GCV function and conflation of “regularization bias” with “standardization bias”) with a statement of certainty.
In my world, this is wrong. While I have a better appreciation of what is considered right and wrong in Eric’s world – and thus am willing to believe that Eric had no ill-intent (i.e., he honestly felt that his actions were ethical) – I honestly feel that they were wrong. I still feel that they were wrong.
Perhaps this is a partial explanation for the lack of trust among some segments of the lay population. What they feel is wrong is not shared by the academic community.
Ryan, if you makes you feel any better, the same sort of argument was made by Mann in connection with proxy reconstructions. They argued that our criticism of their method was “wrong” because someone else could “get” a stick using a different method.
As you well understand now, the other result has zero to do with methodological criticisms of the first method. We found that the “other” methods also had problems but those were different analyses.
Steve,
Yes. Whether the conflation is deliberate or not, the end result is that an accurate, coherent explanation of the advantages and disadvantages of a particular method is obscured. It is a detriment to progress . . . especially as the misinterpretations embed themselves into the literature via referencing-without-verifying.
I don’t think it is accurate to say that reviews got progressively better. Steig sent the following note to the editor at the time of his first review: “I emphasize that I think that a fundamentally reworked version of this manuscript could potentially provide a useful scientific contribution, and many of the points made do indeed have scientific merit. Indeed, the authors have done a very thorough analysis, and are to be congratulated on this.”
I don’t see this as materially different from his last review.
The quote Lucia gives from the 3rd review is key: “the manuscript
remains flawed in a very basic way and will need to be re-reviewed in any case.” This is the whole crux – Steig is really no closer to approving the current draft.
There a few other points about this episode that I found interesting:
-Steig originally claimed that the final submission was “markedly different from draft 3”. He later corrected this saying: “it has been pointed out that it’s not really very different”. This is an understatement – as Mr Pete demonstrated, there were barely a handful of changed words. So Steig performed a comprehensive review of version 3. He also performed a comprehensive review of the submitted paper for RealClimate. Yet, it had to be pointed out to him that the two papers were essentially identical. I know one’s memory can play tricks, but this seems odd.
– Reviewer D who seems quite knowledgeable and fair-minded, leads with “One particularly pertinent aspect of OʼDonnell et al.ʼs results is that they are consistent with two recurring, informal comments by the community that have emerged in the wake of the Steig et al. paper.” I find this particularly interesting and a little troubling. It appears that the climate community was well aware of methodological issues with S09. This is really not surprising as the community is quite accomplished and the S09 issues are not rocket science. Yet both prior and subsequent to the publication of O10, you would be hard pressed to find any climate scientists publicly expressing this “recurring” view (Trenberth being an exception). It is almost as if the community feels it needs to present a united face and no-one is allowed to break ranks. This seems unfortunate.
Let me get this straight. It is necessary to pretend not to be a reviewer that demanded a method be used, publicly ridicule the author for using his demanded method, with the intent to delegitimize the paper. Is this what you’re trying to convey?
If Steig had done this, it would have been seriously unethical. But that’s not my reading of the present situation.
Once again you garble the facts Keith, and no amount of repetition is going to make it any truer. Steig simply asked the authors to add a couple of additional sentences to address the known deficiencies with their chosen methodology that they themselves proposed and that they themselves thought was the best, just like any honest scientist would have done in the first place without having to be asked. Apparently that is tantamount to treason around these parts. Nobody asks us to justify our methods and gets away with it boyo.
Leaving Steig’s ethics out of the equation for a moment. What would be the rule for the unethical scientist? Shouldn’t the public be made aware?
Generally, no. Let me tell you about a different case in which I was one of the protagonists.
I was doing a literature search on a particular topic for a paper I was planning to write, and obtained a set of papers that satisfied my search criteria. I proceeded through the list, reading the abstracts and printing out those abstracts that seemed most relevant.
At about the 20th abstract, I read something that seemed familiar. I went back through my printouts, and found another abstract by the same author, who happened to be very famous within the field. The titles and journals were different, but the abstracts matched almost word for word.
I retrieved both papers and compared them. Aside from one new figure and for or five minor wording changes, they were identical. (Totally shocked at this point, I went back and looked at the past ten years of this author’s published work. I found no other duplications.
I contacted the publisher, informing her of the duplication and requesting that appropriate action be taken. (Conveniently, both journals were produced by the same publisher.)
I followed up later, and learned that the second paper had been withdrawn from the journal. As I was satisfied with this resolution of the matter, that was the end of it. The sanction fit the crime.
I did not (and do not) think that I should make the public aware. Nor did I publicize the name of the self-plagiarist among my fellow scientists. Unethical scientists tend to suffer a quiet death of loneliness.
So, in this case you didn’t keep the information that you had private, you told the publisher. And the publisher didn’t keep the information private, they publicly withdrew the paper in question. As far as the public were concerned, the problem, in this case the duplicated paper, was solved. In the Steig case, if the problems (obstructionism and publicly attacking a methodology pushed in review) could have been solved without needless ‘publicity’, I’m sure that everyone would be delighted.
(And let’s hope that the conversation doesn’t turn to the ingrained climatological habit of reusing portions of ‘seminal’ texts).
Dr. N-G:
I would like to take a moment to relay a slightly different story. I do not necessarily expect that you will answer the questions I pose. Just considering the questions without answering may illustrate what I am learning are significant differences between codes of ethics (written or unwritten) in academia vs. the commercial world.
Let us say that Bob is a DNA expert at a crime lab. The police send a sample to Bob for him to analyze. Bob happens to know the lead detective on the case (without needing to specify, let’s just say the crime was quite horrible), so the detective lets a few details drop about the investigation. During the conversation, Bob becomes absolutely, positively convinced that the suspect is guilty. Bob is so convinced that the suspect is guilty that he feels a moral obligation to ensure that this monster is not allowed to walk the streets freely.
After the detective leaves, Bob analyzes the provided sample. It’s not a great sample; it’s old and had been out in the sun awhile. But Bob manages to get some results. When he compares the sample to a known sample from the suspect, he obtains a probability of a positive match of 90%. This reinforces Bob’s honest belief that the suspect is guilty.
When trial time comes around, Bob is asked to testify. The suspect has obtained a rather slick and famous lawyer, and the trial is not going well for the prosecution. Eventually Bob is asked what the probability was for the match.
Bob says 99.99%.
The suspect is convicted.
Bob sleeps better at night because he knows that he kept a monster off the streets.
Now, ignoring the obvious legal issues with Bob’s testimony, did Bob conduct himself ethically?
I think this may help highlight some of the differences between “what is wrong” in the academic world and “what is wrong” elsewhere. The average person, when it comes to issues of science, has no recourse but to either accept at face value or reject at face value the statements of scientists. I include policymakers in this category, as few policymakers have the background to be able to question the statements except in the most superficial manner. It would seem unrealistic to assume that scientists do not know this.
So let’s say you have a case where a scientist is 90% sure that he is right. He knows there is some uncertainty, so he knows that what he believes has a chance at not being true. However, he feels a strong moral obligation that he must convince either the public or policymakers or both that he is right. So rather than say “90%”, he says “absolutely”.
Again ignoring the legal aspect, are the scientist’s actions ethical?
Returning to the situation at hand, I think most of us who have taken the opportunity to read the paper, the reviews, and the blog posts have general agreement that Eric’s understanding of iRidge falls short of “expert”. Exactly how much he understands, I do not know . . . but he certainly does not seem to consider himself an expert. This places him in a position where it is not possible for him to judge with certainty how use of this particular algorithm would affect the reconstruction. However, he says, to an audience that is also unlikely to have the knowledge to independently evaluate the claim:
“It is not surprising that O’Donnell et al (2010), by using iridge, do indeed appear to have dramatically underestimated long-term trends—the Byrd comparison leaves no other possible conclusion.”
While I understand that in academia this may not be considered unethical or even outside the norm, is it truly substantively any different than either of the two previous scenarios?
Now, returning once more to Bob.
Let’s say that someone finds out that Bob gave less-than-completely-truthful testimony. Let’s also say that this individual decided to inform Bob’s supervisor – but no one else. Bob’s supervisor looked into it – let’s say he looked at 50 previous cases Bob had given testimony on. He found no other discrepancies. The supervisor decided to simply fire Bob, and then contacted other crime labs and let them know what Bob had done – knowing that, if he did so, Bob’s career in forensic science would die a quiet, lonely death.
Are you comfortable with the supervisor’s decision to keep the incident from becoming a public spectacle?
There have been few times (maybe never before) where one of the physical sciences has shown the potential to have such a tremendous impact on policy decisions world-wide. These decisions affect all of us, yet few of us are qualified to independently judge whether the evidence in support of a particular policy initiative is sufficient.
We (generally) are therefore reduced entirely to trust – taking the word of the community at face value. The unwritten (and sometimes written) code of ethics that governs the interactions of academia with each other may be sufficient for that world. However, in interactions with the outside world, it may be woefully inadequate. A code that allows the expert the ability to state as fact something that may not be fact without also clearly stating the caveats is going to be interpreted as a systematic attempt to mislead.
I have significant difficulty with what I have recently come to understand is standard practice in academia. Though I am willing to accept that individuals may not intend to do anything wrong and may not believe they have done anything wrong, those practices are still at odds with my sense of what is right.
I do not know what your answers to my questions above will be (nor do I necessarily expect that you will answer them). You can safely consider them simply food-for-thought.
One last series of questions . . . let’s just say that the suspect in Bob’s case turned out to be truly guilty. Does that affect your answer as to whether Bob’s testimony was ethical?
What if the suspect turned out to be truly not guilty?
Ryan, that is food-for-thought indeed, but I should castigate you for speaking about the “truly guilty” and the “truly not guilty”.
We should remind ourselves that, while “truth” exists somewhere out there, we don’t have it. What we have are exactly our models of it, our observations of it. That’s all we have of it. We can never compare our iridge outputs or our TTLS outputs with Reality, because such a thing does not exist in the literature to compare with.
This detail is important, because then we have to decide without knowing the “true truth”, if something is ethical or not.
I think that in the short run, in your example, it was quite ethical for Bob to ensure the guy is jailed. In the long run, it establishes a precedent of subjectivity, of shortcuts, of lack of rigorous analysis, etc., that will degenerate as time goes on.
So what gives?
Luis: Less literal. More Rorschach.
“We have to go deeper” ?
Not really. Rather than trying to figure out how I would answer the questions or what my intended message was, just answer them for yourself. If you answer them the same way . . . why is the answer the same? They’re obviously different scenarios. If you answer them differently . . . why? What makes them different?
And whether the “truth” is actually “knowable” in any of the cases . . . simply assume that it is. Does knowing the truth affect your answers?
As an alternative, you could also just label it a bunch of meta-BS and move on. 😀
Knowing the truth only happens in omniscient narratives in novels, or in standard movies. The exact problem here is that you (and anyone else) don’t know the truth. This is why we are discussing science, after all, to pursue something that we don’t have.
If you are omniscient and judging the behavior of someone who doesn’t know the truth, but has clear beliefs about it, you can either judge his actions by some kind of an emotional understanding, or you will judge them the most objectively possible way.
Avoiding here to assume that there are self-evident truths like “all scientists should always speak all the truth, and never lie or exagerate”, etc., one could create a gedanken experiment in which a world where “Bobs” would exagerate the problem according to their own bias, creating thus their own set of consequences in either justice (the analogy) or science, would be compared with another world where “Bobs” would *not* exagerate the problem according to their own bias, creating thus *other* kinds of consequences.
We would then try to understand which kind of world would generate a more efficient system (a more just one), all else being equal, and independently of considerations of human impulses like “but unbiased reports are impossible”, and so on and so on.
I think that such comparison would bring us the correct answer. Intuitively we already know it. To be objective and pursue the data and not your “gut” is the correct path, it is the scientific one, despite all the emotional consequences that arise due to the apparent castration of such “weak” testimony.
It’s weakness is illusory, though. Through such weakness and candor, it builds trust upon its listeners. In the future, whenever such honest testimony states a “80%” probability of someone being guilty, may find the defendent be found guilty, while if the other path is chosen, even if you state a “99%” probability of guilt, perhaps you will run counter a good deal of skepticism.
In Ontario, dozens of convictions are being reversed because of excess zeal of scientist Charles Smith, a pathologist, particularly in regard to baby deaths now shown to be due to natural causes, where parents or babysitters were convicted of murder or manslaughter. Google “charles smith ontario baby death” for the sorry saga.
Luis, to me distinction should be made between “Ethics”, Values,and “Interests”.
“Ethical” considerations would seem to have been trumped by the relative importance of his “Values” considerations ( keeping the streets safe is more important than the ethical considerations of a subject of lesser “importance”).
Luis, maybe I could have been more clear.
Isn’t “Professional Ethics” more about following within given guidelines, something to do with following proper procedures as accepted in the field ?
If so, how can you break professional ethical code for any reason and not break it ?
I don’t understand your question more than the simple contradiction that it is.
The question is if whether is it okay for Bob to be machievellian or not. The trouble is, if one is to be seriously machievellic, for whom the “ends” are what matter, and not the “means”, one would reach two different conclusions, depending on the time frame.
In a short time-frame, perhaps it is “good” (in the machievellian sense) for Bob to be disingenuous. The most likely perpetrator is jailed and everyone’s safe, and so Bob should be commended and looked upon favourably if one discovers what he did.
In a longer time-frame, and considering that Bob’s actions do have consequences at the lack of objectivity of practices, corruption of testimony, etc., in an overall sense and in a longer timespan, this will signify that a lot of bad judgement will be made that was influenced by Bob’s actions being commended and looked upon favourably. Many more honest people will be jailed, and perhaps more criminals will be left loose. Justice will degradate.
So in the long-term, even in a machievellic perspective, such an end is not something to be cheering.
It’s because there is this confrontation between short-timespan huge wins versus longer timespans huge losses that we have codes of conduct, ethics, etc. and so on, so that we remember easily what is actually the best course of action, even if it doesn’t seem the quickest way to get what you want.
Re: Ryan O’Donnell (Feb 16 21:25),
[off-topic] As RyanO may or may not be aware, the hypothetical he describes has parallels with the Duke Lacrosse Rape Hoax case of 2006, with respect to certain stances taken by the Prosecutor, allied police and DNA lab personnel, and allied campus activists — all of whom were convinced of the guilt of the suspect students. [/off-topic]
Re: AMac (Feb 16 22:13),
there’s a slight link to climate. Climate scientist Tom Crowley condemned the Duke lacrosse team. I discussed the story at CA here.
Ryan – Your hypotheticals and non-hypotheticals deserve careful consideration. I’ll have a go with my personal opinions.
No.
No.
Yes. Because a lot could have happened during that month.
For example, there’s no evidence that he did the Byrd comparison during the review process. I am aware that the legitimacy of the Byrd comparison is a matter of considerable dispute, but that is beside the point. Steig believed the comparison; it is plausible that he looked at all the likely reasons for the poor comparison and eliminated all but one, one which was consistent with what you have assumed is his still-limited knowledge of the subject. Having eliminated (he thinks) all possible reasons for the poor comparison but one, he says as much.
Or, not being an expert in ridge regression, and having the published paper in hand, he may have turned to one or more experts in ridge regression and asked them for help in understanding it. Based on this expanded understanding of ridge regression and its weaknesses, he would have felt much more confident in relating the perceived shortcomings of the reconstruction to that particular technique.
Note that he has access to expertise following the publication of the paper that he does not have during the review process if he is keeping the fact that he is reviewing the paper confidential. He can talk to experts about ridge regression without them wondering why he’s suddenly curious about ridge regression. He can bring them particular examples and ask them whether his conclusions seem correct. Precisely because he has behaved ethically, after the paper is published he is in a better position to judge how the technique has affected the reconstruction.
These are just two possibilities. But both are consistent with how I might have proceeded, were I in Eric’s shoes.
Only if all those who were possibly improperly convicted get new trials.
No and no.
Your comments about trust in scientists touch on a matter that concerns me greatly. In many circumstances, the public trusts (or is asked to trust) scientist A to a much greater extent than other scientists would trust scientist A. I think this largely stems from a lack of understanding of the inner workings of science, which is an important aspect of the present controversy as well. I’ll have more to say on this in a few days.
Dr. N-G,
Thank you for considering. Our answers were nearly identical.
A parrallel but inverted example occurred in the OJ Simpson trial for murder.
The issue was whether spots of blood on OJ’s clothes contained a preservative. If it did the police had had placed the blood on OJ’s clothes, if not OJ had a lot of explaining to do.
The prosecution called their expert, a scientist, to give evidence that the blood spots contained no preservative – based on what I recollect to be some form of chromatograph. All agreed that if there was preservative in the blood there would be spikes at points X and Y. A blown up chromatograph was placed before the jury. It more or less flatlined, except at X and Y where there were towering spikes. Johnny Cochrane cross-examined.He pointed at these spikes and asks what they were. The prosecution scientist explained to the perplexed jury, and to hoots of laughter from television audiences around the world, why those spikes weren’t really there.
No doubt he acted ethically as a scientist. He simply misunderstood the rules. The rules were the rules of the court room not the rules of the common room. His explanation was an over the top exaggeration, since p < 0.9 the preservative must be considered as absent.
Perhaps we are discussing a similar confusion. Scientists have grown used to the courtesies and rituals of the common room evolved to resolve spats over arcane and unimportant matters in a manner no one finds disagreeable. This time the stakes are high and most of the stakeholders have never seen a comon room. More exacting and more uncomfortable rules apply. Their comfort can no longer be the benchmark for ethical conduct.
>neither Dr N-G (nor anyone else) suggested that such speculations might be a violation of ethical principles.
Yes, if this were a serious breach I would have expected a comment on that line to show up somewhere.
I do think that speculating on the identity of reviewers is in violation of the policy. You agree that JNG has summarized AMS policy in the first paragraph accurately.
>if there is other information that makes it possible to discern the identity of the reviewer, such information should be redacted
I think posting a review and doing a forensic analysis that the reviewer is Jones violates this policy. You used information that makes it possible to discern the identity of the reviewer.
I don’t see a summary of this confidentiality policy in the authorseditorsreviewers guide you linked.
A PHD candidate shows his thesis to his student advisor, who reads it and says to the candidate, “Joe, I see you are using the X theorum in computing your results, although you do mention that the Y theorum is another alternative. Actually I think the Y theorum is probably a better approach, although there may be some problems with it. But don’t tell anyone that I said so.”
The candidate rewrites his paper using the Y theorum and submits it to the Chairman of the department. Later he is called before the committee to defend his thesis. The student advisor is in the room and when the candidate finishes his presentation, the student advisor raises his hand and says, “Why in the world did you use the Y theorum to compute your results? Totally wrong!”
The candidate looks at the student advisor and says: “You SOB, you were the one who advised me that the Y theorum was a better approach.”
Blame him? Not me.
“A PHD candidate shows his thesis to his student advisor…”
So in this scenario it’s Steig who’s the ingenue PHD candidate, and O’Donnell his advisor is it? Because that makes no sense, yet that would be their relative authority respecting statistical methodology, at least according to both Steig and O’Donnell that is.
[snip]
[RomanM: I know it is difficult for you to control yourself, but that type of language is out of bounds.]
John N-G “If you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse, it is necessary for you to pretend to not be a reviewer. Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed.”
This seems fairly reasonable to me and seems consistent with what I generally expect from reviews (for the reasons outlined – mostly retribution – science can be a contact sport).
As for the party foul, given that there are some new folks wading into unfamiliar territory, why not simply apologize for the error and get on with life?
Interesting how guidelines can have subtle differences. Here’s one from the astronomy community.
http://iopscience.iop.org/1538-3881/page/Ethics%20policy
Note that, although they refer to disclosing referee reports by authors as inappropriate, the only penalty is loss of confidentiality rights by the author.
May not be a bad trade-off.
I posted this comment at the wrong thread originally.
I am not at all certain I know what the objections are to the “revelation”. I think those of us who have studied the reviews of Reviewer A and had an initial interest in the subject matter from the analysis of S(09) found those public reviews helpful in better understanding some of the more nuanced statements in S(09) , the literature that was discussed at length in reviews and pertained to S(09) and O(10) and materials that put were forth in the initial O(10) submission but not published. I think much can be learned in general by making the review process public even with the reviewers being anonymous. Is anyone protesting about making the reviews public? Surely reviews can be an important learning tool and particularly so in taking advantage of the communication that the internet permits.
.
If we get by making the reviews and submissions public, we have only to deal with revealing the identity of Reviewer A. I assume that no bylaws or rules of the publisher were infringed with the revelation and since it is difficult to get an agreement on the terms of the gentlemen’s agreement involved all that is left is to determine if anyone was individually injured by the revelation.
.
O’Donnell made the revelation and I assume, save his wanting to injure himself, he was not hurt by it. That leaves Eric Steig. First we have a situation where the reviews when first made public were going to raise some very serious speculation by interested parties that Steig was Reviewer A. Given blogs propensity to search for all the clues in cases like this one there was a good chance most would have a strong inclination who the reviewer was. Certainly Reviewer A revealed a strong ownership to S(09) through his reviews.
.
The issue here in my mind then comes down to what harm was done to Steig by interested parties knowing that he was Reviewer A. It was rather obvious that Steig had specifically a strong interest in maintaining as much of the West Antarctica warming claimed in S(09) as he could. He wanted every avenue explored in the review/reply process that he could get out of that process to afford alternative methods to find one that might have the greatest WA warming, i.e. in his mind the “correct” warming. Has not that been the thought process revealed outside the review process by Steig? Was there really anything in the review we know about Steig and his position on S909) versus O910) that he had not already revealed to the public? Actually the way Steig attempted to make points in the reviews versus his non review approach would I think give higher grades (given we forgive the tenacious tenor of his reviews) for Steig, the reviewer, than Steig, the blog commenter. I, therefore, judge that any harm that has come to Steig has been from his actions outside the peer-review process.
“(1) What is the reasonable expectation of the reviewer for privacy or anonymity?
(2) Would the reviewer be subject to personal criticism or retribution for the content of his or her review?
(3) Would the expectation of anonymity be diluted, such that future reviewers would be less willing to undertake a review?
(4) Might future reviewers alter the content of their reviews out of a heightened concern that their identity and review contents would be made public?”
Retribution or personal criticism for a review with tangible harm to the reviewer would indicate more fundamental problems with the process in my estimation. And why would fundamental problems be expected to be solved or even obscured by secrecy – to the contrary, in my estimation, they would be made worse. You are obviously treating the symptoms.
Indeed. Because scientists are human, and there doesn’t seem to be a good way around that.
‘…and there doesn’t seem to be a good way around that’.
Given another chance would you rephrase that?
Nope.
It seems clear that through Steig’s assistance as Reviwer A, O’Donnell et al were able to produce a better paper.
From what I can see, O’Donnell et al have produced what looks like a better result than did Steig, largely because of Steig. If Steig did not exist, O’Donnell et al would not have produced the paper they did.
O’Donnell should remember Newton’s words, thank Steig gratiously for his assistance in helping him and move on.
The current argument is a disservice to all involved and it will take the bigger man to walk away.
Secrecy breeds mistrust. Reviews carry much more weight if the reviewer’s credentials appear on the document. Otherwise, who is to say if the paper was peer reviewed or pal reviewed.
Anonymity is important both so that it is the relevance and insight of the review which is important (rather than if it’s a renowned professor or unknown post-doc), and so that reviewers don’t hold back on their criticisms in fear of reprisal and are more willing to stick their necks out with a range of suggestions which may or may not be useful without fear of getting egg on their face if the suggestions turn out to be unimportant with more considered inspection.
Having been thru the flawed peer review system, I would like to say that it has been my experience that typically revewers that are politically motivated to kill a manuscript issue reviews that are very negative, but have no scientific basis. It is the Editor’s responsibility to choose unbiased reviewers (assumes he is unbiased
which is not always the case). It is also the Editor’s responsibility to discount any obviously biased or unscientific reviews. As usual both of these responsibilities were not carried out by the Editor for Ryan’s manuscript. Nor did Reviewer A recuse himself.
I would also mention that when publishing an article that had a direct impact on one of our Reviewers previous work, the famous scientist carefully checked the content, made helpful suggestion, and identified himself.
There is a considerable difference between the two types of scientists both in honesty and competence.
Jerry
Steig had the temerity to compare his review of O’Donnell et al 2010 with my review of Wahl and Ammann (the Jesus Paper). The situations were entirely different and point both to the role of the editor and why the distrust arises.
Steig’s reviews were the only ones calling for major revisions. Despite the fact that no unconflicted reviewers ever called for major revisions, Broccoli required two major revisions.
Steig complained that he wasn’t sent the responses to the Third Reviews, but he did receive responses to his first two reviews, the correspondence to that point totaling about 70-75 pages or so.
Compare the situation with Wahl and Ammann – see CA posts here, here, here for correspondence and reviews. No major or, to my knowledge, even minor changes were required as a result of my review. I was not sent a Response from Wahl and AMmann; indeed, I do not know if Schneider even required them to provide a response. I was not provided a copy of any subsequent version (there seem to have been two).
At the time, I was relatively inexperienced in academic practices and the idea of someone adverse interest acting as a reviewer seemed very odd to me. As I’ve observed elsewhere, I think that a much better practice was that of GRL editor Saiers in respect of comments and, if the situation arose again, I would recommend that the editor handle the situation as Saiers had done. I sent my review to Schneider as an email and I don’t know whether it was sent to Wahl as anonymous, non-anonymous or not at all. It seems to me that best practice is that if the editor insists on a conflicted reviewer, the conflicted reviewer should sign the review so that the responding authors are fully informed.
Prior to submitting my review, I asked Wahl and AMmann to provide verification r2 statistics – then a controversial issue since they had claimed to have shown that all our results concerning MBH were “unfounded”. (That its verification r2 was ~0 was a key result – mentioned ironically in the first NAS letter to the Barton COmmittee offering to take carriage.) Amazingly they refused. They also dissembled on the status of their concurrent submission to GRL on which key results depended. (The GRL submission had been rejected.) I met AMmann for lunch in December 2005 and he once again refused to disclose the verification statistics. I strongly urged him not to adopt that course of action. The verification statistics were reported only after an academic misconduct complaint.
BTW Jones’ review of Wahl and Ammann is in the Climategate emails and is a puffball review. This is another problem. While we hear much talk of the rigors of academic peer review for the Team, there is nothing by which I or anyone else can verify this and the available public evidence (Jones’ puffball reviews of Mann, Schmidt, Wahl) suggests that the reviews are puffs. Some of the errors in Steig et al e.g. confusing eigenvalue separability with eigenvalue significance or placing physical significance on Chladni patterns, (especially the former), likewise do not suggest particularly insightful reviews.
I’m thankful for this insight, as I was wondering what your comments would be regarding this repeated comparison made by Steig.
I suppose in light of the present conversation, all the actions that took place inre Wahl & Ammann were done under the accepting (and unassailable) eye of the editor.
Steve,
I was very naive when I began to publish. I thought an Editor would be unbiased and competent. Unfortunately that is not always the case, especially when the Editor is receiving funding that might be impacted adversely by the manuscript. Also,if the Editor is not a strong scientist (many take the position just to bolster their resume), he/she might be intimidated by a stronger colleague’s review even if that review is politically motivated. Unfortunately, especially in the less rigorous sciences, this hapens much too frequently.
Jerry
It seems at least Gavin Schmidt is willing to acknowledge that there are times when the editor can participate in some less-than-stellar behavior with regard to submitted publications…but at the same time still maintains their supremacy:
“[Response: … In dealing with comments on a few of my papers, it has been clear (to me at least) that the comment [by a reviewer] had nothing very much to add, or was greatly deficient in some way – in those cases, I’ve said so plainly. Some of those comments were published, others were not – but in every case it is the editor’s decision, not the reviewer’s. Editors are not stupid, and they know they need to balance multiple interests at once. On the whole they do it well, but there are clearly some occasions when they lose the ball (the Soon/Baliunas fiasco, a couple of comments James Annan was involved with, etc). – gavin]”
Steve,
Unless something has changed, I thought the American Met Society always required that any references either have been published or in press?
Jerry
What kind of namby-pamby stuff is this???
Seems to me like it could be a lot more productive if you spent all this time doing science instead of gossiping and crying. I’m sure hundreds of scientists in ALL fields of science have had drama at one time or another with a paper in review.
R. Lee has some tissues for ya.
John N-G wrote:
You have spoken elsewhere in this thread of the “culture” in the realm of peer review. Notwithstanding Lancet editor, Richard Horton’s recent observation (cf Muir Russell report) that:
As an outsider to this “culture”, could you describe for me the criteria by which those more conversant with this “culture” than I would determine that a paper be deemed as “not worth publishing”.
So if, for example, a highly respected journal were to publish a cover story featuring a paper that was very quickly demonstrated to lack any scientific “novelty” (apart from artefacts that derived solely from faulty math and/or methodology and/or inept application of statistics by one who has admitted that it is not her/his forté)…
a) What would this suggest about the “standards for accuracy” of such a journal amongst those who are more familiar with the “culture” than I might be?
b) Should such a journal have chosen to remain silent and/or ignore an unchallenged peer-reviewed paper highlighting the flaws of the original, is this within the range of acceptable norms of the “culture”?
c) To use a baseball analogy, how many such strikes before the acceptable norms of the “culture” would dictate that the journal was “out”?
Regarding peer review efficacy, I agree that it’s poor. (Sort of like the efficacy of democratic government.) Mostly it’s a case of papers that should not be published (in their final form) getting published anyway rather than good papers not making it through. This characterization doesn’t apply to journals such as Nature and Science, which heavily weight factors other than the intrinsic accuracy of the paper in their publication decisions.
Regarding your first question, you’d have to do a survey of editors. In the relatively small sample of papers I’ve reviewed, they tend to be either papers that don’t say anything useful (“Hey, look, we simulated a storm with our model”), use a method that is known to be poor, try to use the correct method but have the wrong interpretation of the results, or are just fundamentally misguided (“Hey, this technique everybody uses is wrong because it doesn’t tell you A” when in reality nobody ever uses it for that purpose). Almost all submitted papers need to be modified to some extent; the decision for immediate rejection depends on whether there’s much hope that the authors are capable of whipping the paper into shape. Even if immediate rejection is recommended, the authors should come away with two or three reviews explaining what’s wrong with the paper and how it might be fixed.
a) In my case, it would confirm my earlier impression that papers in that journal seem to be disproportionally flawed.
b) I refuse to answer on the grounds that I might be sued.
c) I suspect that if you did a poll among scientists regarding which journals were (a) most prestigious and (b) most reliable, you’d get different lists.
I’d like to add to the list of reasons papers get rejected. Bear in mind– my area is fluid mechanics, with an engineering slant focusing mostly on multiphase flow. Back when I was doing research, and being sent papers to review, I rejected some quite bad papers. These included:
1) A fairly complicated analysis whose major result stemmed from mis-application of conservation of momentum to a control volume. It wasn’t that complicated in so much as they had assumed 1-d. But owing to forgetting the sign convention tallied momentum “exiting” the control volume as “entering”. In the process, they believed they had explained why a certain term appeared in flows through porous media. The paper was submitted to a journal whose focus was multiphase flow rather than porous media. (Owing to overlap in aspects of modeling, this is uncommon.) I suggested something they might look at after they correct the mis-application of conservation of momentum, but advised rejection not revision. (Of course the authors could, if they wish, correct their general methodology and resubmit at the same journal or other journal.)
2) A paper that was very similar — and I mean very similar to one published two or three years previously in the journal to which it was submitted. (I got a special thanks for noticing this as the other two reviewers gave the paper favorable reviews, which it would have merited had it not been very similar to the previous paper.)
3) Papers that appeared to represent useful work that is specific to an industrial application focusing on those aspects that would assist in a specific design but without quite enough additional effort to elucidate an issue of more general interest. These might usefully appear in some sort of periodical, but would be mis-placed in the journal to which they were submitted.
The 3rd represents a sizeable fraction of papers rejected from the more engineering-research oriented journals. It is also the agonizing class of papers for people in the application/development side of R&D because it is not true that one can “always” find an appropriate journal where you can publish your computer runs or even industry type experiments to understand something about intricacies of flow inside toilet bowl design ABC and the engineer involved probably cannot get his supervisors to permit him to devote sufficient time and company resources to “find” the publishable problem that is the “nut” of the work.
More complicated situations:
I was occasionally asked to reviews papers written by foreign authors whose English was so poor I could not understand the paper at all. if these did not seem to fall in category (3) above, I suggested the authors find a good copy editor trying to figure out how to say this kindly. These sometimes came back and were ok. Usually, I never saw these again.
One paper that I returned to the editor stating that I felt a conflict of interest prevented me from reviewing it.
Other than that, I mostly saw fairly decent papers on topics that matched the journal topics pretty well. Sometimes I advised publication with minor revisions– and I might insist on seeing the rewrite or I might not. Sometimes, I advised a second round with major revisions. I consider a “favorable” review to be one in which I said the paper should be published with minor corrections leaving the decision to correct entirely up to the authors.
Thanks, for your response …
and
A survey and a poll?! Interesting suggestions – and when my “Big Oil” cheque arrives, I think they would be well worth doing. But in the meantime, I can only handle the administration of one survey at a time … and I’ve already got one running:
Calling all scientists – an invitation to speak for yourself!
As for your:
Reality check: Was this in reply to my “b) Should such a journal have chosen to remain silent and/or ignore an unchallenged peer-reviewed paper highlighting the flaws of the original, is this within the range of acceptable norms of the “culture”?
If so, surely you jest, sir!
I must be dead set dumb, because it all seems entirely clear to me :-
1. Ryan was wrong. He shouldn’t have ‘outed’ Eric without permission. Just good manners for mine.
2. Having outed Eric, posting “Reviewer A’s” reviews was also wrong. Not so wrong if they could still claim the veneer of anonymity.
3. Eric was wrong. “Duplicitous” covers it nicely.
Can we not all get back to the interesting stuff – like whether the planet is about to burn / freeze / whatever – and put this behind us?
Surely enough is enough? We now all have – I’d have thought – fairly well informed opinions. The “data” is only too public. The “methods” are thoroughly documented.
We will all judge the protagonists on what they have done to date and on how that colours our view of their actions in the future.
I disagree. Since Eric went public with a lot of negative things to say about O’Donnell, there was no reason at all to keep his reviewer status anonymous for two reasons:
1. Eric didn’t have to confess to O’Donnell that he was indeed Reviewer A. The fact that he did is suspect and unethical itself, when in conjunction asking Ryan to not “tell anyone” about it. It smells of manipulation, and puts Ryan in a strange awkward place. Ryan does not have to abide to such a thing, specially after 88 pages of hell review.
2. The reason for anonymity is to prevent scientists for being shy in their criticisms, and giving “all they got” against the paper, without fear of social backlash in some sort. But Eric evidently does not fear this (rather is quite happy in doing such against Ryan, McIntyre et al), and goes public with all he’s got. Surely Ryan ousting Eric will not cost him any social damages for being a contrarian reviewer, since everybody knew that was already the case with Eric’s RC post.
3. When there is a form of duplicity between Reviewer’s A attitude and considerations and then Eric Steig’s criticisms, the only way to expose such a thing is to oust Eric. There is no other way to do this. Should Ryan “suffer alone” in this? Why should he, after being backstabbed?
In conclusion, and in the form of wide speculation, I’d venture that it wouldn’t be far off of my impression that Eric exposed himself to Ryan in bad faith, exactly to prevent Ryan from writing the piece he did here.
Consider. Had Eric not exposed himself to Ryan, while advising him to shut up about it, Ryan would have been found without any kind of ethical barriers to counter Eric’s last post with the statements of “Reviewer A”, since everyone “knows” that reviewer A was part of the “Team” (even other reviewers “knew” it, as it is plainly clear in their own messages inside the reviews).
Just a suggestion.
Sorry I ended up writing three reasons 😉
I cannot see how publication of anonymous reviews by the authors can be considered unethical. Unless the reviews contain some original information for which the reviewers claim their copyright (in this case this information should be clearly marked so), their publication will just make the peer-review process more transparent and understandable to everybody.
I can report the following case when we intended to publish reviews resutling from our GRL submission. Having asked twice if the AGU or GRL have any reservations wrt our intention, we did not receive any reply. This, as well as the case with the AMS guidelines that are currently being changed, suggest that the Internet era has brought new realities to the system. Will the system become more transparent or less so?
The Royal Society, for example, has the following ethics rules:
Since I have been both an author and referee of Royal Society journals, I got interested about these rules and approached several authorities in the Society with a query: what these exceptional circumstances are and who decides whether any particular circumstances are exceptional. I never received a reply.
If a science is demonstrating apparent success, people will not care how the scientists have achieved that, with help of peer-review or without it. If people become unsatisified with the results they are presented to, it is natural that a discussion is initiated about the quality control processes (including peer-review) that are used in the science in question to produce those results.
Some people think that (in spite of all its drawbacks) the overall impact of the peer-review has been positive — indeed, we apparently witness scientific progress in many areas. But others may well dare an opinion that science has been thriving despite the modern peer-review as we know it. One should also take into account that the unwritten rules of peer-review may differ in different disciplines. Personally, I agree with Dr. Richard Smith that the peer-review system should be itself studied from a scientific viewpoint.
Anastassia:
Your post crystallized the issue for me with its emphasis on “exceptional circumstances”. Surely, Steig’s public critique of OLM10 constitutes “exceptional circumstances” given Ryan’s understanding of what Steig knew. (The fact that Steig had not seen the authors’ response to his third review does not change the issue – just as it does not change the fact that it would have been more appropriate for both Steig and O’Donnell to share their blog posts with each other before going public.)
I find it passing odd that lying or dissembling is being condoned and presented as “ethical” in order to avoid breaching a journal’s editorial policy. Surely this suggests that the policy itself is flawed. Note that the Royal Society’s policy removes this internal contradiction.
Finally, what recourse does a reviewer or author have when an editor handles a manuscript/review in an inappropriate fashion. This to my mind (I hope) fall under the “exceptional circumstances” and the remedies can surely include public disclosure of reviews, comments and correspondence.
Just a question. The iridge results were identified for future work. Will the work of the original paper now be presented as “future work” (speaking of the TTLS and reintroducing the “community” to chladni patterns formally)?
In answer to my:
“You are obviously treating the symptoms.”
John N-G replied:
“Indeed. Because scientists are human, and there doesn’t seem to be a good way around that.”
I find that reply sad and funny at the same time – if I am correct in interpreting that to mean that since humans must be part of the process and humans have these weaknesses we must treat those weaknesses by not revealing them to the public or interested parties of the process.
Fundamental problems are best approached and resolved by allowing them to see the light of day. As, for an example, I note the Nixon tapes and what they revealed about, not just Nixon, but politicians in general. I judge a number of people thought a President would never engage in those thought processes and with the language used and provided a very healthy skepticism of our politicians after that revelation. It certainly has not had any effects on making politicians scarce for running for that office or any others for that matter.
Lancet. Autism. Transparency.
<>
Not if the other party tries to use the agreement to harm O’Donnell.
O’Donnell made the agreement to protect Steig at Steig’s request. He did not make the agreement to allow himself to be attacked, and would not have agreed if Steig had said that was the purpose of the agreement.
Steig violated the agreement by using the confidentiality agreement as a shield to hide behind and attack O’Donnell.
At that point the agreement was violated and O’Donnell was not longer bound to maintain confidentiality.
Honestly, if your a cool guy, you’d keep that in confidence. However I can see both sides of the situation. I would say that its up to the discretion of two in conflict. Kinda like how a police officer has the most discretion when they are in the field.
Forgive my total stupidity…
But on the “Trackbacks” … “#3” showing the post on “Another NewYorkTimes” seems to be lifted word-for-word and placed into that blog without any attribution. Is that a problem?
7 Trackbacks
[…] Are Review Comments Confidential? Over the past week, there has been considerable controversy over the outing of Eric Steig as Reviewer A, together with […] […]
[…] Nielsen-Gammon writes as follows in a comment in the preceding post: If you are a reviewer and wish to remain confidential while remaining […]
[…] Nielsen-Gammon writes as follows in a comment in the preceding post: If you are a reviewer and wish to remain confidential while remaining […]
[…] Are Review Comments Confidential? at Climate Audit […]
[…] Nielsen-Gammon has warned us that it would be unethical within the climate science community to speculate on the identity of […]
[…] papers, such as O’Donnell’s rebuttal to Steig et al (Antarctica is warming) where Steig himself inappropriately served as a reviewer, and a hostile one at […]
[…] skeptical papers, such as O’Donnell’s rebuttal to Steig et al (Antarctica is warming) where Steig himself inappropriately served as a reviewer, and a hostile one at […]