John Nielsen-Gammon has articulated a doctrine of what might be termed justified disingenuousness as applying to climate scientists acting as reviewers. I criticized this doctrine in yesterday’s post.
In comments to that post, Nielsen-Gammon said that I made unrealistic assumptions about the academic world, that I was (in effect) too idealistic, perhaps even a pollyanna about the hard truths of modern academic climate science. (I’m expressing his points in more lively terms, but I think that the characterization is fair. Readers in doubt of my characterization are asked to review N-G’s comments here.) Nielsen-Gammon paints a picture of a world where scientists fear reprisals, if they have the temerity to criticize the dons. Where safety exists only in anonymity. Accordingly, Nielsen-Gammon has harsh words for those who break the sanctity of the witness protection program. While many CA readers have criticized the witness protection program, Nielsen-Gammon’s words of caution deserve attention, lest the cure be worse than the disease.
Fear of reprisals is definitely at large in modern climate science. Some surprisingly prominent climate scientists have told me privately – and only under the condition that their identity remain confidential – that, in their opinion, our criticism of Team-style proxy reconstructions had ended that line of argument and that 1000-year proxy reconstructions would be unable to advance without the development of more reliable proxies. Despite their apparent prominence, for fear of reprisals, they were unwilling to express these views publicly or to allow me to use their names in support.
Police undercover agents are also permitted latitude for “justified disingenuousness”, but, even for police undercover agents, there are limits. Surely there are limits in climate science as well. I submitted the following incident to Nielsen-Gammon for assessment:
One of the complicating issues in this has been Steig’s public statements at realclimate and elsewhere. Another example not in Lucia’s list is a comment at RC from Steig on March 24, 2010. ( His review had been submitted three weeks earlier on March 5.) Steig’s allegation that we had been “unwilling” to do the work of re-analysing his data was completely untrue. At the time, as Reviewer A, he knew that we had submitted a comprehensive article. (Santer’s allegation was untrue as well. McKitrick and I had twice submitted an article showing that important Santer results did not hold up with updated data, and each time the article was rejected with scurrilous reviews.) Steig:
Eric Steig says:
24 Mar 2010 at 7:04 PM
comment-page-1/#comment-167866Here are my edits of Ben Santer’s comment, so that it applies to me:
In fact, my position on this matter was that Mr. McIntyre’s data requests were superfluous and frivolous, since Mr. McIntyre already had access to all of the raw climatemodeldata my colleagues and I had used. Mr. McIntyre also had access to all the algorithms required to calculate intermediate “value-added” information from the raw climatemodeldata. With some work – which he was unwilling to do – Mr. McIntyre could have replicated all of the
calculations performed in the2008 SanterSteig et al.InternationalNature paper.
Journal of ClimatologyAnyone see a pattern here?
While I find it hard the doctrine of justified disingenuousness in climate science as pretty hard to choke down, it seems to me that here Steig went far beyond any conceivable doctrine (however repugnant). Steig’s allegations were untrue on multiple counts.
That we now know that Steig was Reviewer A (who submitted his first review on March 5, 2010) makes this assertion even worse than we previously thought. Extensive analyses of Steig et al 2009 had obviously been carried out at CA and tAV in 2009. But, at the time that Steig made this comment, he knew that I was one of the coauthors of O2010, which had carried out a comprehensive analysis of Antarctic data from the bottom up. His public statement about being “unwilling” to do the work was completely untrue and malicious.
The “unwillingness” to do work in connection with Steig et al 2009 is not the only untruth here. Santer’s allegation is also untrue – something that Santer knew as well. Ross and I had twice submitted a comment showing that key results of Santer et al 2008 fell apart with updated data. Our comment had been twice rejected. (McKitrick et al 2010, published later in 2010, revisited this topic using different methods and was accepted in a statistics journal.) The first rejection was promptly reported to Phil Jones by Peter Thorne in a climategate letter – search “fraudit” – before anyone unconnected with the review process was aware of the rejection.)
Steig’s statement about data availability was also untrue. At the time of my initial inquiry, no Steig data had been placed online. Some of the data was made available very grudgingly. As of August 2009, Ryan’s requests for underlying AVHRR data had been stonewalled. The AVHRR data was made available only after a Materials Complaint to Nature (which, despite criticism, is taking a harder line on data obstruction by climate scientists than Science and some other journals.)
Nielsen-Gammon replied by email as follows to my a request for a specific comment on the Steig incident:
Steig is on record as strongly favoring the peer-review process for legitimate criticisms of his work, and frowning upon blog postings with incomplete analysis as a way of advancing the science. It seems to me that Steig is criticizing the blog postings. Note that Steig retains past tense: “was unwilling to do” instead of “is unwilling to do”.
I’d also point to the fact that after the paper does come out, he compliments you for doing the work.
If I remember the context correctly, one of the initial concerns that featured prominently in your blog posts was problematic data from station ‘Harry’, etc., which when the calculations were finally performed turned out to be of little or no consequence to the reconstruction results. You’d have to ask Steig yourself if that’s the sort of thing he was referring to.
I will, however, acknowledge, using the same words as Ryan O’Donnell used with regard to his conduct immediately following Steig’s RealClimate post, that Steig’s comment was “not optimal”.
“Not optimal” are faint words relative to his criticism of Ryan O’Donnell breaking the code of omerta on peer reviewer identity. Nor do I believe that a rational code of conduct would acquiesce in scientists making this sort of untrue statement that Steig made here, while preventing the targets from defending themselves through exposure of Steig as Reviewer A.
Climate Audit 
108 Comments
“… team-style proxy reconstructions had ended that line of argument and that 1000-year proxy reconstructions would be unable to advance without the development of more reliable proxies.”
Amazing.
“… team-style proxy reconstructions had ended that line of argument and that 1000-year proxy reconstructions would be unable to advance without the development of more reliable proxies.”
Amazing.”
Not really … this has been evident for many months. Gavin Schmidt’s comment on RC to the effect that he didn’t care what the temperature was 1000 years ago is sufficient to realise that defence of the Hockey Stick has been abandoned – palaeo reconstructions are now regarded as irrelevant
IMO, the core of the current punch-up here is that anonymity protects reviewers from reprisals but it also protects those who wish to initiate reprisals from within the safety of anonymity. The psychology here is not even interesting … Al Capone understood it, Machiavelli expounded on it. It is commonplace
But I thank John N-G for his pointed contributions (genuinely)
So, to summarise the current guidelines, when you take on the solemn duty of being a reviewer of a paper like O’Donnell et al it’s essential to your anonymity and the efficacy of the process that you attack the authors in public using various untruths, some of which you only know are untrue because of your role as reviewer, some of which you would have known were untrue in any case. With this method the probability of anyone guessing you are in fact Reviewer A can be made as low as you are prepared to go.
The secrecy involved here and in certain government functions would appear to be a convenient cover for those who might want to then “leak” information to their advantage. Any astute observer of the dealings of the US government knows this goes on all the time. Furthermore, those who have access to leaks, of which there are many, can preferentially leak the information they have depending on their political agendas.
Finally how silly those who do things in secret are made out to be when these leaks are eventually revealed and as much for the fact that they attempted to keep the information secret than for what it actually revealed. A few leaks here and there can then have the effect of all the participants involved in the secrecy losing credibility.
Having said that, I continue to doubt that outsider opinions are going to change in any meaningful way the transparency of peer-review any time soon and primarily because the groups involved are not affected or influenced by outside opinion. And again the only influence that might be exerted is that on the advocate/scientist interested in public policy.
I’m sorry, but I believe you made a typo: Accordingly, Nielsen-Gammon has harsh words for those who break the sanctity of the witness protection program. Should that not read: Accordingly, Nielsen-Gammon has harsh words for those who break the sanctity of the witless protection program.???
Sometimes I have trouble following his points, maybe that’s the problem.
Steve, MMH2010 was published in Atmospheric Science Letters, not a stats journal.
Though I haven’t followed the full exchange between you and John N-G, the MMH case is an interesting contrast, since Ben Santer was a reviewer of MMH, and his review was lengthy and critical, but he signed it. Under the circumstances I think if an editor has solicited a review from the party whose work is being criticised, it makes sense that they give it their best shot but also that they eschew confidentiality, so that everyone knows the parties involved are not disinterested. It gets a bit smelly if someone submits an anonymous review, then later makes public pronouncements on the paper as if he had not already seen it nor failed previously to convince an editor to reject the paper. That seems to me to be a part of the objections to Steig’s handling of O10 and aftermath.
I tend to prefer to sign my reviews when I make comments that could be perceived as self-serving, such as when the only references that I suggest the authors add are those that I co-authored. If I’m reviewing a paper that’s openly critical of my work, any comments might be perceived as self-serving, so that’s a very good reason for revealing my identity.
I doubt this is a common practice, because I almost never receive signed reviews of my own papers. Some may not be in a position to feel as immune to retribution as I do (or be as naive as I). Some may rarely make comments that could be perceived as self-serving. Some may prefer the authors to respond to their reviews objectively rather than in the context of their identities.
This means that you are a good man, JNG. And a wise one, looking at the egg on someone else’s face this month.
When you say, “I think if an editor has solicited a review from the party whose work is being criticised, … they [should] eschew confidentiality, so that everyone knows the parties involved are not disinterested” you imply that in all other case reviewers are disinterested.
But that is obviously not the case. Every scientist has an interest in obtaining support for their own findings, methods and hypotheses. A qualified reviewer will almost invariably, therefore, have an interest, positive or negative, in the paper that they review.
Moreover, one cannot say that, if all reviewers are assumed to be interested parties, all must eschew anonymity, because without offering anonymity, editors would, in most cases, be unable to obtain reviews.
Alfred, you assert that, “…without offering anonymity, editors would, in most cases, be unable to obtain reviews.” This could be the case, but it would be helpful for any research to confirm it. In the meanwhile there are reasons to think otherwise. From his posting on this at WUWT (Peer Review, Pal Review, and Broccoli, Feb.17) Willis Eschenback argues that, “…if reviewing a paper offers a chance for a scientist to get his name and his ideas enshrined on the record in a scientific journal … why do people assume that scientists would not jump at the chance? I know I would … and it is true whether I might agree or disagree with the paper.” Simple, and to me, very sensible.
As a footnote, Willis has some thoughts about reforming the system—well worth reading.
Ron,
When Eschenbach speaks of enshrining a reviewers ideas “on the record in a scientific journal,” he is talking of something quite different from the mere attribution of confidential reviews seen only by editors and authors.
I think it is a great idea, but as I related on this thread (at Feb 18, 2011 at 3:32 PM) my own attempt at naturalSCIENCE.com to establish such an open review process failed for lack of willing participants.
Well there was one exception: Sherwood Rowland, Nobel laureate submitted a letter which was an indirect critique of a paper by Arthur Robinson and others (which I did not publish but to which I provided a link) “Environmental Effects of Increased Atmospheric Carbon Dioxide,” which argued, as I recall, that rising atmospheric carbon dioxide concentration was an unmitigated benefit to humanity.
And getting that comment was like pulling teeth, involving the mediation of Karl Alberts, then President of the US National Academy of Sciences.
But now may be time for someone to to try again to establish an open review process. However, there is some potential downside. Nervous reviewers may pull their punches, especially when reviewing the work of important people, and others may engage in grand-standing to the detriment of balanced comment.
Readers will, I hope, note that by quoting the phrase “not optimal” I was inferring an analogy between Steve’s and my views of the seriousness of O’Donnell’s conduct (that is, the alleged ethical violation itself, aside from whether it was justified in this case) and Steve’s and my views on the seriousness of Steig’s conduct. I do not expect Steve to condemn O’Donnell harshly for his particular balancing of ethical considerations, nor have I done so except to emphasize the seriousness of the ethical violation. So don’t expect me to condemn Steig harshly for his own particular balancing.
Why does Steve’s treatment of O’Donnell enter into your calculus for the appropriate way to treat Steig? Tit-for-tat does not seem consistent with an unbiased search for truth…
Dr. N-G can correct me if I’m wrong, but I don’t think that’s what he was saying. He has issued no verdict on whether he believes what I did was justified given the circumstances and has not made that request of Steve, so he has no intention of issuing an analogous verdict on Steig’s behavior. This is not the same thing as saying “well, ’cause Steve didn’t condemn O’Donnell, I won’t condemn Steig”, which is what you seem to be interpreting.
Thanks, Ryan, you got it right.
John,
I think I understand the subtlety, but by analogy, does that mean I should expect you to “emphasize the seriousness of the ethical violation” (of Steig)?
James
No, I don’t think I have any special insights into the ethical seriousness of being misleading. I was a party to the issue of whether publicizing or naming anonymous reviews was ethical, and it was clearly an issue that most people did not have direct experience with.
Dr. Gammons,
You may not have any special insights into the ethical seriousness of being misleading, but you have some insights. You are the one that stated it was necessary to pretend not to be a reviewer. Acutally, I believe that takes some special insight. What I see, are a clear lack of rules for the review process. The editor himself should have contacted Steig whem he was made aware of Steig’s public statements, and demanded Steig reveal himself to have been “Reviewer A”. The editor should have never allowed Steig to be “Reviewer A”. You guys need to clean up your house, and do a “Rule Review”. Just make sure Steig is not “Reviewer A”
Keith–
I’m sure JNG has insights, but I don’t think his having insights compels him to make public proclamations about anyone’s ethical violations in comments here, at his blog, private email or even over a beer, with the person with whom he is speaking to “promise, cross their heart, never ever tell!”.
I have observed that many people have been asking JNG to make certain types of proclamations. I’ve seen requests here, at Eli Rabett’s blog, I’m pretty sure read something that looked like Eric making direct requests that he make a proclamation.
I think at this point, now matter how much each of us would like to know how he currently judges the balance of ethics, I think people maybe ought to stop pressing to make overly specific proclamations.
(I do think the hypotheticals involving chickens on a prior thread were fun.)
Lucia,
I don’t see where I am asking for any proclamations.
Re: John N-G (Feb 18 11:27),
Would this describe “non-optimal” as well?
Reviewers are constantly reminded to be constructive and helpful in their reviews, and editors generally try their best to provide encouragement where encouragement is warranted. But nonetheless, it sometimes feels like you’re [the reviewee I presume] walking up to a house in a strange neighborhood, ringing the doorbell, and when the door opens you’re standing there on the porch, blindfolded, with a sign taped to your chest saying, “If you don’t like the looks of my face, hit me.”
At least you have a good sense of humour… Darn near choked laughing — because that is so spot on. Having sampled the “research lifestyle” and academia I prefer to fund my own research and seek peer review outside academia. The reviews are more to the point — though no less stringent I assure you.
Further to the point I was trying to make here…
Anonymity isn’t the problem. Seriously bad behavior, the inability to behave like a gentleman is the issue.
Is it reasonable to assume that the Editor has taken note of this behavior? Can we expect that Steig’s participation with AMS publications will be sanctioned, curtailed or halted?
Or is this par for the climate course?
It seems to me you are making more of this issue than is justified.
When you suggest that a reviewer “wishing to engage in public discourse on the reviewed paper, should disclose that he was a reviewer” your rationale is that by making such disclosure he has “no need for dissembling.”
But to speak publicly about a paper, an anonymous reviewer who wishes to remain anonymous has no need for dissembling. He need only take care to avoid revealing what he could have known only by virtue of having reviewed the unpublished manuscript.
It is true that he may accidentally make such a revelation and in that case then, yes, his cover is blown. But that is the result of ineptitude, not a necessary consequence of commenting publicly.
Should an anonymous reviewer in that situation attempt to negate the effect of such self exposure by dissembling, then sure, he deserves whatever public disapprobation he gets.
But your suggestion that a reviewer who wishes to remain anonymous should refrain from public comment is neither realistic nor logically justified.
I think he’s saying that reviewers can make public comments with regard to papers they anonymously review, but they can’t (or rather ‘shouldn’t) make comments that are untrue, and either disparaging or damaging the credibility of another scientist or author– especially if those comments can only be reasonably defended by way of revealing confidential information.
Creating a situation that permits reviewers to conduct themselves in that manner knowing that authors can’t respond adequately without revealing confidential material has now been shown that may also ‘not be realistic or logically justified’.
Demands to “Prove it…Show is the evidence!” can quickly change to “Hey, you weren’t supposed to see that!” or “You aren’t supposed to have that!” Some contexts of the Climategate fallout demonstrate this. Some of what Steve is saying is also bearing witness.
Justified disingenuousness seems to have some sort of limit, whereupon continued provocation will release the hand of “confidentiality” tied behind the back…creating a kind of ‘justified disobedience’.
“I think he’s saying that reviewers can make public comments with regard to papers they anonymously review, but they can’t (or rather ‘shouldn’t) make comments that are untrue, …”
But that’s not what he said.
What he said was that a reviewer “wishing to engage in public discourse on the reviewed paper, should disclose that he was a reviewer,” i.e., that an anonymous reviewer has no right of public comment, which is, as I argued, neither logical nor realistic. To deny the right of public comment is to exclude someone from science, because except by open discourse, there is no advancement of science.
Peer review is a strange sort of “open discourse”. In any event, the point others have made is that anonymity serves to protect those who do not wish to get embroiled, or who fear reprisal. This would seem not to be the case for someone who goes on the record subsequent to publication. What is the reason for maintaining anonymity in this scenario, other than to present an argument as fresh or objective, that in reality was presented to an editor and either rejected or satisfactorily addressed? Or to introduce a Trojan horse as Steig seems to have done?
I fail to see what climate scientists gain by this war against transparency.
“Peer review is a strange sort of “open discourse”.”
I did not call peer review “open discourse.” What I referred to as “open discourse” was the public discussion from which SM suggested anonymous peer reviewers should be barred.
“the point others have made is that anonymity serves to protect those who do not wish to get embroiled, or who fear reprisal. This would seem not to be the case for someone who goes on the record subsequent to publication.”
The situations are not comparable. As I pointed out on another thread, there are potential rewards for public comment, i.e., in the enhancement of public reputation, which are not available to a peer reviewer whose comments are made available only to editors and authors — peer reviewers are explicitly denied the right to publish their “confidential” appraisal of a manuscript under consideration for publication.
What is this “war against transparency” that you see being waged by climate scientists?
No one has suggested that a reviewer can’t make public statements or critiques about the paper they reviewed.
“No one has suggested that a reviewer can’t make public statements or critiques about the paper they reviewed.”
But SM suggested that a reviewer “wishing to engage in public discourse on the reviewed paper, should disclose that he was a reviewer,” i.e., that an anonymous reviewer has no right of public comment. So yes someone did suggest precisely what you deny was suggested.
Re: Alfred Burdett (Feb 18 15:00),
when you are offering securities to the public, disclosure obligations are more stringent. I suggested a similar answer to the very unsatisfying spectacle of disingenuousness – which is not what the climate debate needs.
My suggestion (and it’s only a suggestion) is that the dilemma could be avoided by disclosure.
I haven’t seen any arguments yet that persuade me that such a policy wouldn’t be an improvement.
I have no objection to disclosure, but having managed the peer-review process for thousands of manuscripts, I don’t think, as I indicated elsewhere, that editors would find many willing reviewers if disclosure of reviewers’ identities (to authors) were required.
There would likely be more incentive for a reviewer to reveal their identity if the review process is made public. At least, in that case, a reviewer has the possibility of reaping recognition for their effort and any ideas that they put forward in their review — benefits that counteract in some measure any reprisal by authors for what are deemed unfair criticism.
However, as I may have mentioned on another thread, my experience with an attempt at naturalSCIENCE.com to instigate an open review process, indicated great reluctance among scientists to engage either as authors or reviewers. For the most part, the only contributions we got were from a Nobel Prize winner, two Crafoord Prize winners, one winner of the US National Medal of science, a number of distinguished emeritus professors, one or two practicing scientists of notable stature and several very bright graduate students. Career scientists, however, were as a whole simply not interested.
A possible downside to an open review process, even if it could be made to work, is that reviewers will pull their punches, especially when commenting on the work of important people, or of people who may review their own work or research funding applications.
Re: Alfred Burdett (Feb 18 15:32),
You seem to be missing the point. People aren’t asking that a reviewer must be required to reveal his/her identity. But if that person wants to join in a public debate of a published paper, that person should either reveal his/her identity as a reviewer or at least not complain if identified as a reviewer. Any damage to the reviewer would moot since those who might have been upset (and able/willing to do anything about it) would know anyway.
Not so. SM explicitly suggested that a reviewer “wishing to engage in public discourse on the reviewed paper, should disclose that he was a reviewer,” i.e., that an anonymous reviewer has no right of public comment.
But I agree that a reviewer has no reason to complain of being identified as a reviewer since, as I stated on the previous thread:
[authors are] free to reveal both the contents and, if known, the authorship of a peer review, unless the journal to which they have submitted makes it a condition for the consideration of an article not to do so. No journal that I know of makes this stipulation.
So science breaks down to “important people”, or “research funding applications”. I get it now.
“So science breaks down to “important people”, or “research funding applications”.”
You better believe it. Unless your an independent like SM, which is why independents have much to contribute.
Except when the gatekeepers don’t want them to.
“Except when the gatekeepers don’t want them to.”
Twaddle. Publication of O’Donnell et al., a paper by outsiders which refuted the main conclusion of a Nature cover story, refutes your paranoid presumption that the literature is somehow hobbled by climate scientists.
The extraordinary public recriminations over the handling of the O’Donnell et al. paper that has been published, is surely more indicative of a lack of balance on the part of some participants in the debate than of a real problem with the management of the scientific literature.
What is noticeable is that although “peer review” has been subject here and at WUWT to much opinionated comment by people with seemingly little understanding of how the scientific literature is compiled, no radical ideas on how the literature might be compiled in a more satisfactory way have been forthcoming.
Instead, there have been ill-considered demands for this or that limitation on who can participate or for the imposition of these or those terms on which participation should be permitted. For example, the idea that an interested party should not participate in the review process, even when that individual may be better informed on the subject than anyone else, and as if there was anyone in a narrow field of research, i.e., any of the prospective reviewers, who is without an interest that may prejudice them either in favor or against a particular new claim.
The fact is, the literature is wide open. There are dozens of journals in virtually every field and hundreds of journals in some fields. It is inconceivable that any cabal could control all these publications some owned by learned societies, some by individuals, others by multinational corporations, with editorial offices in many different countries. If you have ideas that not one of the journals in your field will publish, the chances are your efforts are outre in the extreme.
Actually, the entire debate over confidentiality of the reviewer’s identity in this instance is a straw-man debate. The relevant standards of the journal involved are explicit. Steig should not have been involved in the “peer” review process of the O’Donnel et al. article, period. The adequacy of his work is what the paper under review was addressing. Under those circumstances, the appropriate pattern is a “comment – response” cycle, not a shot at “peer review” and certainly not an “anonymous” roll. He would have extreme difficulty being objective in any case, and as an anonymous reviewer there would no professional/social constraints that would govern his behaviour either. Peer review should not be anonymous.
If the journal editor is doing his or her job properly, it is for them to assess whether, or to what extent, a reviewer’s comments are objective and then make a decision on the acceptability of the manuscript. To deny Leibniz the right to review Newton’s critique of his work would be idiotic, even if Leibniz insisted on anonymity.
There is a difference between “review” and “comment.” The peer review should determine whether a paper is “publishable” or not. Leibnitz as an anonymous reviewer would be in a position to suppress Newton’s publication, to the loss of us all, if the publication is of substantive importance. That is why critqued authors are allowed comment at the time of publication and why they are NOT permitted to review papers critical of their work per the AMS guidelines. I notice that Willis Eschenbach has taken cognizance of this in detail over at WUWT.
The actual peer review failure is really the editor’s failure to implement the guidelines he was supposed to enforce. Steig’s behaviour was ethically dubious, but clearly human in that he obviously was attempting to defend work which he had invested a great deal of effort in. Likewise, Ryan, while acting perhaps a bit hot headed, was responding predictably to the antics of Steig who seems to have been talking about of both sides of his mouth. The editor’s actions however are a different matter. He ignored the explicit standards of the AMS in inviting Steig to review the draft paper, and in permitting the process to drag on as it did.
“The peer review should determine whether a paper is “publishable” or not.”
I don’t believe any editor worth his or her salt would acknowledge this to be the case. It is the editor’s job is to write a decision based on their own assessment of the work, an assessment made in the light of, not dictated by, reviewers comments.
If an editor’s decision is based simply on a poll of reviewers, then the outcome becomes a matter of almost pure chance, since a sample of two of three reviewers cannot possibly provide a reliable indication of the general consensus of the specialists in the field.
However, if as you assert, Steig was unqualified as a reviewer by virtue of some AMS policy, then if anyone was at fault it was surely the eEditor who asked him to act as a reviewer. Or are we suppose to assume that editors operate in such a perfunctory manner that they either do not bother to see what a paper is about before selecting reviewers or they fail to consider the suitability of potential reviewers according to the content of a paper?
It seems to me that scientific communication should be as free as possible and that to hedge the peer review process about with either elaborate journal-mandated protocols or legal limitations is highly undesirable. The privilege of anonymity that journals grant reviewers seems to be a practical necessity in obtaining candid reviews. Beyond that why not leave responsibility for maintaining the integrity of the review process to the editors. Some journals will succeed in this respect better than others, and that will eventually show in the relative standing and prestige of the various journals.
John N-G,
I must say John that you have a particularly obtuse writing style. I find myself reading your posts several times before I understand the riddles of your style. Am I the only one who has to decipher your writing.
I don’t find it obtuse…I think it’s very ‘chosen’. If someone wants to craft the exact way you want to go in a sea of landmines…the path will often look irregular.
Alan Greenspan might also be characterized as an ‘obtuse’ communicator when he testified at the fed. An impulse to tread carefully can easily create difficult wordings.
He may feel free-er to speak more loosely in other settings. I can understand why this might not be one of them.
“….If someone wants to craft the exact way you want to go in a sea of landmines…the path will often look irregular.”
I second that… I would call the writing style ‘precise’, rather than ‘obtuse’.
I find JNG’s writing style clear.
I would like to say that I find Dr. N-G’s writing style quite clear. I appreciate the way he gets to a point and explains it concisely.
Fear of retribution is an interesting idea. In business, such things are out in the open, and yes, there can definitely be fear of retribution. On the other hand, the world of climate science seems more like a non-profit to me. Limited real power, so lots of opportunities for petty games. I still think it’s the editor with the power, and the reviewer misconduct should have been dealt with at the time. The authors should not have been coerced into writing a different paper. As far as venue, the reviewer chose the venue, so I don’t see that the journal has a dog in the hunt at this point – their issue should be solely with the reviewer.
As an aside, my rule is that if a paper switches analysis techniques, it invalidates the paper (this is due to people analysing a bunch of different ways to show some result). I consider the analysis plan an integral part of the effort. In this case, I’ll make an exception (though grudgingly), since the result was truly a new paper.
So Dr. N-G defends lying, deceit, and duplicity as acceptable or possibly “not optimal”. Well it is “climate science”, so why am I not surprised. These people make mafia thugs look like gentlemen.
Per his comments above, I believe he’s being very careful with his words, and trying to avoid passing judgement.
It seems pretty clear to me that there is a problem here, but that Dr. Neilson-Gammon is trapped within it, rather than being an instigator.
My experience is as the editor for one of the large electrical engineering journals (IEEE Transactions on Microwave Theory and Techniques). Out of more than 1,000 reviews that I received, I never saw anything like the set of reviews from Reviewer A. The length of the reviews and the clear conflict of interest made the process utterly unfair to the authors and indirectly to the readers. In my opinion, the result was predictable from the history, and this means that the editor make a mistake to ask for a review from Reviewer A.
In addition, as a researcher, I have had the personal experience of having a paper submitted that attacked my work in electronics. I declined the editor’s invitation to review the paper, and this is what Reviewer A should have done.
David, thanks for commenting. David Rutledge’s webpage is here / His most recent online talk is about Peak Coal – an topic that interests me from the point of view of mining and minerals as well as climate. I’ve been meaning to host a thread on this and I urge readers to visit.
David used to have an online talk in which Climategate was discussed at one point. David observed (my approximate transcription):
“I have had the personal experience of having a paper submitted that attacked my work in electronics. I declined the editor’s invitation to review the paper, and this is what Reviewer A should have done.”
Can you offer any particular reasons? After all, as the author of the criticized work, you were, surely well qualified to point out any errors in the critique.
Hi Alfred,
If the critical paper was accepted, I would have had a chance to respond after it appeared. If it was rejected, I would have been able to tell the author honestly that I was not responsible for blocking his paper.
David,
I respect your position. However, I don’t see it as obligatory for other scientists to take the same position. If someone criticizes my work (which may be in a very obscure field about which few people have specialized knowledge), and if I believe the criticism is misconceived, why should I not, acting as an anonymous reviewer, point out what I believe to be the author’s error? After all, none of the other reviewers may have my special knowledge or expertise.
True, I am an interested party, but I am also a scientist attempting to argue a case on the basis of facts and logic. It should be up to the Editor to decide whether I have managed to do that effectively. Furthermore, any other reviewer in the same field of research is likely to be an interested party to a degree, whether he supports or criticizes the work under review.
And it must be understood that it should not be my decision as a reviewer that determines acceptance or rejection of the paper. That decision is the responsibility of the editor who should weigh the evidence, taking into account my possible bias. Thus the question of my blocking publication does not arise. If publication is blocked it was blocked because of a decision of the Editor’s.
On one occasion I experienced refutation of my work, not by a reviewer but by the author of another published paper. As a result, my paper, among my first published efforts, was ignored for more than a decade. Then, after I had changed fields entirely, my work was confirmed by another publication, following which my paper became, if not a citation classic, at least a standard reference in that particular area of research.
Had I been given the opportunity, as a reviewer, of pointing out the logical errors in the paper that contradicted mine, progress in a minor field of research might have been ever so slightly accelerated.
Hi Alfred,
Your points are valid. I meant it not as matter of policy or ethics, but rather judgment on the part of the editor and the referee. In many situations it would be appropriate for an author of a paper being criticized to be a reviewer, as in your case.
Dave
Hi Steve,
Steve’s transcription is accurate, but for reference I put the video (223MB) and slides (5MB) up at
http:/rutledge.caltech.edu/TalkDaveRutledgeJan2010.flv
and
http:/rutledge.caltech.edu/TalkDaveRutledgeJan2010.pptx
However, the part of the talk that deals with fossil fuels and climate has been updated since then, and for this material it would be better to follow the link at my web site.
In my engineering experience, the kind of sensitivity analysis that Ryan O’Donnell does on Eric Steig’s approach is considered essential for design reviews and Ryan’s results would be a show stopper. My customers are buying a real product, and that reality is not just clever statistics. If I get it wrong, my customers get hurt, my company goes bankrupt and I lose my job.
“In my engineering experience, the kind of sensitivity analysis that Ryan O’Donnell does on Eric Steig’s approach is considered essential for design reviews and Ryan’s results would be a show stopper. My customers are buying a real product, and that reality is not just clever statistics. If I get it wrong, my customers get hurt, my company goes bankrupt and I lose my job.”
Well in my engineering experience, which I am pretty sure exceeds yours David, the kind of sensitivity analysis that Steig does on O’Donnell’s approach is considered essential for design reviews. See how that works? Both ways it’s called.
We tend not to decide the outcome of our reviews before we hold them you see, or to dismiss any constructive criticism during them as obstructive, or (gasp) ban anyone who offers any criticism from future reviews. We actively encourage it. Can you believe it? It’s crazy isn’t it? Call us old fashioned, but we’re a global behemoth who have become quite attached to our hegemony, and our customers like us to be right, not myopic.
If there are any show stoppers, we have this curious predilection for finding out that the show actually needs stopping. We leave the launching of space shuttles with frozen ‘O’ rings to those who choose to dismiss the shouts of ‘Stop the show!’ coming from their most qualified people.
Boro Nut
Please point me to said sensitivity analysis and perhaps I will consider what you write.
Shoddy science might result in awkward questions from your peers.
Shoddy engineering might result in awkward questions in front of a twelve peers.
Protecting anonymity may be institutional in science, and it might be Obstruction in engineering.
“Justified Disingenuousness” is the same oxymoron as Schneider’s “Effective science”.
Steve said up top:
I’m having some difficulty finding any of my comments to that post in which I said anything at all about Steve McIntyre’s assumptions about the academic world. I would be grateful if someone would provide a more specific link to a particular comment, so that I can see if I’ve done something for which I should apologize.
John N-G
“I’m having some difficulty finding any of my comments to that post in which I said anything at all about Steve McIntyre’s assumptions about the academic world.”
I think this is what you are looking for….
“But I knew that what was obvious to a scientist familiar with the review and publication process might not be so obvious to a relative outsider.”
“Steve asks whether any breach of AMS journal policy actually occurred.
Certainly there was no breach of any published policy. The issue of publicizing reviews is not touched upon in the AMS’s guidelines for authors, editors, and reviewers. Likewise, it would not violate AMS journal policy to visit the home of the author of an unfavorable review and let the air out of the tires of his or her car. That issue, too, is not touched upon in the guidelines. Precisely because the issue is not discussed in the AMS’s guidelines, I thought it would be useful to outline the bounds of ethical behavior for the benefit of those not so familiar with the process.”
Thanks. If that’s it, then I’ll clarify. Back in December, I didn’t really know what assumptions Steve would make, whether they’d be too idealistic or too cynical. I think one can make a strong case for either possibility based on what he has written. In either case, I thought facts would be better than assumptions.
http://en.wikipedia.org/wiki/John_Nielsen-Gammon
“John Nielsen-Gammon (born 1962) is an American meteorologist and climatologist. He is a Professor of Meteorology at Texas A&M University, and the Texas State Climatologist, holding both appointments since 2000. His research group uses a combination of observational and computational techniques to study the characteristics, dynamics, and forecasting of certain weather phenomena. Much of his recent work has involved air pollution meteorology. He writes a popular online column on climate science for the Houston Chronicle.”
I may disagree, and in fact disagree with lots of people, about the proper level of transparency in various processes, but if Dr. Nielsen_Gammon tells us that that there is a real danger in his area of work with retribution from others in the field through knowledge of how they review and who they are, I assume he knows what he is talking about. With his experience with climate science, SteveM’s lead into this thread, in my view, is in agreement with John N-G .
Since no one here is going to change anybody else’s mind about the transparency required, perhaps we could discuss whether what has been noted about climate science applies equally as well to other sciences or are there special considerations applying to climate science – that we might also discuss.
But is he also not saying, in effect, that the peer review process often actually has more to do with maintaining the reputation and position of so-called established scientists than with progressing the science?
Note that it is not merely actual danger, but perceived danger, that would affect the content of the review. As the present case shows, the danger need not be confined to the academic realm: one of the rare points of fact in this case is that the lack of preservation of reviewer anonymity was connected with public accusations that were later apologized for.
Regarding your broader disciplinary question, anonymous peer review is common throughout the geosciences and was not developed for any specialized climate science need. So far, reviews of my climate-related papers have not seemed different in tone or quality than reviews of my papers in other subdisciplines.
In the present case, it is clear that Steig has/had no fear of any kind of reprisal. If he did, he would not have made the public comments designed to discredit O10. Any chance Steig apologized? Not a chance, the guy will not even engage in a discussion of ethics, that he started. Silence is a very bad indicator in this situation.
I am going to repeat a comment I made in earlier thread as it seems more germane here and provides additional evidence about the reluctance of climate scientist to speak freely.
Reviewer D who seems quite knowledgeable and fair-minded, leads with “One particularly pertinent aspect of OʼDonnell et al.ʼs results is that they are consistent with two recurring, informal comments by the community that have emerged in the wake of the Steig et al. paper.”
So it appears that the climate community was well aware of methodological issues with S09. This is really not surprising as the community is quite accomplished and the S09 issues appear reasonably clear. Yet both prior and subsequent to the publication of O10, you would be hard pressed to find any climate scientists publicly expressing this “recurring” view (Trenberth being an exception). It is almost as if the community feels it needs to present a united face and no-one is allowed to break ranks. This seems unfortunate.
There is a very important point here I think people are missing.
Anonymity for reviewers is practiced to protect younger, less influential scientists from being afraid to point out errors by older, more established and influential researchers who might seek some form of retribution – like withholding a promotion. Anonymity is not designed to protect establishment viewpoints and researchers.
Steig is the more established researcher, much more than O’Donnell. And Steig represents the majority opinion on climate change. The fact Steig is using anonymity to hide behind his attempt to keep a paper from being published which exposes weaknesses in his own paper. Well, it is unseemly. This kind of disingenuousness is not justified. It is an embarrassment to science.
“The fact [is] Steig is using anonymity to hide behind his attempt to keep a paper from being published”
You cannot develop rules about the scientific publishing process based on assumptions about people’s motivations.
Steig may have deliberately sought to suppress a paper that he realized correctly identified errors in his own work, but you have no means of knowing this. Much more likely, Steig actually believed O’Donnell’s paper was flawed. Heck, it contradicted Steig’s own work, so it had to be wrong!
The practical issue’s are: (1) did Steig raise any valid criticism of O’Donnell that enabled O’Donnell et al. to improve their paper, and (2) did the Editor give undue credence to questionable or verifiably false claims in Steig’s criticism of O’Donnell?
To say, “This kind of disingenuousness is not justified. It is an embarrassment to science,” assumes that one has a privileged knowledge of the state of another person’s heart and of the quality of the Editor’s judgment. But neither assumption is justified.
No doubt many peer reviewers behave badly or show poor judgment under the influence of personal interest. But that’s just how flawed humanity is.
The thing about the science publishing process is that almost nothing that is vaguely plausible is denied publication, eventually, although it make take some persistence, e.g., submission to several journals (serially, not at the same time). In the process of successive submissions, a sensible author will, as did O’Donnell it seems, make revisions that will minimize friction with reviewers.
And that’s the best we can hope for. For as Bertrand Russell I believe put it, in the matter of knowledge of the external world, the serpent in the garden of epistemology is the fact that no one has certain knowledge of empirical reality. In science, we just have to blunder along, trying not to get too upset by the obtuseness and self-serving behavior of rivals.
I was not judging heart, but actions. And I think you are confusing commonplace with acceptable. Even if I was to accept that Steig’s behavior was commonplace (and I don’t), I do not agree that it was acceptable. If Steig’s concern was the science, he had many options available to him – anonymous peer review, signing his review or keeping his powder dry and publishing a comment on O’Donnell. He chose to anonymously suggest the editor “insist” on a major revision and then chose to publicly criticize O’Donnell for that choice.
Anonymity in peer review is a necessary evil to protect the less powerful researchers and allow them to present their evidence without fear of reprisals. Steig is the more powerful researcher, representing the majority climate position and he hid behind anonymous peer review to protect his own reputation. I don’t see how that is defensible. And I noticed that you did not try to defend it.
“Steig … hid behind anonymous peer review to protect his own reputation.”
You are imputing motives, i.e., judging what was in the reviewer’s heart.
“I don’t see how that [use of anonymity] is defensible.”
It is not necessary to defend Steig’s use of anonymity as a reviewer: as a reviewer, anonymity was his privilege.
If Steig later blew his cover and contradicted what he said in the review, then yes, he needs to justify the contradiction, e.g., perhaps he changed his mind, or face criticism that either his review or his later comments were not sincere.
But either way, I see nothing here that brings the practice of anonymous peer review into question. It is not (or certainly should not be) the reviewer who decides whether a paper is accepted. That is the responsibility of the Editor, and in this case, evidently, the editor accepted that responsibility and decided in favor of the authors.
What you need to understand is that there is no certainty about any scientific conclusion. All that a journal editor can do is attempt to make a reasonable assessment of the claims before them based on the evidence presented, including evidence from their own experience, comments of reviewers, rebuttal of reviewers comments by authors, and quite often hours of hard re-analysis of the data or literature search.
Most of this is work that editors (and reviewers) do in their own time with little if any payment. The extent of such essentially volunteer work to maintain a high standard of scientific communication is truly astonishing. To turn on those who do this work, when they appear to have done the job with integrity and with the investment of considerable time, seems totally unfair.
Think about it. O’Donnell et al. come up with a conclusion that contradicts what has already been established in a paper published by Nature. The reviewer says it’s crap, or something like. As Editor, what you gonna do? Reject it probably. Then the authors send you 47 pages or 88 pages of rebuttal. Oh, God! That’s at least half the week-end shot, by the time you’ve read the rebuttal, reread the review and tried to think it all through, and perhaps gone to the library and read some relevant stuff. Then you have to formulate a new decision and present it to the authors with your reasons, your directions for any necessary revisions, etc.
That’s what happens if it’s a well managed journal. And it seems that O’Donnell et al. were in the hands of a well managed journal and were well treated (which is not to say that there would have been anything unnatural about it if they had felt, initially at least, indignant that the world had not immediately and unquestioningly accepted the importance, beauty and truth of their work).
Since the paper in question was a purely statistical exercise, and Steig reminds us that he’s “not a statistician”, why was he a qualified reviewer in the first place?
Jeff,
You misunderstand the role of an editor. A journal editor is not in the position of God, who knows all, he is in the position of a judge who must try to discern the truth from the evidence presented. So when A accuses B of misrepresenting the weather in West Antarctica, what will the judge do? He will as B what he has to say in his own defense. B, in this case, was Steig.
SteveMc
In a partial rejoinder to Kenneth Fritsch’ response above, what has always stuck me as both curious and instructive is that when the full identity of their opponents is known -as in your case without a PhD – the Team and their various alcolytes and supporters in the MSM -when polite- refer to you as “Mr” McIntyre. This is a slur, the only intent of which is to diminish your purported abilities as a researcher in the eyes of the “great unwashed”.
Those involved, instinctively would probably not be so readily inclinde to threat someone like Ross Mc the same way. Why? The PhD. It is unfortunate to say, but that is why – in your case in particular- the other side in refering to you has chosen to use “Mr” – as a put-down.
Hey, no PhD [forget about the MSc or PEng or other qualified observers], no PhD. Doesn’t count… Then of course, comes the belittleling of the PhDs – like Ross and a few other who have had the temerity to publish- or those PhDs who have not published, relegating our insights to the composter. Only PhDs in “Climate Science” need apply to be heard….
As a mental and philosophical thought experiment it would interesting to think of an academic debate about “Climate Science” as it stands today involving representatives of the IPCC, and sub groups subscribing to its hypothesis: the “Team” and their various institutional supporters on the one hand, and the likes of Galileo, Koch, Mendel, Planck and Einstein [and whomever you care to add to the list..] on the other. No PR -just staight Handsard reporting of the argumentations…
[for what it is worth, tetris has a PhD in a subject matter at the heart of the matter at hand]
PS: Keep up the good work Steve
Take away the careful manipulation of the MSM in support of one line of argument, and not much would be left in fairly short order.
The belittling use of the title Mr. works both ways. Our very own National Post refers to their libel-suit-antagonist climate science professor Andrew Weaver as Mr. Weaver. Likewise, they refer to James Hansen as Mr. Hansen.
However, there can be an amusing reverse snobbery among those titled Mr. In England, for example, the College of Physicians would not admit the surgeons or allow them to style themselves Dr. Later, when the prestige, not to mention income, of surgeons greatly exceeded that of a GP’s, the College relented. However, the surgeons spurned the invitation to membership and, to this day, proudly distinguish themselves from mere doctors by styling themselves Mr.
And in British academia the acquisition of a PhD (standing for PhilleD-up, according to Stephen Leacock) was, and perhaps still is, deemed superfluous for a truly talented scholar. John Ashworth, for example, who headed Margaret Thatcher’s think tank, served as Vice Chancellor (President) of Salford University and later as Director of the London School of Economics before settling down as a professor of the History of Science, began his academic career as a professor of biochemistry at the age of 21 and as a plain Mr.
Much has been disclosed in recent days regarding the “cultural” norms of (what one sincerely hopes apply only to) peer-review in “climate science”.
Set aside for the moment the matter of whether or not Steig should have been chosen/permitted to be Reviewer A without full disclosure of potential conflict of interest.
[speculation alert!]
This newly disclosed “doctrine” [for want of a better word at the moment!]of “justifiable disingenuousness”, suggests to me that there may have been a rather interesting parallel at play during the various “inquiries” subsequent to Climategate; i.e. an extension of this heretofore unknown (well, certainly not commonly known) “doctrine”!
Just as members of “the team” have accorded themselves the right (or perhaps a “scientific licence” – somewhat akin to “poetic licence” in literature) to apply statistical methods in ways that are … hmmmm … unique to “climate science”, perhaps it was knowledge of this “doctrine” (and “cultural” norms) that was applied to and/or coloured the conclusions of the various inquiries (cf inquiries’ views on “trick”, “decline”, “violation of confidentiality” etc).
If nothing else, acceptance of both “the team” behaviours and the conclusions of the various inquiries – not unlike acceptance of literature in which poetic licence has been copiously invoked – requires nothing less than a “willing suspension of disbelief” on the part of the reader.
I am pretty sure my parents have engaged in sex, but I don’t phone home and ask them if they had it last night.
It is improper to ask anyone if they reviewed your paper/grant. Someone might state that they were a reviewer; as in-
“Just finished reviewing your XXX paper; it was very nice, I liked the YYY methodology. Any chance you could send me a pre-print so I can cite it?”
That is about it.
You can sometimes guess and then see if you can make their cheeks go red by damning/praising that sagacity of Referee #3, next time you meet them.
A author can ask the editor to send a thank you to a particular referee for a suggestion (I got one for asking a Japanese Group looking o ethanol effects in cell cultures to add a note linking mM ethanol to driving limits and the number of measures drunk by an average sized person).
If you are a referee you should not give the game away by suggesting the authors add 5 references, all by “DocMartyn”. You shouldn’t use a buzzword in your review that you use in speech or text (like doable).
Asking someone if they were a reviewer is the height of bad taste.
Among my first blogs, which have always been under my true name, was a short c.v. The concept of blogging under an alias was foreign to me, was not even considered, except for a couple of times elsewhere where I used a bit of R-rated humour that I expected might be snipped from the eyes of my family.
In my working career, I was often on national media in Australia and I was advertised as a link between the resources industry and government. My name was in clear print authorising paid advertisements criticising green policies. It was advertised that I was an industry link person between police and the public for a large gold-stealing racket which had already seen 2 people missing, presumed dead.
For all these connections there was a price. There were threatening phone calls, particularly from some people who could see from media that I was out of town, so my wife could be attacked. There was a device that looked like a bomb placed on our BBQ table outdoors. Mt car was rolled down the driveway slope, smashing into the home. Another car had tampering so the headlights lights suddenly stopped – on a dark winding mountain track at 100 kph.
Yet, I still choose to be open and public about identity. The dominant reson for this is credibility. Not mine, but others’. Those who are open about identity are read here before those who use nom-de-plumes. Steve has not concealed identity, to the contrary, we hear of his private assaults on the squash world. That gives him a large plus in credibility.
It’s not just publication review that needs discussion about identity. Blogging gives you a choice, be yourself or use a front. There will be a few cases where the latter is justified, a very few, but the choice remains.
However, for one who blogs openly, there is a constant suspicion that the alias is used because a place is reserved for some form of concealment, be it mild deceit or major misleading.
Don’t you agree that the best bloggers tend to be open? Can you not see that if all climatology authors got together and agreed to full disclosure in publications and for blogging, the problems above would disappear almost overnight and the world would be a more normal place?
Reviewer anonymity = personal insecurity.
I wonder: should my Doctrine of Justified Disingenuousness apply only to anonymous peer review, or should it apply to every situation in which you have promised to keep a secret or feel otherwise obligated to do so?
For example, suppose Misty has told you in confidence that she is pregnant. Some time later, you are chatting with your friends, and one says “Did you see that bulge in Misty’s abdomen? I wonder if she’s pregnant! OMG! What do you think?” Which response would you choose?
(a) “Sorry, I’ve suddenly decided to never comment on any possibilities of pregnancy.”
(b) “Yes, she is definitely pregnant. I promised not to tell anybody, but I don’t want to be disingenuous.”
(c) “Gee, I don’t know.”
JNG–
I wouldn’t promise to keep a pregnancy a secret for very long because it’s pointless.
Still, I’m willing to tolerate some disingenuousness. First, let me list things that take disingenuousness much to far– your friends will consider you a liar if you do thiese. You cannot
(d) for no reason at all, volunteer that you heard Misty say she was worried she had fibroid tumors that had enlarged her uterus to the size of a 22 month pregnancy.
(e) pop into a conversation out of the blue, where no one has asked you anything and “volunteer”, “Oh! I heard Misty tell me she’d had tubal ligation last year.”
For the person who was unwise enough to promise not to reveal a preganancy taht is now in it’s 6th month and who is is put in the position of a direct question, I might advise the best strategy is changing the conversation.
Here are possibilities:
(f) Her abs? Look at mine! I’ve been going to the gym non-stop for 9 months and I still can’t get them under control. What do you do? (Women will generally start discussing their diets and abs. All will be saved.)
(g) OMG! When I was 14 visiting relatives in El Salvador, I asked Teofilo’s wife if she was pregnant. She turned sooooo red. My mom shusshed me. How embarrassing!
That said, your (a) is acceptable if true. The problem is that despite the fact (g) is a true story, I have never been above to not comment on the possibility that other people are pregnant. My friends would know (a) was a lie.
I’m having trouble following the analogy. Is Eric pregnant?
Maybe Ryan’s pregnant, and Eric is his “doctor.” And Eric says, “Please don’t tell anyone I’m your doctor … now I’m going to blog about your pregnancy.”
Nice suggestions, Lucia
How about these also
h) You should know better than to ask me that question
i) Don’t go there, “is she pregnant or just fat ?” is not a good question
j) Err, nothing to do with me, honest
k) Oh no, my wife is gonna kill me..
What’s wrong with “maybe” or “don’t ask me,” or simply upraised shoulders, palms up and a suitably perplexed facial expression. Is that lying? I don’t think so. We have a right not to disclose what we do not wish to disclose or feel obliged not to disclose.
Or even, “why don’t you ask her?”
Jonh, these are typical of the guidelines one works under in the medial side of science.
Click to access 088.pdf
“In informal discussions about specific patients, no mention should be made of either the patient’s name or of any references such as room number, personal or social information which might serve to identify him/her to any who overhear.
To describe specific patients, staff members, or units of the institution may breach
confidentiality even if the name is not used – i.e., if the information you give would allow the hearer to make an identification.
In general, discretion should be used in discussing patients in areas of the institution accessible to the public, even if the anonymity of patients has been assured.
Further, similar discretion should be used whenever one may be overheard by employees and
healthcare workers who do not have a “need to know” with respect to the patient about whom you wish to speak.”
I am allowed to state that I work at a very good Hospital, I cannot identify a patient who has attended the hospital.
The correct answer to your questions is
(d)is that the time? I have to take something out the incubator.
“Regarding your broader disciplinary question, anonymous peer review is common throughout the geosciences and was not developed for any specialized climate science need. So far, reviews of my climate-related papers have not seemed different in tone or quality than reviews of my papers in other subdisciplines.”
I suppose your reply here leads naturally to the next question that I have about anonymous peer-reviews and that is what is the rationale given by those disciplines that use strict anonymity and those that use less or none. What can we say about the comparison of the product coming out of these disciplines? I would also suppose that anonymity is not strictly related to discipline but more to publishing organization and we therefore have a further division to look into.
I am guessing here, but publishing has evidently such a large impact on academicians’ careers that the need and desire to publish would not be diminished by the publishing house’s policies on anonymity. It would therefore seem to be an issue of anonymity, or the lack thereof, coming down to its effects on the reviewer and in the worst case scenario resulting in not finding sufficient numbers or quality of reviewers to do a proper job or in effecting how they do their job. Have there been any new ideas presented or discussed that might prevent this impasse and also provide transparency at the same time – or is their simply no desire or reason seen to change something that has been done a certain way for a reasonably long time?
Excellent questions. Since I’m not an editor or publisher, I’ve not had to confront these issues directly. I’m aware of attempts to determine which approach is better, but I suspect that the continued existence of multiple systems means that results to date are inconclusive. Hopefully others can chime in with something more authoritative or helpful to the discussion.
A Witless Protection Program.
==================
Please go back to past experience, please. Ayn Rand, “Atlas Shrugged”, a dozen pages into Chapter III. Dr Ferris is from the State Science Institute, which is trying to control industrialists like Mr Rearden. Quote:
“Did you really think that we want those laws to be observed?” said Dr. Ferris. “We want them broken. ….. We’re after power and we mean it. …. we know the real trick, and you’d better get wise to it. There’s no way to rule innocent men. The only power any government has is the power to crack down on criminals. Well, when there aren’t enough criminals, one makes them. One declares so many things to be a crime that it becomes impossible for men to live without breaking laws. Who wants a nation of law-abiding citizens? What’s there in that for anyone? But just pass the kind of laws that can neither be observed nor enforced nor objectively interpreted—and you create a nation of law-breakers—and then you cash in on guilt. Now that’s the system, Mr. Rearden, that’s the game, and once you understand it, you’ll be much easier to deal with.” End quote.
Can you not see that it’s a “Trick” – a word which has various definitions – to encourage duplicity to make an honest man guilty?
There is NO place for a Doctrine of Justified Disingenuousness in science. It’s a control mechanism for control freaks. We have enough of those.
“There is NO place for a Doctrine of Justified Disingenuousness in science.
Oops, my coding didn’t work. The words “place for a” in the above should have been struck through. So allow me to try again:
There is no doctrine of justified disingenuousness in science.
In his book, “heh Climate Fix”, Roger Peilke Junior wrote of a number of papers in climate science that, as the authors later stated, were written for a political and not a scientific purpose. These were papers that were published in the most prestigious of scientific journals.
snip
sorry, this has nothing to do with “justified disingenuousness”. I’m going to snip the other side of this discussion as well.
Alfred Burdett https://climateaudit.org/2011/02/18/limits-to-justified-disingenuousness/#comment-256100
Twaddle yourself. When the editor specifically ignored some of the important policies he is supposed to uphold (primarily reviewer conflict of interest), then yes, it gives the appearance of gatekeeping.
You’re right, I’m as far from being a scientist as one could possibly imagine, but that doesn’t mean I have no appreciation for abrogation of duty.
“When the editor specifically ignored some of the important policies he is supposed to uphold ”
As I understand it, from what has been written here, the only relevant policy was that the editor should be made aware of any conflict of interest that a reviewer might have. But in the case of Steig, the only conflict we are aware of arose from the fact that it was his work that O’Donnell et al. had attacked. That conflict was obvious and thus needed no declaration.
The mere existence of a conflict does not necessarily disqualify a reviewer, particularly when the reviewer has been attacked and might reasonably expect an opportunity to respond, or where that reviewer is among the best qualified people to comment. (Steig may not have been the best qualified reviewer, but he certainly understood the issue since he published on it, and as is typical of most areas of research, there may have been precious few other people reasonable well informed of the subject matter of the paper for review).
Reviewer bias, real or potential, does not of itself constitute reason for disqualification because it is not the reviewer who makes the decision on whether to publish: that task is for the editor, who must weight the arguments and take account of potential bias that parties to the debate may have.
To argue otherwise would be to disqualify virtually every competent reviewer since anyone engaged in the particular field of research has an interest in promoting or demolishing particular ideas and hypotheses.
“As I understand it, from what has been written here, the only relevant policy was that the editor should be made aware of any conflict of interest that a reviewer might have.”
So you’re saying that since the editor was made aware (in fact he was aware before hand, since he selected Steig, and knew the conflict would be there) he doesn’t have to do anything else?
“To argue otherwise would be to disqualify virtually every competent reviewer since anyone engaged in the particular field of research has an interest in promoting or demolishing particular ideas and hypotheses.”
Straw man. There is no need to select the opposed paper’s author as a reviewer, especially not as an anonymous reviewer.
“So you’re saying that since the editor was made aware (in fact he was aware before hand, since he selected Steig, and knew the conflict would be there) he doesn’t have to do anything else?”
No I am not saying that and did not say that. If you want to know what I said, please re-read what I said.
as for:
“There is no need to select the opposed paper’s author as a reviewer, especially not as an anonymous reviewer.”
I did not say there was a need. What I said was that there were several possibly good reasons to choose Steig. Whether Steig reviewed anonymously or not was his choice, not the editor’s.
So, no, no straw man here.
You said “That conflict was obvious and thus needed no declaration.”
To me, such an obvious conflict should have been avoided. So, since it was obvious, the editor should have taken action to avoid it.
The paper in question is purely a statistical exercise, if I understand it correctly. Steig was not uniquely qualified to critique it, by his own admission as not being a statistician. So again, leaving Steig out of the equation would not have disqualified “virtually every competent reviewer”. That’s the straw man.
To me, such an obvious conflict should have been avoided.
To you, maybe. But that does not mean that the “editor specifically ignored some of the important policies he is supposed to uphold (primarily reviewer conflict of interest)” as you previously claimed.
So your statement that “the editor should have taken action to avoid it [recognized conflict of interest]” is unwarranted.
The journal policy was that there be disclosure of conflicts, not that those with any kind of conflict be prevented from acting as reviewers, which would be impossible since almost any qualified reviewer will likely be in some degree conflicted.
“Steig was not uniquely qualified to critique it”
No one said he was.
“So again, leaving Steig out of the equation would not have disqualified ‘virtually every competent reviewer’.”
No one said leaving Steig out of the equation would have disqualified virtually every competent reviewer.
Inventing straw men seems to be your specialty, not mine.
What, then, is the point of declaring a conflict of interest if the editor isn’t obliged to avoid such things when other, probably more qualified, reviewers are available. Can we conclude that such a declaration is an empty policy?
“To you, maybe. But that does not mean that the “editor specifically ignored some of the important policies he is supposed to uphold (primarily reviewer conflict of interest)” as you previously claimed.”
It seems so to many other posters here as well.
“The journal policy was that there be disclosure of conflicts, not that those with any kind of conflict be prevented from acting as reviewers, which would be impossible since almost any qualified reviewer will likely be in some degree conflicted.”
Again, you’re really reaching. Going from an extreme conflict of interest (Steig), to any kind of conflict of interest. I never said “any kind” of conflict. As you pointed out, the conflict was obvious, therefore it should have been avoided. And if the reviewer lettering system is any indication, Steig was the editor’s first choice, not the last, having exhausted any other possible candidates.
“No one said leaving Steig out of the equation would have disqualified virtually every competent reviewer. ”
You implied it by stating that review would be “impossible” if reviewers with “any kind of conflict of interest” were excluded. Your words, not mine. And again, I never said “any kind” of conflict of interest. Every juror has some preconceived notion of the outcome of a trial, there’s no way around that. But when a blatant conflict arises, such a juror is excluded. That’s my point.
This particular debate has nothing to do with “justified disingenuousness”.
Please don’t debate the same issues over and over again.
O.K., here’s my comment which I trust will be deemed on topic.
The discussion derives from the claim that:
“If you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse, it is necessary for you to pretend to not be a reviewer.”
But that claim is absurd, since it can require only slight verbal facility to avoid revealing that, as a reviewer, one has seen a manuscript prior to publication.
Even a direct demand to know whether one was a reviewer of a particular paper can be, as discussed above, easily deflected without dissembling or falsehood, e.g., by saying “if I did review the paper and wished to be identified I would have signed the review” — a statement that entails neither prentence nor falsehood.
So the claim that “Scientists expect this [i.e., disingenuousness] and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed” is nonsense.
Thus, when you say “Scientists may expect “disingenuous” conduct from other scientists, but the public doesn’t” you are making a highly misleading, not to say disingenuous, statement based on a nonsensical premise.
And your statement that “The last thing that the climate debate needs right now is more climate scientists being “disingenuous” in their communications with the public – there lies the road to ‘hide the decline'” you express a nonsensical and seemingly malicious non-sequitur, because the false premise that scientists are so clueless as to be unable to conceal whether they anonymously reviewed a paper except by dissembling, i.e., lying, is in no indicative of a willingness of scientists to lie about the scientific truth, something that is clearly implied by your use of the phrase “hide the decline.”
In fact, you seem to be engaging here in a verbal sleight of hand: using the ambiguity of a scientist’s words to imply an admission, where none was intended, that scientists are always ready to engage in scientific fraud.
One thing that will surely result from such views, which reflect so adversely on those who voluntarily participate in the peer review process, is that editors may experience great difficulty in finding scientists willing to review any paper that you may in the future submit for publication.
Steve- in the case at hand, Steig’s conduct was such that Lucia characterized him as the Rod Blagojevich of Science and Steig the Shameless.
Re: “Steig’s conduct was such that Lucia characterized him as the Rod Blagojevich of Science and Steig the Shameless.”
Lucia’s characterization of Steig as the Rod Blagojevich of science may or may not be justified — though it is hard to believe that any scientist could compete in dissimulation with a Chicago politician. But even if justified, such characterization of a journal reviewer is unlikely to be productive.
I devoted a good many of my earlier years to scientific research and thus can claim to be an old hand when comes to journal publishing. Experienced as I became, whenever an editor accepted any critical comment on my work, my reaction was always the same. I wanted to KILL the editor.
What I found, however, was that a more diplomatic reaction yielded better results. Thus I would employ a response such as the following:
Dear sir
(Or madam, if the Editor is a woman — you have to use your brains. I’m just spelling this out on broad lines.)
Thank you for your most perceptive and instructive comments on my manuscript in which you indicate that my presentation is abominable, my methods unsound, my hypothesis feeble-minded and my overall approach amateurish in the extreme.
As it happens, research in climate science is not my sole vocation. During the daytime, I work as a professional vendor of cat’s meat and am able to pursue my scientific interests only in the evenings after I have sold the last skewerful. Nevertheless, I feel sure that under your wise direction I shall, bit by bit, be able to improve my manuscript until it achieves the standard necessary for publication in your important journal.
Yours etc.
(with apologies to P.G. Wodehouse)
“What, then, is the point of declaring a conflict of interest if the editor isn’t obliged to avoid such things…”
Well it might be best to ask the AMS, not me. But one obvious point, is that the Editor, aware of a conflict can look out for and discount any consequent bias.
“Again, you’re really reaching. Going from an extreme conflict of interest (Steig)…”
I’d call it an extreme conflict if a reviewer worked for the Coal Alliance and tens of billions or hundreds of billions of dollars for carbon credits were on the line. What you’re calling an extreme conflict consists in the fact that Steig’s own analysis did not conform with O’Donnell’s. So are you saying only those whose work confirmed O’Donnell’s would be free of conflict. Obviously, that’s nonsense. There would still be a conflict, but in that case a conflict that would create bias in favor of, rather than contrary to the authors.
“You implied it by stating that review would be “impossible” if reviewers with “any kind of conflict of interest” were excluded.”
To equate reviewers with “any kind of conflict” with Steig and Steig alone is simply twisting my words.
Anyway, that’s it, as far as I’m concerned.
Good night.
Nielsen-Gammon paints a picture of a world where scientists fear reprisals, if they have the temerity to criticize the dons.”
But that slices other ways including the dons politicking privately against the peon scientists.
And in this subject, climate science, it seems to me that those on the “in” are using anonymity to protect their positions of power/money (funding).
IOW, N-G’s statement may have validity only in a narrow context.
“You know, I think that the only real moral crime that one man can commit against another is the attempt to create, by his words or actions, an impression of the contradictory, the impossible, the irrational, and thus shake the concept of rationality in his victim.”
Ayn Rand, “Atlas Shrugged” 1957 p 488 of Penguin Classics Plume 1999 pub.
To All,
This comment is triggered by the letter from the U.S. Department of Commerce Inspector General mentioned by the ‘New Light on “Delete Any Emails”’ post of today. The Commerce Department IG’s letter can be found at http://www.oig.doc.gov/oig/reports/correspondence/2011.02.18_IG_to_%20Inhofe.pdf.
The IG’s letter considers whether NOAA failed to adhere to appropriate peer review procedures for its own work, to the extent that such concerns might arise out of the Climategate emails. In the course of discussing that issue, the IG applied the mandatory peer review procedures issued by the U.S. Office of Management and Budget (OMB) in 2005, entitled “Final Information Quality Bulletin for Peer Review,” available at http://www.whitehouse.gov/sites/default/files/omb/assets/omb/memoranda/fy2005/m05-03.pdf. In turn, the OMB Bulletin “requires agencies to adopt or adapt the committee selection policies employed by the National Academy of Sciences (NAS) when selecting peer reviewers
who are not government employees.” The applicable NAS Policy on Committee Composition and Balance and Conflicts of Interest can be found at http://www.nationalacademies.org/coi/index.html.
The OMB Bulletin and the NAS policies are of course not directly applicable to the situation of Steig reviewing O’Donnell et al. They are, however, informative as to how the U.S. Government addresses conflict of interest and disclosure issues when work by one of its agencies is subjected to peer review.
As explained below, if the O’Donnell et al paper had been U.S. Federal agency work, then the selection of Steig by the journal to review O’Donnell et al would have constituted an impermissible conflict of interest under the NAS Policy on Committee Composition and Balance and Conflicts of Interest. The transparency procedures in the OMB Bulletin would have also prevented use of the proposal by Nielsen-Gammon that “If you are a reviewer and wish to remain confidential while remaining engaged in scientific discourse, it is necessary for you to pretend to not be a reviewer. Scientists expect this and know that reviewers may need to be disingenuous when talking publicly about a paper they have reviewed.”
As background, the U.S. Government “requires each agency to subject “influential” scientific information to peer review prior to dissemination. For dissemination of influential scientific information, [the OMB Bulletin] provides agencies broad discretion in determining what type of peer review is appropriate and what procedures should be employed to select appropriate reviewers.” **** Regardless of the peer review mechanism chosen, agencies should strive to ensure that their peer review practices are characterized by both scientific integrity and process integrity. **** “Process integrity” includes such issues as “transparency and openness, avoidance of real or perceived conflicts of interest, a workable process for public comment and involvement,” and adherence to defined procedures.” A scientific assessment is considered “highly influential” under the OMB Bulletin if the responsible agency “determines that the dissemination could have a potential impact of more than $500 million in any one year on either the public or private sector or that the dissemination is novel, controversial, or precedent-setting, or has significant interagency interest.”
With respect to the proposal by Nielsen-Gammon that reviewers “pretend to not be a reviewer” and recommending that “reviewers may need to be disingenuous when talking publicly,” the Nielsen-Gammon proposal is contrary to express transparency requirements in the OMB Bulletin. The OMB Bulletin requires that peer review reports for highly influential scientific assessments be made public, along with inter alia the identify of each reviewer. The operative section of the OMB Bulletin:
“requires that agencies instruct reviewers to prepare a peer review report that describes the nature and scope of their review and their findings and conclusions. The report shall disclose the name of each peer reviewer and a brief description of his or her organizational affiliation, credentials and relevant experiences. The peer review report should either summarize the views of the group as a whole (including any dissenting views) or include a verbatim copy of the comments of the individual reviewers (with or without attribution of specific views to specific names). **** The agency is required to disseminate the peer review report and the agency’s response to the report on the agency’s website, including all the materials related to the peer review such as the charge statement, peer review report, and agency response to the review.”
With respect to the standards for selecting reviewers and the choice by the journal of Steig as a reviewer, the OMB Bulletin requires as follows:
“b. Conflicts: The agency – or the entity selecting the peer reviewers – shall (i) ensure that those reviewers serving as federal employees (including special government employees) comply with applicable federal ethics requirements; (ii) in selecting peer reviewers who are not government employees, adopt or adapt the National Academy of Sciences’ policy for committee selection with respect to evaluating the potential for conflicts (e.g., those arising from investments; agency, employer, and business affiliations; grants, contracts and consulting income). For scientific assessments relevant to specific regulations, a reviewer’s financial ties to regulated entities (e.g., businesses), other stakeholders, and the agency shall be examined.”
The National Academy of Sciences Policy on Committee Composition and Balance and Conflicts of Interest, in turn, expressly says that “an individual should not serve as a member of a committee with respect to an activity in which a critical review and evaluation of the individual’s own work, or that of his or her immediate employer, is the central purpose of the activity.”
“Reviewing One’s Own Work
It is not uncommon for individuals serving on committees of the institution being used in the development of reports for sponsors to find that their own published and professional work, in common with others in the field, is part of the technical basis and literature for the committee. This ordinarily would not constitute a conflict of interest. However, an individual should not serve as a member of a committee with respect to an activity in which a critical review and evaluation of the individual’s own work, or that of his or her immediate employer, is the central purpose of the activity, because that would constitute a conflict of interest, although such an individual may provide relevant information to the program activity.”
I hope this is useful.
MK
2 Trackbacks
[…] controversy – and the supremacy of reviewer confidentiality (even to the point of “justified disingenuousness“), I thought that the following in the IAC’s guidelines was quite interesting: […]
[…] Source: https://climateaudit.org/2011/02/18/limits-to-justified-disingenuousness/ […]