Iridge versus TTLS. What if a key text on this conundrum of the day resided in an anonymous open peer review? Would we, within the ethical standards of modern climate science, be entitled to speculate on the identity of the author of these pearls? Or would that be an ethical violation “as bad as possible”? Even if we merely speculated privately, alone here on the internet, just you and I, dear reader? Sometimes truth is stranger than fiction.
Burger and Cubasch, a rejected submission to Climate of the Past (discussion paper here), tested a variety of multivariate methods on a standard set of proxies (MBH98), including the two RegEM variations of Schneider 2001 (Ridge, TTLS) that are the source of the present dispute.
The comments of Reviewer #2 here are of particular interest as he commented on the advantages and disadvantages of RegEM Ridge versus RegEM TTLS – the topic of today’s controversy.
Reviewer #2 featured RegEM Ridge as his primary case:
The RegEM method of Schneider (2001) accomplishes regularization through the use of ridge regression, introducing a “regularization parameter” h that specifies the degree of inflation (1 + h^2) of the main diagonal of the covariance matrix . The parameter h determines the degree of smoothing of the estimated missing values. Schneider (2001) uses Generalized Cross Validation (“GCV”) to determinate an objective estimate for h….
Reviewer #2 continued with a powerful condemnation of RegEM TTLS relative to RegEM ridge (see bolded section below), observing that “an optimal regularization of TLS leads directly to ridge regression, and not truncated TLS”:
the implementation of RegEM as defined by Schneider (2001) and employed by Rutherford et al (2005) involves only one statistical model, the solution of the above equation above using ridge regression for regularization, using GCV to select the regularization parameter h. It is true that there are a number of other possible ways to regularize the EM algorithm, including Principal Components Regression (PCR), truncated total least squares regression, and ridge regression. Schneider (2001) however specifically favors one unique regularization approach, ridge regression, since it arises as a regularization method when the observational error in the available data is taken into account.
Ridge regression regularizes a total least squares regression, provided the relative variance of the observational error is homogeneous. This assumption is appropriate when, as in applications to paleoclimate reconstruction (e.g. Rutherford et al, 2005; Mann et al, 2005;2006), available data series are standardized prior to their use in CFR. Even if this assumption is not met, the true regularized estimates are close to the estimates provided by ridge regression. The alternative models proposed by Burger and Cubasch are likely to provide estimates with greater variance.
Consider, for example, their use of truncated “Total Least Squares” (TLS). This is not an optimal approach to regularizing TLS. Under the assumption of homogeneous relative errors in the standardized data as discussed above, an optimal regularization of TLS leads directly to ridge regression, and not truncated TLS, which is indeed the reason Schneider(2001) employed ridge regression in the RegEM algorithm.
Given Reviewer #2’s endorsement of RegEM Ridge over RegEM TTLS, there is obviously a strong temptation for O’Donnell co-authors to try to figure out who he is and perhaps appeal for his endorsement of RegEM Ridge in the present controversy.
Unfortunately for this option, we have recently learned (through the kind intervention of Nielsen-Gammon here), that, even in the case of open review comments, ethical standards within the climate science community forbid speculation on the identity of Reviewer #2 (except in private). (See also here) Nielsen-Gammon:
Regarding your first question, whether it is unethical in my opinion to speculate on the identity of reviewers in open review systems, I would regard it as either unprofessional or unethical, depending on the circumstances. The reviewer has chosen to be anonymous or is required to be anonymous (depending on the journal), and that choice or requirement should be respected. It rises to the level of unethical if it is done in the context of a criticism of the reviewer, because the subject of the criticism may be unable to respond effectively and truthfully while remaining anonymous. One of the key purposes of anonymity is to protect the reviewer from personal criticism.
Regarding whether speculating privately is equivalent to speculating publicly, I think not.
Thus, much as the authors of O’Donnell et al would like to invoke the assistance in the debate of Reviewer #2 (with his strongly advocacy of the technique used in O’Donnell et al 2010), it seems that, within norms of the climate science community, it would be “unprofessional or unethical” to speculate on the identity of Reviewer #2.
Unless, of course, we do so privately. Like the Climategate correspondents.
Climate Audit 
116 Comments
Yes, it would be nice if this man could be called to assist in your debates. I’m sure the man would support you and criticize Steig’s methodological complaints.
Re: MikeN (Feb 18 23:55), yes, it’s amazing how strongly he supports RegEM Ridge.
I wonder who he is.Re: MikeN (Feb 18 23:55),
Afterward, he could reveal his name, taking a page from Popeye the Sailor Man with “I yam what I yam, and that’s all that I yam.”
I have to admit, I’m still trying to figure out if Ryan is standing by his remarks in the review responsess that Iridge improved the paper.
To some degree this all feels like the authors insisting that Steig take credit for improving their paper, and Steig insisting that “oh, no you don’t, I did NOT improve that paper!”.
I get that there is a degree of “he made us do more work” involved here, and y’know what? As a consumer I don’t give a rat’s *ss if the result was better science. Which my cursory read is that the primary author agrees that was the result.
But then I also read Steve’s comment in an earlier post that he thinks that the original submission should have been published nearly as-is.
So, maybe I should just ask –do the authors believe that the final product was better or worse for Steig’s involvement as a reviewer?
geo-
My understanding is that
(1) In the *context*of the paper– which is showing problems with TTLS (i.e. Steig’s method) a discussion using TTLS is clearer and more focused. It permits readers to understand that main problem in Steig is not use of TTLS specificially, but other factors. Going forward, this will help people who wish to use methodologies understand how to identify problems in methodologies and pick better ones.
(2) If you want a better reconstruction, iRidge is better in the sense of probably teasing out more reliable answers provided your underlying data is “good”.
So, whether the switch to iRidge makes the paper “better” is debateable. It deoends on what the main goal is. Oddly enough, main goal (1) is useful per se. Main goal (2) can be very useful but only if the input data are sufficiently good to permit the answer being useful after processing with the ‘better’ method. Using a better method on bad data would not be useful. In that context making the point in (1) would be better.
lucia–
Thank you, that does help my understanding of the issue.
lucia
“(2) If you want a better reconstruction, iRidge is better in the sense of probably teasing out more reliable answers provided your underlying data is “good”.”
Actually, if the data is “good” – in the sense of relatively small amounts of missing data and ‘noise’ (highish correlations) – then the choice between TTLS and iridge is less important, but iridge is likely to perform better.
But when, as with the Antarctica weather station data we used, there is not only a lot of missing data and ‘noise’ but also greatly time-varying patterns of missingness (which stations have data missing), ridge regression (both mridge and iridge) can be expected to, and does, perform significantly better than TTLS.
That is because the optimum level of regularization needed to avoid over or under fitting depends on the patterns of missingness. Ridge regression uses Generalized Cross Validation to optimize the level of regularization for time step (separately for each missing variable in the case of iridge). TTLS uses a fixed level of regularization (determined by the truncation parameter) throughout.
I don’t disagree that a detailed discussion of TTLS is helpful for understanding where Steig (or maybe more accurately his co-author Rutherford, who seems to have been primarily responsible for the S09 reconstructions) went wrong.
NicL–
I guess by “good” I meant measurement errors are not so large as to make the data almost unrelated to the signal you are trying to detect. For example: No statistical technique can be used to pull out a good answer from data collected from a thermo-couple that was actually broken, or pitot tube that was actually plugged. There is just NO signal in that data.
I much of what you are describing is what I would call “messy/very noisy” data. I would still consider that data potentially “good”. Of course, it could also turn out to be “bad” in the sense I described above.
The reason I used scarequotes is that I don’t think there is really a strict definition of “good” vs. “bad” vs. “messy/noisy”.
Dumb question Nic. What does “mridge” stand for?
Got it myself. The “m” in “mridge” stands for multiple.
Given what we’ve learned of late regarding the “culture” of peer-review – not to mention the doctrine of justifiable disingenuosness – it would not surprise me in the least to learn that there may well be some as yet undisclosed cultural norm and/or doctrine which dictates that, in effect, for some obscure reason of which we may not be aware, it may be permissible for them but not for others.
IOW, heads they win, tails we lose seems to be the underlying paradigm that determines the “climatically correct” interpretation of, well, just about anything and everything in the realm of “climate science”.
Looking at the exchange between reviewer 1 and reviewer 2, looking at how reviewer 2 interprets the NRC findings on Mann, I have a guess at reviewer 1 and a guess at reviewer 2.
Why does everyone think Reviewer 2 is a man, man?
what are we talking about? the anonimity in peer review is instrumental to peer review. once the paper is published, it’s not relevant any longer, and should be dropped so it doesn’t become a ticket for doing a steig
Omnologos – Can you not see a parallel between a prof failing a student in final examinations and the same prof recommending against publication of a young graduate’s paper? Why is the prof happy to be known to all at exam time, but to seek anonymity at review time?
There is a large amount of illogical commentary floating around here as people try to protect their favoutite positions. Remember, people tend to think more when their equilibrium is disturbed, so please think.
I see no reason for any participant to be no-name space.
C’mon Steve, let’s move to another game like whether the ozone hole chemistry is real, whether global sea level is indeed rising, whether dendroclimatology is dead, whether most past proxy studies are suspect because the instrumental calibration is suspect, like the REAL punishments given to miners who rort the books — there are many topics that people have now realised are dodgy and they are have started to back away from them.
Isn’t the analogy of a student evaluating his professor’s work better. The student creates a recommendation that may affect the professor’s work. Or more commonly, a VP asking someone for their opinion of on one of his pet proposals. A candid assessment may have sever career limiting consequences. Retaliation and coercion is inevitable. hence the need for anonymity.
Another example is the case of expenses. it has been a policy, fro the companies fro which I worked, that if people at different levels in the company are travelling together that the superior pay all expenses and claim these from the company. It would be too easy for a higher level person to coerce the subordinate into payments that are not justified. Retaliation is something that any managementpolicy must be aware of.
Why not simply cite Anonymous (2006) in your support? Its not ideal, its not peer reviewed and the only guarantee that the reviewer is competent is that the editor thought that the reviewer was worth contacting.
Isn’t that more or less what Steve has done?
The Herndon link is recommended to all readers. It is common sense. It will appear alien to those it accuses, who will have a difficult task to rebut it in a broad sense rather than by invoking existing habits.
Seconded – also, because in slightly different bureaucratic guise, I imagine this is happening in other countries as well.
It certainly rang some bells for this UK resident …
Geoff:
I do not see a Herndon Link. Did I miss soemthing?
neither me
Yesterday, upon a stair, I saw a man who was not there ….
http://knol.google.com/k/american-science-decline-the-cause-and-cure#
RegEM Ridge versus RegEM TTLS
Is the O’Donnell outcome very different using the TTLS approach? Does one approach lead to results more compatible with Steig? Surely that’s the real issue – and if the answer is Yes, the next question is Why?
Why is this debate taking place in review comments of climate journals anyway ? This is something that surely, if of any use at all for real problems, should be carefully reviewed in the statistics journals – by professionals.
The whole RegEM thing makes me uneasy. Are there any other scientific fields making use of this tool ?
I share None’s concerns about RegEM itself, particularly the way it incestuously recomputes the covariance matrix from its own previous-step infilled data. This gives a specific answer even when the covariance in question is not statistically identified (aside from the restrictions of positive definiteness).
RegEm per se is not the topic of this particular thread, so this isn’t the place to go into it, but I’m sure we haven’t heard the last of it on CA.
Meanwhile, on the perils of incest (also OT), I highly recommend the 2002 novel Middlesex by Jeffrey Eugenides. My wife and I couldn’t put it down!
Dick, if I remember correctly, the TTLS method first employed in the original version of O10 actually showed a smaller, statistically insignificant trend. Steig’s recommendation for iRidge in the comments comes about because Ryan used iRidge as a check on the TTLS reconstruction, and he mentioned in the paper that the trend in it was larger. Ryan also stated that the authors viewed that the higher trend (which was still significantly lower than the claim in S09) was probably closer to reality, but that would be something for future work. Steig pushed for the future work to be done as the main push of the O10, rather than continue with replicating the procedures he used, albeit with a different number of Principal Components. Much of his first 14 page review centered on the claim that Ryan et al. did not sell the case for using more PC’s than he had used (the kgrnd number).
Thanks, Keith.
That’s pretty much what I thought – and makes Steig’s attitude hard to understand. If both methods cast broadly similar doubts on his computed temperatures for the bulk of Antarctica, then arguing about the relative merits of the two methods, or about the legitimacy or even courtesy of such discussion, seems to let him off the hook with regard to the scientific result itself. Is the mud-slinging, intentionally or otherwise, misdirection (and rather effective misdirection)?
To speculate would be unMannly.
Man oh man!
Does that mean that if we are no longer speculating, that we can man up?
The Editor, at least, seems to think that Reviewer 2 may have been a womann.
In his decision he says:
“In our evaluation, we discarded the inappropriate tone of Reviewer 2 and his/her scientifically irrelevant points.”
You can see why s/he may decide to remain anonymous no matter how strong the urge to come forward and defend iRidge. Would you want to disclose your conviction for Gatekeeping simply bcause you wanted endorse the methodology of O10.
“Recent papers in GRL (including the M&M paper) have clearly not been reviewed by appropriate people. We have a strong suspicion that this is the case, but, of course, no proof because we do not know *who* the reviewers of these papers have been. Perhaps now is the time to make this a direct accusation and request (or demand) that this information be made available. In order to properly defend the good science it is essential that the reasons for bad science appearing in the literature be investigated.”
[1119957715.txt]
Mann oh mann. Its like you were shooting fish in a barrel.
Steve:
JNG’s approach leads to these logical and practical absurdities. We maintain anonymity (e.g., voting, jury voting) when it would be too difficult to protect individuals from some form of retaliation. In the case of peer reviews there would be legal recourse just as there is in labor law. Again, it seems to me the middle ground is to not grant anonymity to reviewers when an editor has an a priori basis for assuming a CoI.
As to this particular case – JNG would appear to argue that it is somehow inappropriate to even write to the editor in order to request contact with “Reviewer 2”. This is very bizarre and counter-productive.
Got to hand it to the Team. They’ve successfully made this about O’Donnell’s disclosure of Steig as a reviewer instead of Steig’s actions and O’Donnell’s results.
There’s no denying the Team’s skill with the techniques of Spin. However, the resultant front page Spectator article represents a spectacular backfire. It is true that the “Speccie” has a smallish circulation by number, but it is required reading for many an influential figure, not least most of the senior members of the UK Government.
This is pure speculation on my part, but I would be surprised if recriminations were not flying at Team HQ over the mistake in indulging Steig and his sordid little salvage operation.
Back to the topic at hand, I can’t remember the sequence of events surrounding somebody – in the spirit of the recent discussions I shall call him/her Dr X – citing Ian Jolliffe, who was an anonymous referee on a notable 2004 Nature submission, in support of his/her position on decentred principal component analysis. I certainly hope that this was done in full accordance with the ethical guidelines promulgated recently on these pages.
Incidentally, I am sure it is pure coincidence but the writing style of Burger and Cubasch’s Reviewer #2 appears to be remarkably similar to Dr X’s.
I agree, I think this line of argument plays into the hands of the Team. There are two core issues that need to remain front and center:
(1) the Team continues to use tricks to subvert and distort the peer review process
(2) “whatever was original in Steig et al 2009 was based on faulty mathematics; and that whatever was correct in Steig et al 2009 was already known.” (O’Donnell)
All other discussions merely distract attention from these important issues.
“Got to hand it to the Team. They’ve successfully made this about O’Donnell’s disclosure of Steig as a reviewer instead of Steig’s actions and O’Donnell’s results.”
Where are you now Ryan? There’s some more disinformation flying about! Someone is actually suggesting this was all about the science, and not a vendetta to gag you. All those countless threads about how it’s all a vendetta to gag you. All in vain. After all that hard work you put in on “Steig’s Duplicity”. All gone to waste. You must be absolutely gutted.
Boro Nut (Feb 19 16:31), “countless” “all gone to waste”..
It’s never “all” been about the science. As many of us note there are scientific, institutional, and personal issues on the table. The nuances of that make it interesting.
your overgeneralizations make you boring.
As someone who has had papers professionally peer reviewed in my own discipline – fire engineering – I find this argument over anonymity for reviewers baffling. It is not normal practice in my discipline, in fact the author is often advised directly who his/her reviewers are to be.
Why should climate science be different?
I think the reason for the invocation of anonymity is to preserve the fiction of review independence combined with the magic fairy dust of rigor to try to hide the strong smell of score-settling, gate-keeping and rampant abuse now emanating from all parts of scientific peer review.
The system is not simply broken, the patient is dying. The strongest smell is coming from climate science.
Peer review is mostly fine in Chemistry, John. It’s probably always been abused by ego-driven personalities, while the rest of us (the vast majority, it seems to me) soldier on with some conscious attention to professional ethics and obligations.
Anonymity in peer review means that I can criticize the manuscripts and papers of friends and colleagues, which I have done, without them getting angry with me. That is, anonymity in peer review preserves collegial and collaborational (and personal) relationships.
It’s an attempt to free the process from personalities and politics, and it’s a good idea. Most of the criticisms here of peer review are appropriate to the abuse of a good method, not to the method itself.
Any system is only as good as the people involved. The climategate scientists systematically and deliberately set out to subvert and corrupt the system. And in large part succeeded.
The system of anonymous peer review is worth saving, in my experienced opinion. But the climategate scientists need to be ejected. All of them. And the institutional editors and officers that went along with them.
If they are not ejected, or if there is no official recognition of what they did, then the people in charge of the system will have revealed themselves as irremediably corrupted, and the climate science peer review system will be screwed. (So far given the repeated UK whitewashes and the general approval, not to mention political exploitation, they have received, there is little reason for hope.)
Democracy is corrupt in a culture of corruption. But Democracy is not the problem.
Thanks Pat, well put.
“As someone who has had papers professionally peer reviewed in my own discipline – fire engineering – I find this argument over anonymity for reviewers baffling. It is not normal practice in my discipline, in fact the author is often advised directly who his/her reviewers are to be.
Why should climate science be different?”
“Eric Steig: As someone who has had papers professionally peer reviewed in my own discipline – climate science – I find this argument over anonymity for reviewers baffling. It is not normal practice in my discipline, in fact the author is never advised directly who his/her reviewers are to be.
Why should fire engineering be different?”
Phew. What a stroke of luck you ended up in Fire Engineering and Steig ended up in climatology then eh? Otherwise how baffled would you both have been? I shouldn’t let it bother you though. I seriously doubt you will be having one of your papers submitted to a climate journal any time soon, unless it’s “H2O : How to put out the Amazon” of course. Still, Eric Steig doesn’t get to ride in a Fire Engine either.
Re: Boro Nut (Feb 19 16:46), So, answer the question, why should climate science be different from other disciplines.
You see the problem here is a cultural one. Those of us from the culture of engineering know what proceedures and processes we use to ensure reliable results.
1. Hostile review
2. public review
3. Open data
4. Open code
5. publishing adverse results.
That’s our experience. If you want to convince us, then you need to understand your audience and what it takes to sell them. Since we understand your math and we understand your science ( some of us use it) we also understand how processes around these things can cause problems. So, we wont be bullied into just accepting results from a process that we
find laughable.
In my engineering specialty before work is reported out to other organizations it is evaluated by senior engineers and scientists in a
“murder board” manner. There is nothing anonymous about this and if the engineer in the hot seat is troubled by this, well that is too bad. This is
only one such review that takes place. The idea is to prevent the circulation of a report that may be loaded with misspellings, bad assumptions or wrong equations from getting off site. We had a real goatstoop several years ago when one of our engineers was getting set to brief some folks from back east and his vu-graphs had not been properly gone over by his immediate management. His presentation started off with 2nd grade typos and mis-spellings. Red faced, I cut off the presentation
and had to apologize to our guests. I then got hold of his managers and read them the riot act. This was an extreme example, most work is quietly circulated around for comments and errors are corrected in a more tempered environment.
Re: EdeF (Feb 19 22:21), these dont understand murder boards and are not culturally adapted to it. For those of use who have been thru them the whole idea of anonymous review is just plain weird.
Engineers learn that our review system requires us to separate the value of the evolving drafts of our work from our self esteem. To say that another way — just because I put my heart and soul into a project does not reduce me as a person to no more than the equivalent of that project.
Our review system has taught me to always strive to be the harshest critic of my own work. In that way I find it much easier to welcome other people criticizing my work and generally it improves my projects if I do so. Criticism of work products doesn’t need to be personal, and there is no good reason to take it personally.
So, yes, I find it laughable that anyone needs an anonomous review system. If the point is to get more honest reviews, the CRU emails expose that as a joke. If the point is to promote collegiality, the ongoing blog wars make it clear that it is not helping a lot of the participants avoid taking things personally. If the point is to improve the overall quality of the work, well, looking my reviewers in the eye at the comment resolution meetings and hashing the issues out in detail seems a hell of lot more productive to me.
Just as a small ‘thought experiment’, imagine if all engineering projects were henceforth required to include a significant component of originality in the work, extending the bounds of engineering knowledge. Any engineer that couldn’t do this on EVERY project that he worked on loses his career.
Signed reviews may be the most appropriate for Engineering: I certainly wouldn’t suggest that a few isolated examples in one field exposed it as a ‘joke’: http://en.wikipedia.org/wiki/List_of_bridge_failures
Re: Chris E (Feb 20 13:10),
Requiring “originality” has nothing whatsoever to do with the
process of review.
Further, did you actually look at the bridge failures and see if any were actually engineering failures. This is not to say that there are not engineering failures. The point I would make is twofold.
1. If you want to convince folks like us you had better take notice of our requirements. especially whe we explicitly ask you to.
2. If you use a different process than we are accustomed to you had better address
our concerns with that process. We are well aware of the limits of processes we used. Sometimes people die because of our mistakes. Lecturing us about the limits
of the processes we use doesnt help your case in the least.
3. Avoiding the issue on the table will get you nowhere in discussions with people like us. We fire guys for trying that shit. get it.
Steady on Steve, I’m not trying to convince anybody. The ‘issue on the table’ in my above post was jcspe’s comment that he found scientific norms “laughable”, which I felt deserved a response. I put up the bridge failure link as a response to a comment implying that the CRU emails make scientific norms as a whole a “joke”. And yes, I did check that some of them were engineering failures, and I actually studied some of them in my undergrad degree many years ago. If that affronted you I’m sorry, I meant no insult.
I am a bit puzzled about your other comments though. It should be clear from my other posts on this thread that in my opinion, scientists should be free to have their theoretical discussions in whatever way works best for them, but when their results have implications for the public then those results should be filtered through the Engineering profession.
I’ve copped criticism on submitted scientific papers for my work being ‘similar to that published by XXX, and hence not worthy of publication’. I’ve also been grilled for submitting engineering designs in the real world, on the basis that ‘we’ve never done it this way before’. The point being, that the two professions are different, and bagging the scientific process just because it’s not engineering is not valid. If scientists pretend to be engineers then by all means sink the boots in, but this is no reason to be critical of what should be an internal process in the scientific community.
I suspect that if we discussed this at length we would find ourselves in raging agreement; the problem in my view stems from a scientific ‘work in progress’ being inappropriately presented as ‘facts’. The academic review system does actually work, eventually, if it’s left alone to do its thing. If Al Gore and James Hansen had kept their mouths shut, then maybe by 2030 we’d have some supportable Science that the Engineers could do something useful with.
Re: Chris E (Feb 20 18:14),
Steady on Steve, I’m not trying to convince anybody. The ‘issue on the table’ in my above post was jcspe’s comment that he found scientific norms “laughable”, which I felt deserved a response. I put up the bridge failure link as a response to a comment implying that the CRU emails make scientific norms as a whole a “joke”. And yes, I did check that some of them were engineering failures, and I actually studied some of them in my undergrad degree many years ago. If that affronted you I’m sorry, I meant no insult.
#####
I missed your other comments. My mistake. No I don’t take affront to it.
Its an old debate, with engineers arguing that there is no such thing
as an engineering failure. There is a spec failure. ( haha)
I suspect we are in agreement. Sorry I was bouncing around reading random comments.
No stress, you can buy me a beer at the EGU conference in Vienna next April and we’ll call it square. 😉 Funnily enough, I’ve got a talk/poster to present there (the organisers haven’t decided which yet) about how several eminent forest fire scientists don’t seem to grasp the basics of statistics.
Re: Chris E (Feb 20 20:51), Sounds fair.
In regard to bridge failures, engineers in Canada wear an iron ring on the little finger of their left hand. I have read that this practice has been taken up by some engineering schools in the US. Originally, the iron for these rings came from th wreckage of the Quebec bridge. This is a bridge over the St. Lawrence River at Quebec City. It collapsed during construction. The river is tide water at that point. Construction workers ere trapped in the wreckage. Nothing could be done to rescue them from the tide. The only thing that could be done was to bring out priests to grant them absolution as they drowned.
Graduate engineers in Canada take an oath and wear the ring of iron in light of the responsibilities that they are about to take on. The example of the Quebec bridge disaster as exemplified by the iron in their ring is a lesson in humility for them. The ring of iron is not a badge of pride but a marker for humility. Engineers are well aware of the potential for disaster in their profession. As a result, there is an established practice of rigor in engineering research and practice. if the lives of millions or the life of one depend on your profession’s output then the use of hand waving arguments in research papers is not acceptable.
On the finger of their WORKING hand. Otherwise the lore is mostly accurate.
Building a better or safer bridge provides no mechanism to affect a fundamental change in government, policy, funding, etc. Therefore it is not a means to an ends other than building a better bridge.
This is the reality of comparing engineering to post-normal science. Those who want to build a better/safer bridge gravitate to engineering. Those who want to socially engineer the world gravitate to the arm-waving, post-normal side of things.
Re: Boro Nut (Feb 19 16:46), My question is simple.
Is your point that the review was handled adequately and that there is no basis for complaint?
‘Climate science’ should be different to Fire engineering because Science should be different to Engineering. Physicists develop equations, chemists imagine new retardants, engineers apply these essentially theoretical things to make the world a better place. Different standards apply because they are fundamentally different professions, and each has evolved methods that work well within the bounds of that profession.
If policymakers want a building fire-proofed, asking the chemists to do it would not be wise. Why is Climate different to Chemistry? Simply put, it’s because the policymakers DO talk to the ‘climate scientists’. Academic peer review mostly works just fine when nobody in the real world cares, but it’s not up to the job of adjudicating ideological disputes.
Maybe what is needed is a cadre of Climate Engineers (with impeccable statistical backgrounds), to stand between the scientists/advocates and the policymakers. The Scientists could then have their confidential peer-reviewed theoretical disputes away from the public eye (where they can’t do any damage), and the Engineers could develop real solutions to real problems with full accountability. If the Scientists want to say something directly to the policymakers, then they’d have to play by the Engineers’ rules…
You begin by pointing the obvious — science is not engineering; point taken. I was hoping you would then develop the theme and at least analyze why the process common in engineering is not suitable for science. And then end up proposing a whole new layer of “Climate Engineers” separating the climate scientist from the public. While the original question “Why the difference” still goes a begging.
Fair enough, I didn’t expound on the point too much because I thought it may be a little off topic, and because I’m largely just thinking out loud rather than presenting a considered opinion.
I would see the “difference” in that Scientists are required to develop new knowledge, which entails some imagination, risk and the certainty of plenty of mistakes and blind alleys. I guess the system of blind review has developed to minimise the limitations to imagaination while still including some quality comtrol. The primary directive in Science is to Find New Stuff.
Engineers on the other hand are expected to apply proven principles (albeit often in imaginative ways). The work of Engineers often has direct consequences for human life and safety, so their primary directive is Don’t Screw Up. If that has the unfortunate consequence of limiting their creativity at times, then they have to live with it.
The whole climate schemozzle is immensely damaging to Science, because some ‘scientists’ are now taking the position that their work quailifies them to make pronouncements on policy. In my opinion scientific outputs should be filtered through the Engineering profession before being considered ‘considered’ enough to be policy-relevant.
If the ‘climate scientists’ were not advocating huge imposts on the Public’s daily lives, would anyone in the Public care very much what they said and did? I would guess not. But if Climate Scientists produced research that convinced Engineers to insist that things be designed differently, then I for one would accept the necessity, because on the whole I trust the Engineering profession to Not Screw Up.
Just to try to avoid being flamed by both scientists and engineers, my basic training and practical experience is in engineering, but (alas), Forest Engineering is an almost-dead profession so now I work in science.
Engineers on the other hand are expected to apply proven principles (albeit often in imaginative ways). The work of Engineers often has direct consequences for human life and safety, so their primary directive is Don’t Screw Up. If that has the unfortunate consequence of limiting their creativity at times, then they have to live with it.
Mathematicans regard ‘rigor’ as an essential element of their profession. Using the analysis above, I suppose that this has the unfortunate consequence of limiting their creativity. Rigor is essentially the requirement that every statement made in a proof can be backed up. They do not have the attitude that mathematical methods can be applied in situations that violate their assumptions. Unfortunately many climate scientists appear to regard such requirements as impediments to their creativity. They can devise and use new methods without understanding their basis and sue to these to make sweeping statements.
If you think that engineering isn’t creative and original then you do not know much about engineering. It has to be more creative than yet another multiple regression on proxy project in climate science. How many have there been?
Sorry Tom, that wasn’t my implication and I hope no engineers took offence. By ‘originality’ in that context I meant ‘untested theories’.
Dr. N-G comes across as a serious, dedicated scientist. But I fear his approach on the current system of peer/pal review leaves open the very real possibility of the Tragedy of the Commons. It would take just a few self-interested individuals, intent on promoting their own view at the expense of opposing views, to game the system and thereby reduce the quality of the science. When discovered, this could lead to the public disparaging _all_ science. While the (meager) gains would go to the cadre of gamers, the loss of trust would affect us all.
Any system set up to work so long as everyone behaves as a ‘gentleman,’ and without rational restrictions on anonymous behavior, and no accountability for said behavior, is unfortunately doomed to failure simply because human nature is what it is. This is not a ‘robust’ system. It will be non-trivial to reform it into a working system without these drawbacks. Time to get to work.
I don’t get the point of this post. By pretending that what you are doing is speculating privately when clearly it is not, are you merely disagreeing with Nielsen-Gammon’s position that such public speculation is unethical? Or is the post just an opportunity to rub his nose in it because your arguments haven’t yet made him change his mind?
Chopbox:
Of course, I don’t know what Steve had in mind with this post.
However, I speculate that possibly the reviewer #2 could be the same as reviewer A??
Wouldn’t that be ironic.
But I thought Steve was scrupulous in not speculating on that point in the post – which I thought was a bit of dry humor and meant as a sort of joke.
The last comment, about speculating in private email – I assumed was a reference to that sort of behavior in the Climategate emails.
At least that is my take on this post.
I am totally guessing however.
Re: ZT (Feb 19 20:00), You might think so. 😉
Even if we merely speculated privately, alone here on the internet, just you and I, dear reader?
Studious ‘anonymous’ referee #2 fails to get ‘its’ correct in the eighth word of the response. Anyone might think that this was written by an angry, aggressive, email writer. One shouldn’t speculate, however.
This thread brings up another question I have had about the apparent need for secrecy that many involved in the peer review process think is necessary. Are the same human weaknesses that John N-G notes can lead to real or perceived intimidation in the peer review process transferable to what SteveM points to here, i.e. private communications outside the peer review process? Can we assume that what John N_G notes for peer-review could well be coloring the view of the consensus and that because of real and perceived intimidation that the so-called consensus is influenced significantly by these same human fears? In other words, is the consensus acclaimed by the IPCC and used as justification for moving rapidly on AGW mitigation, actually an artifact of human fears that are otherwise suppressed in the peer-review process.
Even the casual observer would, I think, see the lack of “private” comments outside the secrecy constraints of peer review on certain climate science papers. The authors of these papers are vocal in public (and private emails) as are critics outside the domain of climate science, but other observers from within climate science seem to be mostly silent.
Ahh….a climate science version of “Grand Theft Auto”. ‘Twas only a matter of time.
So David Smith if that picture is an example of what John N-G refers to when he notes the human weaknesses that must be protected against with anonymity in peer-review, I would suggest he does not go far enough. From those dangers, I would think that the reviewers words could be mixed and garbled a bit so no one would ever be able to determine who the wrote the review.
Kenneth, the whole idea of anonymous peer review and of misleading coworkers and other interested parties as to one’s actions is funny and sad.
These are the people who want to lead the rest of us away from the abyss, from our sins, so as to save the world? Hmmm, no thanks.
I like the mental image of climate scientists using the voice disguiser in the movie “True Lies” top deliver peer reviews. It’s the scene where Arnold Schwarzenegger doesn’t want his wife (Jamie Lee Curtis) to know it’s actually him.
Perhaps the reviewers of climate science papers should should be known as Boris, Doris and Natasha, in honor of the movie.
David, to me it is a Boxer (Animal Farm) moment when I hear obviously very smart people, like JNG, convince themselves they have to make ethical exceptions for the greater good, especially when alternatives seem to be extant.
If there were an alternative that clearly worked better, we would embrace it.
Honesty from all parties would be an appropriate alternative.
And Kenneth, to be sure, to be sure, the science should also be made garbled so that nobody can understand if it is good or bad or attributable to a person or a chimp with a typewriter. Seems the wheel is coming full cycle.
The post opens an interesting question.
What is “anonymity”?
Given a sample of writing and a limited population, there is no such thing as “anonymous”. I’ve done it myself a few times, and I’ve worked with others who did it as well. That was back before nifty tools like web searches.
That assumes, of course, that the editors are themselves ethical, and preserve anonymity. I would not trust that assumption myself.
as an outside to all this fancy mongering, I would say it is OK to have the reviewers guard anonymity for as long as the review procedure runs.
Once the article is approved they should all tell who they are, and make it public?
Actually it would be useful info to add to the articles abstract.
Common sense, people!
Think outside the enamelled goldplated taxoverpaid for “we did it always like this” – box.
modern climate science = research into the practical limits of justifiable disingenuousness
Posted at Stoat: #44, John N-G “Revealing the identity of the reviewer while simultaneously publishing the complete content of the reviews makes this particular ethical violation as bad as possible.”
John, the reviewer freely revealed himself to the author. Professional confidentiality is required by journal policy but belongs only to the reviewer. One that is given away, the reviewer has surrendered all ownership. Professional ethics were irretrievably breached by that admission alone.
After that breach, the only ethic remaining was that of personal recognizance. The author made a personal promise to to the reviewer to keep the admission private. But the professional barrier was already voided.
However, the reviewer then used the anonymity of reviewer status to launch specious criticisms. These criticisms were dishonest, in that they secretly abused the power reviewers have to require methodological changes in a study.
That dishonesty, in turn, voided the implied contract of non-disclosure. That is, the specious credibility of the reviewer’s later public criticism turned exactly upon, and required, the anonymity of reviewer standing.
This dishonorable abuse of privacy released the author from any obligation of his earlier promise.
Your criticism requires the reviewer to have remained honorable. He did not.
“As bad as possible”. Worse than creating entirely spurious results and hyping them in Nature magazine? Worse than all the well-documented abuses of the peer-review system exposed by Climategate?
JN-G tries to present himself as a reasonable man in his interactions here, but when he is on home territory, posting on Stoat, the Team member who was stripped of his Wikipedia administrative privileges for repeated gross breaches of editing policy, he can be seen in his true colours.
David S, my Stoat comments were quoted in ClimateAudit comments way back on February 11. By now they should be common knowledge here, and I don’t appreciate being accused of duplicity. If you’d like me to repeat them here again under my own name, I will.
Regarding the substance of your comments, by “as bad as possible” I meant breaching anonymity in the worst possible way. The universe is full of more serious breaches, and even within the realm of peer review I would hold scientists familiar with the peer review system to a higher standard.
Do you honestly think Steig et al. thought their results were spurious when they published them? Do you think their (anonymous) reviewers thought they were spurious too and thought “hey, this is garbage, but let’s let them publish it anyway…they’ll be grateful”.
Pat, Steig has reported that he revealed himself privately to O’Donnell to end the ongoing speculation and because O’Donnell had pretty much figured it out anyway. His freedom to keep his identity completely secret had already been eroded considerably. His actions since then demonstrate that he had no expectation that he would be publicly identified as a reviewer. One can argue that he was stupid for assuming that O’Donnell would treat confidentiality the same way that an ordinary scientist would, but that’s a different issue.
Meanwhile, my disagreement with you over what Steig actually did leads naturally to our disagreement over whether the implied contracts (personal and professional) were voided.
John N-G, can we agree on this much: if Eric Steig relied on his anonymity as referee to publicly criticize a method he imposed as a referee, then he had no further expectation of privacy.
On the other hand, if Ryan O’Donnell was disingenuous in his recitation of what Eric Steig did, then Ryan violated his ethical contract to maintain Eric’s privacy.
I’m not saying either one acted as described. I’m only setting up the conditional. Is it fair?
I nearly agree (sorry) with both of your hypotheticals. The only thing that gives me pause is that not only are the two individuals at issue, but so is the system of anonymous peer review set up by the journal.
I can imagine a qualified reviewer being asked to review a paper on a subject of some public controversy and thinking, “If I review this and am critical, or get misinterpreted, or make a mistake, or do something or other that the author interprets as being unethical, what’s to stop one of the authors from guessing or tracking down who I am and then accusing me of malfeasance all over the Internet? It’s simplest just to say I’m too busy. I just don’t need the hassle.” So the pool of willing, qualified reviewers is diminished.
Now, under your first hypothetical, S would have done harm to the peer review process through his actions. This doesn’t necessarily excuse O doing further harm, at least not without first attempting to pursue remedies with the journal or the professional society.
P.S. BTW, I think that neither hypothetical fits the actual situation.
P.P.S. Let me take this opportunity to amend my previous comment at 7:45 PM above. I DO appreciate being called duplicitous if I have been such.
“Pat, Steig has reported that he revealed himself privately to O’Donnell to end the ongoing speculation and because O’Donnell had pretty much figured it out anyway.”
So how are we to interpret Steig revealing himself in this way, and with this timing?
If O’Donnell is certain that Steig is Reviewer A without personal confirmation, where is O’Donnell’s obligation to keep quiet about it?
On the other hand, if Steig is certain his identity is known anyway, making the (now redundant) admission to O’Donnell gives him an opportunity to impose conditions that he otherwise wouldn’t get.
Great post, Pat.
Thanks, Phil. William Connolley is a great editor.
I’ve replied to his and Marco’s responses there, but my reply is being held for moderation (unlike my first post). Will it emerge fully formed, or be WC-mutated, or even evaporated? Time will tell.
Except, Pat, I think Ryan crossed a line when he asked Steig if if he was reviewer A. In doing so, he put Steig in an awkward position. He could either lie, say nothing (which would probably be an admission by silence), or tell the truth. He chose the latter, to his credit.
However, Steig’s later antics in renouncing the method which he felt the Editor should insist upon are problematic.
Jeff, all Eric would have had to say in reply to Ryan is, ‘Ryan, that question is out of line.’ There’s no awkwardness at all. Eric would merely need argue the principle.
It seems to me that all this attention to the “letter of the law” in scientific practice is completely beside the point. The truth is that none of the participants have done anything that isn’t very familiar to anyone in any field. The institutions of scientific research aren’t a full-fledged legal system, and were never meant to stand up to bad-faith participants. When those bad-faith participants show up, the community usually deals with them by closing ranks and using “extra-legal” means to protect itself.
That’s why the defenders of climate orthodoxy have such an easy time falling into, and defending, their practices. They honestly believe that they’re battling creationists, or perpetual motion hawkers, or the equivalent, and everybody in science has plenty of experience with *those*, and knows that when faced with particularly clever and persistent ones, you “do what you have to do”, so to speak.
Of course, sometimes the “cranks” aren’t cranks at all, and it’s the field itself that’s gone off the rails. (That happens more often, I think, than a lot of people realize, though usually on a small scale–a little community of researchers spinning up in mad pursuit of some misguided obsession, to the consternation of colleagues.) In those cases, it’s the job of other peer disciplines to play the role of the “scientific community”, and activate its collective immune system.
So where are the statisticians–other than our host and a few scattered colleagues? Surely shredding statistical malpractice in other disciplines–especially big-impact ones like climate science–is worth the attention of a few bigshots in the field? What about the botanists and the use being made of trees as proxies? If things are as bad as the “coffin, meet nail” post suggests, this field should be a goldmine for fish-in-a-barrel publications in other fields, where the global warming establishment hold little sway. Why isn’t it being mined more intensely?
I posted some of this on an earlier thread. I hope it is worth repeating here. Below is the policy from a British university thaat deals with the role of reviewer and author.
“Those taking part in peer review should:
apply vigorous objectivity in all assessments;
review in accordance with the guidance provided for the process, complete the review as specified and on time;
respect the confidentiality of any information sent for review and not disclose any information provided, any opinions given, or the details of the invitation to review;
report any conflicting interests as required by the requesting organisation and University policy;
not allow vested interests or personal bias to influence their imparrtial assessment of work to be reviewed in either apositive or negative way;
only accept assignments for which they have the expertise, returning any which are outside their expert knoowledge;
not take advantage of any new data or privileged information they have had access to during the review process either in the capture of ideas to further their own research and/or activities;
conduct a fair assessment of the work and not deliberately disadvantage a competitor in the field;
review objectively work that challenges acceepted views, crosses traditional boundaries and /or is wholly innovative;
be aware that the review may identify practice which falls below good conduct (whicch might be genuine error or malpractice) and which should be reported in coincerns.
Submitting work for review
Researchers should not take actions, directly or indirectly, to influence the review of their own work or that of others, positively or negatively.
Where work is reviewed the authors usually respond to reviewer comments. authors should accept comments and respond to the factual points made. where an author supects an infringement of the principles outlined above this should be reported to the appropriate authority (e.g. journal editor, grant manager).
Where an author considers there might be reasonable grounds for appeal he or she should first discuss the details with colleagues within the university.”
In an comment on an earlier thread WillR said he found this policy “idealistic”. I don’t agree. I find this a practical policy. A policy such as this usually starts as a draft by the senior person in Human Resources. It is usually circulated to managers and staff representatives. If lay representatives wish, they will usually have access to professional advice. After all have had a say and these representations have been considered and the policy amended as thought necessary it wil become a part of university rules. Some rules, not all, will be incorporated into the employment contract giving further weight to them. The purpose of the rules is to try to ensure that employee behaviour conforms to the standards of the university. In particular, disciplinary procedures are intended to be normative of employee behaviour.
Nothing is said here about anonymity or being disingenuous to preserve anonymity. I doubt if there is a problem in this university or in any university that I have encountered that requires peer review to be done anonymously. If it is a problem I would expect to see an explicit reference to it in university policy and procedure.
I have seen, in a different university, reference to honesty being a guiding principle of behaviour in research and the review process to be undertaken with regard to that principle.
One way of testing the matter of anonymous review might be to try to find an example of univerrsity policy which refers directly to the problems of reprisal raised by Nielson-Gammon. Does the University of Texas, for example, identify this as a problem in any of its policies and procedures and how does it deal with the matter?
In my view, university procedures offer a way out of some of the problems discussed in this blog regarding the behaviour of climate science. Complaints about alleged breaches of university policy in the conduct of peer review should be raised with the reviewer’s university (the Human Resource department), the reviewer and the journal editor. It should be stated that unless the reviewer conforms to the policy requirements of journal and university a formal grievance will be made. If necessary, a grievance should be lodged and seen through to the end. It is probably counterproductive to raise such issues on blogs.
I recognise that these things are a pain to do. But if one has gone to all the trouble of being an author, it seems to me to be absolutely necessary to try to ensure, through the university that individuals do not behave as they wish rather than as their employer would wish.
Suppose there is difficulty obtaining data that a blogger or commenter wants to try to replicate a bit of work. Look at what the university policy says. Most will have adopted a policy of making data available. If a request for data is rejected when, in accordance with policy, it should be available, tell the Human Resources department with your second request. Make it clear that a formal grievance requesting that the university take disciplinary action will be lodged if university policy is not met.
If volunteers can be found to make the use of university policies a systematic approach to dealing with bad behaviour all to the good.
You can get your boots that a number of the Climategate offenders will have had it put to them in no uncertain manner, informally, that there is a need to change behaviour. Their coats might yet not be on shoogly nails but that message will have been conveyed.
We’ve read the speculation, but still have not mannaged to figure out who it could be? Any assistance would be greatly appreciated.
Regards,
A.H. Ockey
S. Tick
C. Reator
You might be interested in this editorial in the BMJ last century.
You have to register to read the entire editorial.
One interesting thing I noticed was this:
It seems the BMJ employs in house statisticians which is something that many of the climate journals should consider.
In other journals where the atmosphere is far more amicable, it works as follows: for the period of the review all reviewers are anonymous, but reviewers tell you after publication themselves “Hey BTW, I was your reviewer A.”
Anonymous review is useful during the review process because you focus on scientific arguments and not on the person. After publication it’s plain silly to maintain anonymity.
IMO that is because in “normal” science the politics is local. In climate science because national and world politicians were called in so early, politics is not local anymore. So it is by definition pettier.
“Democracy is corrupt in a culture of corruption. But Democracy is not the problem.”
I think Pat Frank’s argument for anonymity in peer-review is a little too pat. Obviously pure democracy without any constraints for protecting individual freedoms can be as tyrannical as any system that prohibits/inhibits voting. It would be a travesty to protect such a system or even a system headed in that direction with throw away lines. Likewise it is simply too easy to throw up anonymous peer-review and simply point, in the broken window fallacy approach, to what it is supposed to cure. What is needed is a discussion of the history of anonymous peer-review, the product that it has put out in areas where it is strictly practice versus those where it is not, documented benefits and problems with it and alternatives to it.
You’re right, Kenneth. I should have qualified Democracy, with ‘constitutional.’ And I’d further qualify constitutional with ‘grounded in Enlightenment political philosophy.’
“Unfortunately for this option, we have recently learned (through the kind intervention of Nielsen-Gammon here), that, even in the case of open review comments, ethical standards within the climate science community forbid speculation on the identity of Reviewer #2”
It seems to me that this is a reasonable choice amongst several as to how to handle matters of this type. The various treatments each have advantages and disadvantages – or perhaps better just to say different, equally valid effects.
The effect of treating matters as described here is to put massive weight on the original review comment. In the absence of being able to call on the support of the author, the comment itself must be treated as highly authoritative.
Obviously, one can argue the merits of this approach, but I don’t see that there are objective criteria by which one might argue it’s inappropriate.
A quick question: It is the editor who ultimately decides whether a paper is published or not and therefore it is that person’s human weaknesses that might easily corrupt the system or at least make it work less efficiently if that person has biases (a given potential for the rationale of using secrecy). If secrecy is the answer in avoiding biases then what protections are afforded the editors decision making process.
What if there was a pool of eligible reviewers from whom the editor selects at random and never knows who did the review? It might take a little effort to screen the communications in an effort not to reveal the reviewers identity, but if preventing human weaknesses from entering into the system is preeminent then I guess it would be deemed well worth the effort. Establishing a pool might be performed by an algorithm using the history of potential reviewers and then notices sent in clandestine fashion to the reviewer pool.
Further posted on stoat: Really, one has to laugh at the commentary here about my posts.
In #62, William ignores the meaning of “reasoned” in the “reasoned views” of a reviewer, in order to suppose that an editor can reject a reviewer’s cogent criticism without repudiating his own choice of reviewer.
Reviewers have the power and right to demand changes, William, on the consequence of recommending rejection. An editor can accept or reject that judgment. However, given that the reviewer validly supported that judgment, then an editor who dismisses the judgment is professionally remiss.
Your judgment of my experience of peer review, on the other hand, is of no consequence, especially considering that your reply ranges from tendentious to facile.
I am indeed the guy you found exemplified at labome.org.
#63, Eli, you wrote: “Editors ALWAYS have the right to ignore or accept the advice of referees and trying to say they cannot is arrogant idiocy.”
Except that I wrote, “Editors have the power to ignore the reasoned views of the reviewers they, themselves, have recruited..”
So, on the one hand I’m not guilty of “arrogant idiocy,” and on the other your statement leaves us wondering what one may label a misrepresentation of the obvious.
If I, as an author, disagree with a referee, I never explain why a s/he’s “a fool.” I carefully explain why I think s/he’s wrong. Your disrespect trivializes the process.
In peer review, it is always the reviewer’s call to label error, but the call must be demonstrated analytically or by reference to published science. If the reviewer’s argument is correct, it’s the editor’s professional obligation to support that call. To do otherwise is to violate the obligation.
And to suppose that editors may be whimsical in their decision (Wlliam: “editors, not reviewers, make decisions about what is acceptable and what is not…”, and Eli: “Editors ALWAYS have the right to ignore or accept…”) is to completely misconstrue the review process.
Here is what may be a 100+ revelation for you, Eli: being thoughtfully wrong on occasion is an honorable estate in science.
William, regarding your response to #64, I think it’s weird that Eli refers to himself in the third person.
You’ll be relieved to know, in any case, that my last first-author paper was in 2009, two in 2008, and more to come.
Given the nature of the conversation here, William, I’d appreciate it if you’d not snip my posts for the duration. I promise to be civil and on topic.
Posted on stoat: #66, William in response: “[Ah, excellent, thank you. That makes everything clear: PF is one of the wackos…”
Let’s see you refute the E&E paper on analytical grounds, William.
… “(so it is entirely appropriate he should be on this thread).”
I came onto this thread because I was alerted to the poor thinking that was already on display here.
“Presumably that means he is deliberately misrepresenting peer review rather than just ignorant of it -W]”
Good job, William. There’s nothing like character assassination to win a substantive debate.
Down below, in #67, Hank referenced my earlier conversation with Michael Tobis. In a nice parallel with your tactics, William, Michael enriched his position with an unpleasant innuendo, but then went on to close comments. Presumably, you have more courage than that.
Coming from an accounting background, I am bemused by all this talk of the need to hide the identity of reviewers.
(Before I begin I acknowledge with great sadness that there have been too many cases of “pal reviewed” audits).
I was never audited by a “pal”.
As a commercial accountant, being audited was never pleasent.
Good auditors make it a practice of not getting too friendly with clients and keeping a proper, business like distance between them.
The worst audit I ever went through was by a former school friend.
He gave me hell.
It turned out that there were aspects of accounting theory that I had not fully absorbed into practical routines, (although I could reel them off it asked).
So my accounts had to be revised at the eleventh hour in a mad rush to meet company deadline.
I succeededed and the accounts then received a clear audit and I learnt an invaluable lesson.
From then on, my accounts were never queried.
Thanks to my former school chum.
I met the deadline.
Looking back, he grew in my respect.
In later years, we frequently met on cordial terms.
But I would never reguard him as a friend.
In my mind, there is work,
there are good relations between colleagues
and there are friends, true mates outside work.
If we mix these up, we do so at our peril.
The shortcomings of climate review by mystery men have been exposed.
Time for a change.
But no more speculations, please:
Click to access cp_news.pdf
http://www.agu.org/journals/ABS/2007/2006JD008272.shtml
Anyway, might be a bit annoying for the author that referee debunks your work with a paper that is not yet published.
Timeline:
4 July 2006 – accepted (source: Anonymous Referee #2)
22 November 2006 – received (source: AGU, http://www.agu.org/journals/ABS/2007/2006JD008272.shtml )
26 November 2006 – in press (source: Mann, Climate Over the Past Two Millennia, AREPS preprint)
20 February 2007 – accepted (again?) (source: AGU)
23 June 2007 – published (source: AGU)
???
This timeline is disturbing. Did anonymous Reviewer #2 actually cite as ‘accepted’ a paper which had not even been submitted yet?
Had JGR solicited such a paper from Mann et al. before July 4, 2006, or was there any basis at all for use of the term ‘accepted’ by Reviewer #2
There may be a prima facie misconduct case to press with the editors of Climate of the Past. Remember what the editor’s final note said about the verbal behavior of Reviewer #2:
Climate of the Past does not rate well at all on handling reviewer criticisms of Juckes, et al (2007), either (this time ignoring all criticism):
(a) protect Mann & friends from responding adequately to criticisms in 2006 by allowing abuse of review process;
(b) protect Mann & friends from responding adequately to criticisms in 2007 by allowing abuse of review process.
Is there a hint of a pattern to be examined here?
As a layperson with only one peer-reviewed paper to my credit and with no desire to have private peer-review processes be regulated by government bodies, I continue to be a bit frustrated by the lack of a more in depth discussion of the peer-review process and the motivations for using anonymity. I truly think that any changes would have to come from within the organizations involved with peer-review and that will only be instigated when these organizations feel some pressure on their collective credibility.
I keep going back to my view of what is really at stake with more open peer-review and how my experiences in communicating with scientists and technical people bears on my view. I simply do not see a major problem with scientists openly discussing and criticizing their view of the science and interpreting scientific results as long as the topic remains the science. In fact, a true and pure scientist wants his work commented on by others in his field as he knows that is the only way forward. What then is it that would bother scientists in these discussions. I think it must be that they do not like being assessed or judged based on their personal abilities as applied to the science, i.e. criticisms of scientific works becomes a personal issue. The issue may well then be bruised egos and the ill feelings arising from knowing who is doing the real or perceived bruising.
What then is the way out of this seemingly tough conundrum. Surely it cannot be invoking an anonymity process where that anonymity puts the editor in a position where that secrecy can hide the biases that the secrecy of the process attempts to avoid in the reviewers. After all, the editors must be every bit as human, and with those weaknesses, as the reviewers and authors.
Also it must be that science is done every day outside the peer-review process and that communication cannot be done anonymously even though I would think that the same human weaknesses apply.
Kenneth, I share your frustration with many of the comments on peer review. Some of the issues are nuanced and too many commenters make stereotyped statements.
Steve, Kenneth – I dug around a bit looking for some good, detailed discussions about the current peer review system, its problems and alternatives. The journal Nature hosted an extensive set of contributions about peer review which I hope are old enough to be outside the paywall. There’s also some more recent stuff, such as this entry, at Adventures in Ethics and Science.
Steve – John, I am familiar with the Nature discussions, but thanks anyway. Unlike some readers, I am not trying to re-examine the entire practice of peer review – as opposed to its implementation in the present context of the Team .
Who was that masked man?
“…ethical standards within the climate science community…>
Isn’t that a questionnable description given what was revealed by the leak from the CRU of EAU about key “scientists”and the behaviour of some “scientists” in conferences and media appearances?
How is it that the “climate science community” gets to have its own standards – shouldn’t they be the same as for any science?
“Using a better method on bad data would not be useful.”
Gosh, I thought process was everything these days. 😉 Oh! you mean the _total_ process should include ensuring the data is of good quality. What a concept!
Re “omnologos” on Feb 19 at 4:08 pm:
– so the peer review process is at arm’s length, without the ability for Sally to phone George and say “what are you talking about – what do you mean by [specific phrase]?
– certainly after publication reviews should be revealed, if only to show the process is being followed
Sorry, my reaction to much of the debate is flippant.Sounds like a bunch of amateur lawyers (a category that includes huge numbers of individuals with law degrees and licenses to practice law) playing political games – which includes making smoke screens and picking on small things to discredit the opposition in the eyes of those too dense to look at principles. It can seem difficult to really nail such people, except from their pattern of behaviour. (Note that people caught in a bind of their own making often flop around as they don’t know which way to go to escape. The other tactic often used is to keep repeating the mantra – climate alarmists do that regularly, both repeating their alarmist line and dealing with a setback by repeating a phoney positioning line such as “Climategate didn’t change the science” (merely discreditted their version of it ;-).
Stephen McIntyre, is this type of statistical analysis common in mineral exploration geology, where theorizing about the size etc. of the ore body is a big deal financially and attracts slippery behaviour or worse? If so, is it misunderstood and misused as much as in climate “science”?
I once did preliminary investigation work on complaints against professional engineers.
Anything where speculation might occur – minerals, petroleum, and land development – had a rate of rejection far far higher than complaints about work on buildings.
In one case I made a matrix of statements by several people on several aspects of a case – to my surprise there were very few aspects where even even two people in several were saying the same thing.)
(Speaking of mineral exploration, perhaps there is a parallel between some of the IPCC’s claims like Himalayn glaciers and the Bre-X scam, a basic salted-core case which was revealed by simple due diligence (even though big exerienced consulting companies had supposedly already been involved and people had already invested huge amounts of money.)
Steve- the Bre-X fraud was not discovered by simple due diligence, but by a complicated chain of events in which a joint venture partner was imposed on Bre-X.
John N-G thanks much for the links. I have read about a half of them to this point and found the idea of giving reviewers and reviews space in the published paper the most intriguing. If the prestige issue of publishing is important and reviewers can get recognition and prestige from doing the yeoman’s work of reviewing that would in my mind overcome much of the hesitancy to review and to do it anonymously.
The fact that the publishers and editors control the process comes through clearly and that the potentially better ways of doing reviews online is there for current publishers to use or for new sources of publishing, in competition with the old, to use.
While a collection of thoughts from people who might take initiatives or want change can sound hopeful for change, I recognize that fundamental change in these circumstances is something that takes a lot of convincing to get away from the comfortable status quo.
SteveM, I think I know why you are presenting examples here of how the peer-review system can go awry; it is just that I judge that a system that is easily gamed has more wrong with it then the people gaming it.
2 Trackbacks
[…] This post was mentioned on Twitter by Science Blog News, Mario Gregorio. Mario Gregorio said: Climate Audit – Speculating Privately http://bit.ly/fJ2sHy […]
[…] Speculating Privately Iridge versus TTLS. What if a key text on this conundrum of the day resided in an anonymous open peer review? Would we, […] […]