As readers of this blog know, Juckes et al submitted a paper for online review at Climate of the Past Discussions. See here for discussion. There were many unsatisfactory and even distasteful aspects to this paper. I submitted a detailed online review, as did Willis Eschenbach and another CA reader. I spent time rebutting a variety of unsupportable allegations about our paper. I did so in the belief that the online review process at CPD was a bona fide process. It appears that this belief was mistaken.
The responsible editor, H. Goosse, was a serial coauthor with Michael Mann and not particularly well-disposed towards the MM criticisms of MBH. Although I was an invited reviewer of B’rger’s re-submission, Goosse made no reference to either my review or to Willis’ review in his comments to Juckes et al. However, he did indirectly call for Juckes et al to “strongly reduce” their section purporting to criticize us and only “briefly” mention this controversy:
One exception is section 3 “critic of the IPCC2001 consensus on millennial temperatures”. This part is devoted to a very specific topic, difficult to follow for readers who are not familiar with previous work and to my point of view is not clearly connected to the other parts of the manuscript even in the revised version. This section is already long compared to the other ones of the manuscript, although some parts would require some more detailed information. I consider thus, at this stage, that this discussion should be much clearer if this was let that to another paper or note specifically devoted to this subject. In agreement with the Referee, I would thus recommend that the authors strongly reduce this section and briefly mention the controversy about the “IPCC2001 consensus” in section 2.
So while Goosse was undoubtedly not inclined to do us any favors, he clearly did not accept the Juckes submission. B’rger went to a considerable effort to re-submit a CPD submission and I guess that most of us assumed that Juckes et al would re-submit, just as B’rger had to re-submit.
However, we’ve seen no re-submissions. However, if you look at the references for Ammann and Wahl 2007, you will see:
Juckes MN, Allen MR, Briffa KR, Esper J, Hegerl GC, Moberg A, Osborn TJ, Weber SL, Zorita E (2006) Millennial temperature reconstruction intercomparison and evaluation. Clim Past Discuss 2:1001’1049
So Juckes et al don’t seem to have bothered going to the trouble of re-writing to meet referee comments. But it’s still cited in a Climatic Change article as though it was peer reviewed. We’ve seen examples in the Ammann/Mann corpus of academic check kiting. Surely this is a case of review avoidance if not actual review evasion (borrowing the terminology from tax law.)
And surely this is damaging to the reputation of Climate of the Past and should be protested by its editors. The CP experiment was an experiment in online and open peer review. CP Discussion editors had minimal requirements for posting online, presumably on the basis that online review comments would be taken seriously by authors. Ammann, Wahl, Juckes, Allen, Esper, Hegerl, Moberg, Osborn, Weber and Zorita have demonstrated a total disregard for the CP process by citing (and allowing their article to be cited) even though CP editors had asked that changes be made. And now Climatic Change has acquiesced in this continuing degradation of the currency by permitting Juckes et al 2006 to be cited as though it were a peer reviewed article.
Time for Climate of the Past editors to speak up.
Climate Audit 
23 Comments
One of the major advantages of submitting a paper to Climate of the Past is that it is almost instantly published, and therefore citable, in Climate of the Past Discussion. By minimising the time between submission and publication the risk of being scooped is averted.
Anybody citing papers in CPD should be aware of their status.
AW07:
The problem is that some authors are too lazy to read relevant literature.
Just like in WA07:
Sounds like MM03 made the mistake.. Sheesh.
Nothing surprises me any more. Either things are really going downhill, or, I am simply getting old, cynical and bitter (with ample scar tissue) …. LOL!
I haven’t seen the full details of the editor’s recommendation but I haven’t seen anything to suggest that the paper was rejected and asked to be resubmitted. The editor has asked for revisions. If the author revises the manuscript, either accepting the criticisms or rebutting them then it is perfectly normal in most journals for an editor to make a decision as to whether that revision is acceptable or not. Its true that if the author disagrees with the reviewer then its highly likely to be sent for further review, either to the original reviewer or a new reviewer – and in this on-line experiment reviewing, probably as a new submission for reviewers to comment upon. However, the editor makes the decision and can decide whether or not the revision is acceptable.
Clearly as it is published, the editor accepted the revised version.
The crunch question is, do you feel they have ignored your criticisms and not revised the paper – is the paper still factually incorrect? If so then your only real recourse is to write a comment or a rebuttal.
Oddly enough, when I searched for this paper on Web of Science (the leading method of searching journal papers), it did not turn up! So perhaps Climate of the Past Discussions does not get referenced in this database.
I like the “Hey, It’s Climate Science!” tag myself.
Steve:
There is an interesting column in the Wall Street Journal today about the efficacy, or lack thereof, of the peer review porcess in the medical research literature. Many of the points raised generalized to another applied science discipline.
It’s not like I think that “peer review” as carried out in climate science is any imprimatur of correctness. My point here is that Juckes et al did not complete a peer review process as defined by CP. Here’s what they say:
The Juckes paper only went to the CPD stage and there is no notice of it going through the “second stage” in which the “peer review process is completed”. All its undergone is a “rapid access-review by one of the editors”. Should be good enough for IPCC.
there is a guide to the procedure used at:
http://www.climate-of-the-past.net/
you can see the first stage, and discussion paper, online at:
http://www.clim-past-discuss.net/2/issue5.html
however, i think it is clear from this that the paper has only been into the first stage of the discussion process, and that a final, peer-approved version has not been published. The paper as is has the caveat of numerous strong referee criticisms (plus the editors) hanging over it.
I do not see this is a problem for CPD, or their editors, because they are completely transparent about what the process is, and where this manuscript is in the process.
It may be a problem for the journal that published ammann and wahl 2007, since they are relying upon a reference which has received severe criticism in peer-review, and has demonstrably not yet passed that process.
per
The journal is Climatic Change, whose review process for Wahl and Ammann is already very tainted by their apparent commitment to get Ammann and Wahl in print for IPCC. In supposedly accepting Wahl and Ammann, they didn’t care that Ammann and Wahl were relying on their rejected GRL companion paper. Although I notice that they did not publish Wahl and Ammann until a replacement paper for the rejected GRL paper was accepted – conveniently also at Climatic Change. The editor, Schneider, is committed to the Stick and I doubt that the failure of Juckes to complete the peer review process would matter a speck to them.
#7 Steve M. OK, I see what you are saying here, I had missed the point that the manuscripts get published in CP after acceptance, I had thought that CPD was a bona fide journal. The situation is then more like published conference abstracts where people can say almost anything and there is minimum review but these abstracts are almost never allowed to be referenced or certainly strongly discouraged when writing ‘proper’ papers. The main problem then seems to be that Climate of the Past Discussion assigns a volume number and ‘page numbers’ to the manuscript that makes it look like a genuine published article. In any conventional journal it would not have appeared yet and would not be citable. Presumably, if it is eventually accepted and published in CP, then it appears with a different reference – a good way to apparently double your published output. Maybe this is why it does not appear in Web of Science because it is not a journal paper in that sense.
The onus is therefore on authors not to reference this paper and other journals to question if CPD papers are used to prop up a major conclusion. Another interesting can of worms opened by this experiment in open journal publishing and reviewing.
Thanks for the post, I was wondering about this when I noticed it on the Georgia Tech class syllabus.
RE: per #8: You are aware, of course, that papers do contain material that has not been peer-reviewed – personal communications are one example.
General comments for those who have not gone through the manuscript submission process: Sometimes authors prefer to address criticisms in their letters back to the editor when they resubmit (the “that reviewer doesn’t know what he’s talking about” defense ;)). Sometimes an editor will instruct the authors as to what criticisms would be most important to address.
These glimpses that we are allowed into the climate publishing process provide us lots and lots of caveats to find and raise about the results. Maybe some of that apparent natural tendency towards wariness and skepticism we see in some scientists results more from the publishing process than from practicing the science.
The Climate Science Canon.
Do you guys realize how all of these authoritative contests recapitulate
debates over the canons in religion and the arts.
Just an observation. Walks like a duck.. talks like a duck
re: #12
that is a fair point. it is common for references to include things like books, web-sites, manuals; so it may be that it is sufficient to provide a clear link to something. However, peer-reviewed is normally the standard, and I would anticipate that reviewers could refuse a link if it wasn’t peer-reviewed. I don’t know if this one slipped under the wire, or walked in in broad daylight.
per
#15. I notice that Jan Esper includes Juckes et al (which as noted above, failed to complete the CP peer-review process) in his list of publications (which otherwise are peer-reviewed publications): http://www.wsl.ch/staff/jan.esper/publications.html
Nanne Weber of KNMI includes it as one of her “Recent refereed papers” http://www.knmi.nl/~weber/papers.html .
However Eduardo Zorita does not include it in his list of publications.
Different journals have different review processes, and most scientists and funding agencies will take this into account. Some jounrals, such as Energy and Environment or Climate of the Past (Discussions), publish papers based on editorial assessment (see http://www.climate-of-the-past.net/review/index.html). As the title of the journal suggests, CPD is intended to promote discusion. Our paper has now been accepted for publication in CP (see http://www.cosis.net/link.php?ts=jeav&ms_id=5220&uid=38478).
Reviewing this matter now, I find that the CoP changed links for the two McIntyre comments which were never adequately addressed by the paper’s authors. Here is the first McIntyre comment:
Click to access cpd-2-S697-2006.pdf
Here is the second McIntyre comment:
Click to access cpd-2-S708-2006-print.pdf
The editor(s) and the paper’s authors displayed an extraordinary inability to be responsive to detailed criticism, which would be considered a scandal in more intellectually respectable fields.
#17: Congratulations. The readers of this blog should check the editor Hugues Goosse’s ‘Follow up of the Interactive Discussion’.
Note the date:
Received and published: 21 September 2007
As Hans observed, Mann co-author, Hugo Goosse, who is editor of the Juckes submission, said the following in his capacity as editor of Juckes article on Sep 21, 2007 presumably in response to this post at CA:
CP policies are stated here. They state that:
Willis Eschenbach, Mark Rostron and I all submitted detailed comments to CPD, which were responded to in a cursory and flippant way by Juckes. See here.
In his editorial response at the time, Mann co-author Goosse did not refer to ANY of the detailed comments by Eschenbach, Rostron or myself, but only to the less detailed comments of Anonymous Refereee #1. I don’t know whether this is regular practice at CPD, but it seemed inconsistent with his responsiblities as an editor to me. BTW at the time, I had been asked to act as a referee of a CPD submission (by Bürger) so my views were presumably recognized by CPD. I had met with a CP editor in Dec 2006 at AGU, who had asked me to referee the Bürger paper.
CP policies say: “In general the revised manuscript should be submitted not later than 4 to 8 weeks after the end of the open discussion. ” In this case, this turned into 6 months, but I presume that that happens from time to time.
At this point, CP turned to traditional closed reviewing (which is contemplated as a possibility):
Goosse says above in his notes:
This presumably refers to my points. The probability of Juckes presenting arguments “in a balanced way” seems highly unlikely to me, but I guess that we shall see.
Reads like acceptance after reference 😀