Is Juckes et al 2006 Peer Reviewed?

As readers of this blog know, Juckes et al submitted a paper for online review at Climate of the Past Discussions. See here for discussion. There were many unsatisfactory and even distasteful aspects to this paper. I submitted a detailed online review, as did Willis Eschenbach and another CA reader. I spent time rebutting a variety of unsupportable allegations about our paper. I did so in the belief that the online review process at CPD was a bona fide process. It appears that this belief was mistaken.

The responsible editor, H. Goosse, was a serial coauthor with Michael Mann and not particularly well-disposed towards the MM criticisms of MBH. Although I was an invited reviewer of B’rger’s re-submission, Goosse made no reference to either my review or to Willis’ review in his comments to Juckes et al. However, he did indirectly call for Juckes et al to “strongly reduce” their section purporting to criticize us and only “briefly” mention this controversy:

One exception is section 3 “critic of the IPCC2001 consensus on millennial temperatures”. This part is devoted to a very specific topic, difficult to follow for readers who are not familiar with previous work and to my point of view is not clearly connected to the other parts of the manuscript even in the revised version. This section is already long compared to the other ones of the manuscript, although some parts would require some more detailed information. I consider thus, at this stage, that this discussion should be much clearer if this was let that to another paper or note specifically devoted to this subject. In agreement with the Referee, I would thus recommend that the authors strongly reduce this section and briefly mention the controversy about the “IPCC2001 consensus” in section 2.

So while Goosse was undoubtedly not inclined to do us any favors, he clearly did not accept the Juckes submission. B’rger went to a considerable effort to re-submit a CPD submission and I guess that most of us assumed that Juckes et al would re-submit, just as B’rger had to re-submit.

However, we’ve seen no re-submissions. However, if you look at the references for Ammann and Wahl 2007, you will see:

Juckes MN, Allen MR, Briffa KR, Esper J, Hegerl GC, Moberg A, Osborn TJ, Weber SL, Zorita E (2006) Millennial temperature reconstruction intercomparison and evaluation. Clim Past Discuss 2:1001’1049

So Juckes et al don’t seem to have bothered going to the trouble of re-writing to meet referee comments. But it’s still cited in a Climatic Change article as though it was peer reviewed. We’ve seen examples in the Ammann/Mann corpus of academic check kiting. Surely this is a case of review avoidance if not actual review evasion (borrowing the terminology from tax law.)

And surely this is damaging to the reputation of Climate of the Past and should be protested by its editors. The CP experiment was an experiment in online and open peer review. CP Discussion editors had minimal requirements for posting online, presumably on the basis that online review comments would be taken seriously by authors. Ammann, Wahl, Juckes, Allen, Esper, Hegerl, Moberg, Osborn, Weber and Zorita have demonstrated a total disregard for the CP process by citing (and allowing their article to be cited) even though CP editors had asked that changes be made. And now Climatic Change has acquiesced in this continuing degradation of the currency by permitting Juckes et al 2006 to be cited as though it were a peer reviewed article.

Time for Climate of the Past editors to speak up.


23 Comments

  1. Posted Sep 14, 2007 at 2:39 PM | Permalink

    One of the major advantages of submitting a paper to Climate of the Past is that it is almost instantly published, and therefore citable, in Climate of the Past Discussion. By minimising the time between submission and publication the risk of being scooped is averted.
    Anybody citing papers in CPD should be aware of their status.

  2. Posted Sep 14, 2007 at 2:54 PM | Permalink

    AW07:

    The problem, however, will be more accentuated in situations when pre-calibration climates deviate significantly from the calibration range (von Storch et al. 2004, 2006; cf. Juckes et al. 2006 and Wahl et al. 2006).

    The problem is that some authors are too lazy to read relevant literature.

    Just like in WA07:

    It should be noted that in this experiment, the MBH step of scaling the proxy-reconstructed (i.e. fitted) instrumental PCs—so that those estimated for the calibration period have the same variance as the actual instrumental PCs—was not used, consistent with the method of MM03 (MM Supplemental Information).

    Sounds like MM03 made the mistake.. Sheesh.

  3. SteveSadlov
    Posted Sep 14, 2007 at 2:57 PM | Permalink

    Nothing surprises me any more. Either things are really going downhill, or, I am simply getting old, cynical and bitter (with ample scar tissue) …. LOL!

  4. IL
    Posted Sep 14, 2007 at 3:10 PM | Permalink

    I haven’t seen the full details of the editor’s recommendation but I haven’t seen anything to suggest that the paper was rejected and asked to be resubmitted. The editor has asked for revisions. If the author revises the manuscript, either accepting the criticisms or rebutting them then it is perfectly normal in most journals for an editor to make a decision as to whether that revision is acceptable or not. Its true that if the author disagrees with the reviewer then its highly likely to be sent for further review, either to the original reviewer or a new reviewer – and in this on-line experiment reviewing, probably as a new submission for reviewers to comment upon. However, the editor makes the decision and can decide whether or not the revision is acceptable.
    Clearly as it is published, the editor accepted the revised version.

    The crunch question is, do you feel they have ignored your criticisms and not revised the paper – is the paper still factually incorrect? If so then your only real recourse is to write a comment or a rebuttal.

    Oddly enough, when I searched for this paper on Web of Science (the leading method of searching journal papers), it did not turn up! So perhaps Climate of the Past Discussions does not get referenced in this database.

  5. Posted Sep 14, 2007 at 3:12 PM | Permalink

    I like the “Hey, It’s Climate Science!” tag myself.

  6. bernie
    Posted Sep 14, 2007 at 3:20 PM | Permalink

    Steve:
    There is an interesting column in the Wall Street Journal today about the efficacy, or lack thereof, of the peer review porcess in the medical research literature. Many of the points raised generalized to another applied science discipline.

  7. Steve McIntyre
    Posted Sep 14, 2007 at 3:47 PM | Permalink

    It’s not like I think that “peer review” as carried out in climate science is any imprimatur of correctness. My point here is that Juckes et al did not complete a peer review process as defined by CP. Here’s what they say:

    In the first stage, papers that pass a rapid access-review by one of the editors are immediately published on the Climate of the Past Discussions (CPD) website. They are then subject to Interactive Public Discussion, during which the referees’ comments (anonymous or attributed), additional short comments by other members of the scientific community (attributed) and the authors’ replies are also published in CPD. In the second stage, the peer-review process is completed and, if accepted, the final revised papers are published in CP.

    The Juckes paper only went to the CPD stage and there is no notice of it going through the “second stage” in which the “peer review process is completed”. All its undergone is a “rapid access-review by one of the editors”. Should be good enough for IPCC.

  8. per
    Posted Sep 14, 2007 at 4:28 PM | Permalink

    there is a guide to the procedure used at:

    http://www.climate-of-the-past.net/

    you can see the first stage, and discussion paper, online at:

    http://www.clim-past-discuss.net/2/issue5.html

    however, i think it is clear from this that the paper has only been into the first stage of the discussion process, and that a final, peer-approved version has not been published. The paper as is has the caveat of numerous strong referee criticisms (plus the editors) hanging over it.

    I do not see this is a problem for CPD, or their editors, because they are completely transparent about what the process is, and where this manuscript is in the process.

    It may be a problem for the journal that published ammann and wahl 2007, since they are relying upon a reference which has received severe criticism in peer-review, and has demonstrably not yet passed that process.

    per

  9. Steve McIntyre
    Posted Sep 14, 2007 at 8:51 PM | Permalink

    The journal is Climatic Change, whose review process for Wahl and Ammann is already very tainted by their apparent commitment to get Ammann and Wahl in print for IPCC. In supposedly accepting Wahl and Ammann, they didn’t care that Ammann and Wahl were relying on their rejected GRL companion paper. Although I notice that they did not publish Wahl and Ammann until a replacement paper for the rejected GRL paper was accepted – conveniently also at Climatic Change. The editor, Schneider, is committed to the Stick and I doubt that the failure of Juckes to complete the peer review process would matter a speck to them.

  10. IL
    Posted Sep 15, 2007 at 1:06 AM | Permalink

    #7 Steve M. OK, I see what you are saying here, I had missed the point that the manuscripts get published in CP after acceptance, I had thought that CPD was a bona fide journal. The situation is then more like published conference abstracts where people can say almost anything and there is minimum review but these abstracts are almost never allowed to be referenced or certainly strongly discouraged when writing ‘proper’ papers. The main problem then seems to be that Climate of the Past Discussion assigns a volume number and ‘page numbers’ to the manuscript that makes it look like a genuine published article. In any conventional journal it would not have appeared yet and would not be citable. Presumably, if it is eventually accepted and published in CP, then it appears with a different reference – a good way to apparently double your published output. Maybe this is why it does not appear in Web of Science because it is not a journal paper in that sense.
    The onus is therefore on authors not to reference this paper and other journals to question if CPD papers are used to prop up a major conclusion. Another interesting can of worms opened by this experiment in open journal publishing and reviewing.

  11. cytochrome sea
    Posted Sep 15, 2007 at 2:51 AM | Permalink

    Thanks for the post, I was wondering about this when I noticed it on the Georgia Tech class syllabus.

  12. Deech56
    Posted Sep 15, 2007 at 8:10 AM | Permalink

    RE: per #8: You are aware, of course, that papers do contain material that has not been peer-reviewed – personal communications are one example.

    General comments for those who have not gone through the manuscript submission process: Sometimes authors prefer to address criticisms in their letters back to the editor when they resubmit (the “that reviewer doesn’t know what he’s talking about” defense ;)). Sometimes an editor will instruct the authors as to what criticisms would be most important to address.

  13. Kenneth Fritsch
    Posted Sep 15, 2007 at 3:01 PM | Permalink

    These glimpses that we are allowed into the climate publishing process provide us lots and lots of caveats to find and raise about the results. Maybe some of that apparent natural tendency towards wariness and skepticism we see in some scientists results more from the publishing process than from practicing the science.

  14. steven mosher
    Posted Sep 15, 2007 at 3:20 PM | Permalink

    The Climate Science Canon.

    Do you guys realize how all of these authoritative contests recapitulate
    debates over the canons in religion and the arts.

    Just an observation. Walks like a duck.. talks like a duck

  15. per
    Posted Sep 16, 2007 at 8:21 AM | Permalink

    re: #12
    that is a fair point. it is common for references to include things like books, web-sites, manuals; so it may be that it is sufficient to provide a clear link to something. However, peer-reviewed is normally the standard, and I would anticipate that reviewers could refuse a link if it wasn’t peer-reviewed. I don’t know if this one slipped under the wire, or walked in in broad daylight.

    per

  16. Steve McIntyre
    Posted Sep 16, 2007 at 8:30 AM | Permalink

    #15. I notice that Jan Esper includes Juckes et al (which as noted above, failed to complete the CP peer-review process) in his list of publications (which otherwise are peer-reviewed publications): http://www.wsl.ch/staff/jan.esper/publications.html

    Nanne Weber of KNMI includes it as one of her “Recent refereed papers” http://www.knmi.nl/~weber/papers.html .

    However Eduardo Zorita does not include it in his list of publications.

  17. Posted Sep 24, 2007 at 2:58 AM | Permalink

    Different journals have different review processes, and most scientists and funding agencies will take this into account. Some jounrals, such as Energy and Environment or Climate of the Past (Discussions), publish papers based on editorial assessment (see http://www.climate-of-the-past.net/review/index.html). As the title of the journal suggests, CPD is intended to promote discusion. Our paper has now been accepted for publication in CP (see http://www.cosis.net/link.php?ts=jeav&ms_id=5220&uid=38478).

  18. Jean S
    Posted Sep 24, 2007 at 3:31 AM | Permalink

    #17: Congratulations. The readers of this blog should check the editor Hugues Goosse’s ‘Follow up of the Interactive Discussion’.

  19. Hans Erren
    Posted Sep 24, 2007 at 4:15 AM | Permalink

    Note the date:
    Received and published: 21 September 2007

  20. Steve McIntyre
    Posted Sep 24, 2007 at 6:48 AM | Permalink

    As Hans observed, Mann co-author, Hugo Goosse, who is editor of the Juckes submission, said the following in his capacity as editor of Juckes article on Sep 21, 2007 presumably in response to this post at CA:

    At the end of the discussion phase, some major revisions were required in the manuscript of Juckes et al. (2007) and an additional evaluation of the revised manuscript submitted to Climate of the Past was thus needed. As this second step of the review process is not public, I considered useful to provide here some additional information. The version submitted in April 2007 was send to 3 additional reviewers who make comments and suggestions. The authors submitted in July 2007 a revised version taking into account the majority of the reviewers remarks. A few points were however remaining. Some of them have been modified in the version submitted in September 2007. On the other hand, the authors disagree with one reviewer on some points for which no clear consensus could be gained from published literature. The arguments of the authors appear reasonable from our present knowledge of the field and are presented in a balanced way. As a consequence, I decided to accept the paper for publication in Climate of the Past.

    CP policies are stated here. They state that:

    Before submission of a revised manuscript for publication in CP, the authors are supposed to have answered the Referee Comments and relevant Short Comments cumulatively or individually.

    Willis Eschenbach, Mark Rostron and I all submitted detailed comments to CPD, which were responded to in a cursory and flippant way by Juckes. See here.

    In his editorial response at the time, Mann co-author Goosse did not refer to ANY of the detailed comments by Eschenbach, Rostron or myself, but only to the less detailed comments of Anonymous Refereee #1. I don’t know whether this is regular practice at CPD, but it seemed inconsistent with his responsiblities as an editor to me. BTW at the time, I had been asked to act as a referee of a CPD submission (by Bürger) so my views were presumably recognized by CPD. I had met with a CP editor in Dec 2006 at AGU, who had asked me to referee the Bürger paper.

    CP policies say: “In general the revised manuscript should be submitted not later than 4 to 8 weeks after the end of the open discussion. ” In this case, this turned into 6 months, but I presume that that happens from time to time.

    At this point, CP turned to traditional closed reviewing (which is contemplated as a possibility):

    In view of the access peer-review and Interactive Public Discussion, the editor either directly accepts/rejects the revised manuscript for publication in CP or consults referees in the same way as during the completion of a traditional peer-review process.

    Goosse says above in his notes:

    On the other hand, the authors disagree with one reviewer on some points for which no clear consensus could be gained from published literature. The arguments of the authors appear reasonable from our present knowledge of the field and are presented in a balanced way. As a consequence, I decided to accept the paper for publication in Climate of the Past.

    This presumably refers to my points. The probability of Juckes presenting arguments “in a balanced way” seems highly unlikely to me, but I guess that we shall see.

  21. Hans Erren
    Posted Sep 24, 2007 at 7:02 AM | Permalink

    Reads like acceptance after reference :-D

Follow

Get every new post delivered to your Inbox.

Join 3,192 other followers

%d bloggers like this: