[JohnA: For reasons that I don't understand, posting comments became nearly impossible for some people for a short while. This comment from Willis I thought should be captured for future reference so I made it into a post.]
Willis Eschenbach writes:
Well, on the 4th of this month I posted a couple of questions on the Climate of the Past discussion site. After 10 days (on the 15th), I got tired of waiting for an answer, and I posted the following:
I had made the foolish assumption that this discussion board was an opportunity to discuss the paper with the authors. However, they seem to be curiously unwilling to respond to my questions. If they are not willing to answer questions … what is the point of having a discussion forum at all?w.
In response, today (the 21st) I got a very courteous answer from Hugues Goosse, the Editor in Chief. He wrote:
Dear Willis Eschenbach,
I contact you as the editor in charge of the paper submitted to Climate of the Past Discussion, entitled “Millennial temperature reconstruction intercomparison and evaluation” (Juckes et al.) which is currently in the Interactive Discussion phase. Short Comment (SC) such as the one you have posted on the Interactive Discussion site of Climate of the past (http://www.cosis.net/members/journals/df/article.php?a_id=4661 ) are fully part of the Open Discussion Phase and I clearly understand you wish to get a quick answer to this comment. On the other hand, according to Climate of the Past Discussion rules, a quick answer of the authors is not required. An answer to all the relevant short comments could occur at any time during the Open Discussion Phase (8 weeks) or the Final Response Phase (4 weeks, expandable if required by the authors). I have personally contacted the authors to ask them if it would be possible to get a response sooner than the limit imposed by the rules but this will remain their choice.
If you have additional questions about to Climate of the Past rules, do not hesitate to contact me or one of the chief editors of the journal.
I have answered him as follows:
Dear Hugues Goosse:
Thank you kindly for taking the time to respond to my posting at Climate of the Past. It would be helpful if you could post these rules prominently in the discussion area, so that others will not be misled into thinking that there might be a free and frank interchange on your web site about the ideas put forth in the paper.
In addition, it would be even more helpful if the rules were changed so that a more interactive discussion could take place. One of the benefits of the internet is that ideas can be exchanged and topics discussed in short order. I am quite fearful that by the time the authors get around to answering the questions, that the time frame will be elapsed and the discussion will be closed.
A number of questions have been asked at Climate Audit about this report. Dr. Juckes has graciously replied to some of these questions, but in the main they remain unanswered. As these questions strongly impact the scientific value of the document, and they have not been answered in either forum, the quality of the work is very much in question.
As Editor in Charge, it is your responsibility, not the authors, to see that these questions are answered. Although the authors may choose to wait until the end of the period in order to forestall serious discussion of the paper, it is your responsibility to choose whether or not to print it. I encourage you to go to ClimateAudit, where there are a number of threads regarding this paper, to acquaint yourself with some of the difficulties with their work.
Relevant pages there are:
As you can see by the number of threads about the paper, a number of different, very troubling problem areas have been identified which bring into question the scientific soundness of the paper. These problems include, among others, misidentification of data, one data series being used twice under different names, using older data when newer data is available without justification, not following stated a priori rules for proxy selection, having a calibration period without a verification period, opportunistic flipping of principal components, and lack of adequate testing for robustness. They also neglect to justify their inclusion of foxtail tree ring width proxies, in total disregard of the explicit recommendation of the NAS Panel. In a paper which claims to examine the field of historical temperature proxies in general, this is a staggering omission.
In general, there have been no answers to these questions. I trust that you will take into account the demonstrated unwillingness of the authors to answer the hard questions about their work into account when you decide whether to print their document or not.
In addition, I would strongly suggest that you require the authors to delete the comments about the availability of code and data from Steve McIntyre, or you might find yourself in a very ugly legal dispute. Several of the co-authors of the paper have made it an unfortunate habit to refuse to provide data and methodology to other scientists. To claim that Steve McIntyre is guilty of this (which is absolutely not true), while ignoring the proven misdeeds of some of the co-authors of the paper in this regard, will open both you and the authors to the possibility of a civil suit for libel. I would be extremely cautious in this regard, were I in your shoes.
Finally, I appreciate your taking the time to inform me of the laxity of your rules regarding the author’s responses to questions, it makes the situation clear. I was looking for interaction, and it appears that I will be lucky to get action … but I will take whatever I can get.
Hugues Goosse has replied as follows:
Dear Willis Eschenbach,
The rules of Climate of the Past are clearly displayed at: http://www.copernicus.org/EGU/cp/interactive.html
There is access to these rules from the left menu when in the discussion area, so anyone could consult them easily I guess. In contrast to the majority of scientific journals, for Climate of the Past everyone has access to information on the peer-review process and any member of the scientific community could publish Short Comments, as you did. Compared to the classical review process, the goal is to foster scientific discussion and to enhance the quality control of the paper. However, the Short Comments should be of substantial nature. Answering those comments, as well as the ones of the Anonymous Referees, could require a large amount of work: to perform additional analyses or experiments, to read scientific papers… Consequently, It is not possible to oblige the authors to have a quick answer to all the comments. We could thus not have rapid blog-style conversations (but, as you know, there is several blogs where this is possible). On the other hand, at the end of the process, the authors must have answered adequately all the relevant comments. This is the role of the editors to check that it is actually the case. If some problems are remaining, the editors should require minor or major changes before publication or should consider that the work could not be published because substantial problems have been raised.
I hope that this brief mail has helped you to have a clearer idea of the philosophy of the public discussion of Climate of the Past.
My conclusion from this is that we should collect all of the unanswered questions and inconsistencies in the paper, polish them up, and put them on the Climate of the Past discussion site. That way, they will be part of the official peer review process.
My best to all, and my particular thanks to Hugues Goosse.