Based on my limited experience, it seems to me that journal peer reviewing faces an interesting challenge with the increased use of Supplementary Information (and I absolutely endorse detailed SI and obviously encourage even more detailed SI). In a very non-random of articles that I know inside-out (Team journal publications), my conclusion is that, in these cases, the journal reviewers didn’t even look at the SI or even verify that the SI actually exists. Whether or not this sort of flaccid reviewing extends to other journals or even to other climate science articles not involving the Team, I can’t say. But I think that the observations are well supported in respect to flaccid reviewing of Team articles.
It’s one thing if the SI is merely data. But combined with the use of SI to provide data, in some cases, authors use SI to derive results that are applied in the main article. In such cases, the reviewers should surely be obliged to examine the SI as an integral part of the article.
I’ll discuss two examples – the MBH Corrigendum and the recent Wahl and Ammann 2007/Ammann and Wahl 2007.
We got some insight into Nature’s practices with SI because we had a submission to Nature under review concurrent with the MBH Corrigendum. One of the reviewers of our submission said in respect of a Mann reply point that this methodological point was not in the original article or original SI and that it should have been included in the Corrigendum SI. Both myself and Marcel Crok pressed Nature and eventually found out that not only was the Corrigendum SI not peer reviewed, but the Corrigendum itself was not peer reviewed (it was handled by editors); the Corrigendum SI was not even examined by the editors. Examination of SI did not, in general, appear to be included in the peer review process. I must say that it seemed odd to me that Nature did not peer review the Corrigendum. I would have thought that, if they felt that a Corrigendum was warranted, then there was at least as great an obligation to peer review the Corrigendum as the original article.
In addition to providing data, the Corrigendum SI also included methodological information, which, unfortunately, remained very unsatisfactory. Given the problems with MBH replication that had already then been demonstrated, it’s hard not to think that some sort of peer review wouldn’t have improved the MBH Corrigendum SI, whose methodological descriptions remained evasive and unreplicable.
A second example arises with Ammann and Wahl/Wahl and Ammann. Both articles make extensive references to Supplementary Information. Wahl and Ammann 2007 makes 8 references to its Supplementary Information, while Ammann and Wahl 2007 make 6 references. The SI references are not limited to the provision of supporting data. In some cases, they refer to figures supposedly illustrating results in the text; in other cases, they refer to statistics and tables in the SI.
Key discussion in Ammann and Wahl 2007 is exported to the SI. For example, they state:
Only the MBH 1600-network is significant at a slightly lower level (89%), and the much discussed 1400- and 1450-networks are significant at the 99% and 96% levels, respectively. (See electronic supplement for further discussion and details, including code and tables with established thresholds for a variety of calibration/verification RE ratios and for the other WA scenarios examined.)
The effect of using “princomp” without specifying that the calculation be performed on the correlation matrix (an alternate argument of “princomp”) forces the routine to extract eigenvectors and PCs on the variance-covariance matrix of the unstandardized proxy data, which by its nature will capture information in the first one or two eigenvectors/PCs that is primarily related to the absolute magnitude of the numerically largest-scaled variables in the data matrix (Ammann and Wahl, 2007, supplemental information).
The latter statement is a foolish assertion about tree ring networks that are already standardized to a mean of 1 (under standard tree ring chronology procedures) and which this have the same absolute magnitude. A pseudo-citation for this claim makes it seem more impressive but the pseudo-citation is not to something that itself has been peer reviewed, but to a non-peer reviewed SI.
But where is the SI? No URL is given in either publication, other than the following:
Additional information and illustrations beyond the present text are provided in an electronic supplement and on our WEB site (http://www.cgd.ucar.edu/ccr/ammann/millennium/MBH_reevaluation.html).
The website http://www.cgd.ucar.edu/ccr/ammann/millennium/MBH_reevaluation.html has not been updated in nearly 2 years and contains no mention of Ammann and Wahl 2007 and does currently not contain any of the promised information. This sentence specifically states that there is an “electronic supplement” additional to the UCAR website. The editor and then publisher of Climatic Change were approached about the location of the SI and neither of them knew anything about it. They said to contact the authors.
All of this strongly suggests to me that the peer reviewers did not have access to the SI – otherwise wouldn’t the editor and/or publisher have had access to it in order to supply it to the reviewers?
This says something about the quality of peer review for Team articles – none of the reviewers even bothered to ask for the SI that was cited in the articles or even bothered to determine whether the SI was in existence. I have no doubt that Ammann did up some sort of SI and that the SI will ultimately see the light of day – but shouldn’t the peer reviewers have considered the SI when they were reviewing the article?
BTW I contacted Ammann about this and another question . The other question was:
In Ammann and Wahl 2007, you state without a reference:
Standard practice in climatology uses the red-noise persistence of the target series (here hemispheric temperature) in the calibration period to establish a null-model threshold for reconstruction skill in the independent verification period, which is the methodology used by MBH in a Monte Carlo framework to establish a verification RE threshold of zero at the >99% significance level.
Can you please provide me with supporting references demonstrating the validity of this statement.
Ammann promptly gave a very surly response.
I must say that I find it disturbing that you don’t appear to acknowledge scientific arguments as such, rather you dismiss anything we say and reduce it to below “low level authority”. Under such circumstances, why would I even bother answering your questions, isn’t that just lost time?
I wrote a very measured response urging him to re-consider this refusal, to which Ammann did not initially reply. I then sent a follow-up, copying Nychka, and Ammann said this time that he would answer in a few days but he had other “pressing” business. That was a few days ago, so it will be interesting to see what his response is.