Nature’s Guide to Authors includes an excellent statistical checklist which authors are asked to comply with to "ensure statistical adequacy". I’ve reproduced the checklist below, bolding a couple of interesting criteria. Readers of this blog can readily imagine how this checklist would apply to MBH98 or, for that matter to Moberg et al .
One wonders sometimes if the left hand knows what the right hand is doing at the big science journals. Nature’s handling of statistics reminds me of Science’s handling of data archiving. In both cases, the policy is terrific, but neither journal seems to have any procedures for implementing the policy for paleoclimate articles. Maybe they are better on medical and biological topics.
As you see below, Nature has a policy requiring that "Any data transformations are clearly described and justified". Whatever else one may think of our criticism of Mann’s PC method, it remains unarguable that the PC methodology used in MBH98 was not "conventional" and that the data transformation was not "clearly described and justified." Obviously the editors and reviewers were unaware of this at the time of the original article. But what about at the time of the Corrigendum? At this time, Nature editors were clearly aware of the data transformation prior to the PC calculations. Let’s say that they, in good faith, felt that our own submission had not demonstrated that the data transformation "mattered" in terms of its ultimate effect. That does not excuse them not insisting on a proper description and justification of the data transformation in the Corrigendum.
One could go on and on. I think that I’ve pointed out the small size of the Moberg data set as well as the extraordinary non-normality of key data sets. It is inconceivable to me that Moberg et al. could have considered the Nature statistical checklist below and reported that they were in compliance with it. So one presumes that, at no point in the Nature editorial process, did they ever ask the authors to confirm that they had carried out the statistical checks listed in Nature’s policy or check to see that they had. Just imagine what a questionnaire on MBH98 would look like.
The nice thing about policy statements like this is that they give objective standards for evaluating articles like Moberg et al 2005 or even MBH98. I think that I’ll submit Nature’s checklist to the NAS panel.
Type and applicability of test used
· Comparisons of interest are clearly defined
· Name of tests applied are clearly stated
· All statistical methods identified unambiguously
· Justification for use of test is given
· Data meets all assumptions of tests applied (with particular attention paid to non-normal data sets or small sample sizes, which should be identified in the text as such)
· Adjustments made for multiple testing is explained
Details about the test
· n is reported at the start of the study and for each analysis thereafter
· Sample size calculation (or justification) is given
· Unit of analysis is given for all comparisons
· Alpha level is given for all statistical tests
· Tests are clearly identified as one or two-tailed
· Actual P values are given for primary analyses
Descriptive statistics summary
· n for each data set is clearly stated
· A clearly labeled measure of center (e.g. mean or median) is given
· A clearly label measure of variability (e.g. standard deviation or range) is given
· All numbers following a ± sign are identified as standard errors (s.e.m.) or standard deviations (s.d.)
· Any unusual or complex statistical methods are clearly defined and explained for Nature’s wide readership. (Authors are encouraged to use Supplementary Information for long explanations).
· Any data exclusions are stated and explained
· Any discrepancies in the value of n between analyses are clearly explained and justified
· Any method of treatment assignment (randomization etc) is explained and justified
· Any data transformations are clearly described and justified
Within individual graphs:
· Any distorted effect sizes (e.g. by truncation of y axis) are clearly labeled and justified Clear labelling
· Error bars are present on all graphs, where applicable.
· All error bars are clearly labeled
Many statistical analyses published in Nature are highly sophisticated and outside the scope of this checklist, particularly in the case of some studies in physical sciences disciplines. Authors and referees who have specific suggestions for additional entries to this list are encouraged to send them by email to firstname.lastname@example.org or email@example.com.Nature
will update this checklist at intervals in an effort to ensure that papers published are statistically robust.