There is an interesting controversy at Nature and Science about peer review in the context of Hwang’s stem cell research (google for links.) I’m going to post a comment about this in light of my own experience with both. First, I want to post some information (courtesy of a reader here) about archiving policies at the Journal of Political Economy. For paleoclimate studies, there is absolutely nothing intrinsic to the subject which prevents the implementation of similar policies by Nature or Science as a “best practices” standard. In my view, the focus of invetigation by the two journals in respect to Hwang is not about “peer review” in the abstract, but whether the journal policies meet “best practices” standards.
The Journal of Political Economy has similar comprehensive policies requiring archiving of data as used and source code as a condition of publication. They state here:
It is the policy of the Journal of Political Economy to publish papers only if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication. Authors of accepted papers that contain empirical work, simulations, or experimental work must provide to the Journal, prior to publication, the data, programs, and other details of the computations sufficient to permit replication. These will be posted on the JPE Web site. The Editor should be notified at the time of submission if the data used in a paper are proprietary or if, for some other reason, the requirements above cannot be met. Details of this policy can be found at http://www.journals.uchicago.edu/JPE/datapolicy.html.
The latter link specifies, inter alia:
For econometric and simulation papers, the minimum requirement should include the data set(s) and programs used to run the final models, plus a description of how previous intermediate data sets and programs were employed to create the final data set(s). Authors are invited to submit these intermediate data files and programs as an option; if they are not provided, authors must fully cooperate with investigators seeking to conduct a replication who request them. The data files and programs can be provided in any format using any statistical package or software, but a Readme PDF file documenting the purpose and format of each file provided, and instructing a user on how replication can be conducted, should also be provided.
If the paper is accepted by the JPE, the appendices containing instructions, the computer programs, configuration files, or scripts used to run the experiment and/or analyze the data, and the raw data will normally be archived on the JPE Web site when the paper appears.
This is obviously not an unattainable or utopian goal for paleoclimate (or for IPCC), since it is a standard in effect for highly similar datasets. Here I emphasize the great formal similarity in size and computer programming between applied economics and paleoclimate: one is dealing with relatively small data sets; both are usually autocorrelated (although that’s not relevant to the archiving issues), versions of the data sets may differ so exact citation is important; it is impractical to exhaustively explained methodologies in words sufficiently to permit complete replication.