There is an interesting discussion at PBS on peer review, in which Donald Kennedy, editor of Science, defended their existing "rigorous" processes, but re-iterated:
the journal has to trust its reviewers; it has to trust the source. It can’t go in and demand the data books.
If I criticize Science’s due diligence procedures, I don’t think that anyone will accuse me of piling on after the Hwang affair. Here’s something that I wrote last summer:
the underlying issue is that Science does not seem to either have policies that require authors to archive data or administration practices that ensure that their policies are applied. Since NSF then relies ( a reliance which seems to me to be an abdication of their own separate responsibilities) on journals like Science, with either inadequate policy or inadequate administration, there’s a knock-on effect.
Here are some comments on the PBS interview and some re-cap of past commentary.
The PBS discussion is posted up here.
The discussion of replication and archiving policy has been a theme at this site for some time. An index is on the right frame under "Disclosure and Due Diligence". I draw your attention particularly to the post Replication Policy. That post did not draw as much response as some other posts. As a purported rebuttal, Dano asked:
How many of the 14 Reports in the latest ed of Science have sufficient archived material to permit replication? And the latest ed of Nature has 20 methodable articles/letters – how many of these have sufficient archived material to permit replication?
The answer is probably none of them. I’ve discussed Science in particular in a number of posts, including the following Science Editorial, Science Editorial #2, Letter re Esper, Kilimanjaro Data and others. To date, Science’s performance in ensuring compliance with their own data policies is abysmal.
On to PBS. Scadden pointed out clearly that there are important disincentives for scientists to commit fraud. Scadden:
In some ways the example of Dr. Hwang provides a clear example of why there’s such a tremendous disincentive for this kind of activity to happen. This gentleman’s career and probably his life is ruined.
I agree with this and it’s why outright fraud doesn’t happen a lot. However, it’s probably easier to understand the nuances and degrees of fraud from a business background. There are many degrees of dishonesty between Bre-X on one extreme and the many forms of subtler forms of dishonesty that keep securities commissions in business. At Bre-X, there was salting of assays on an industrialized scale. De Guzman’s "career" was ruined and, if he was thrown from the airplane or if he jumped from the airplane, his life was "ruined". (There are interesting unresolved mysteries here.) So there were disincentives here as well, but the fraud still happened. The Ontario Securities Commission did not react to Bre-X by simply saying that this was a "bad apple", but re-examined processes of due diligence and disclosure. Whether their cures are relevant to the disease is a different issue, but they used the occasion to re-examine what they were doing.
Scadden pointed out that the "process" did eventually work:
And fortunately the process does eventually work. In this case, it unfortunately worked after this had been touted as a major breakthrough.
OK, but surely one can inquire as to whether there are ways of making the "process" more efficient. If you re-read McCullough and Vinod, they discuss replication from an economist’s point of view: can you make the replication process more efficient? One of their answers is the one that I’ve consistently advocated: complete archiving of data as used, accurate data citation and archiving of source code. That a process "eventually" works is not good enough – surely we are entitled to ask whether the process can be made more effective and more efficient. Scadden then points to one of the difficulties for peer reviewers:
And unfortunately you don’t have the opportunity to really see the primary information other than what’s given to you by the scientists.
OK, why not? Is lack of access to primary information impossible or simply a convention? Let’s turn to Donald Kennedy, editor of Science. Kennedy:
It’s not our happiest day at Science, Ray. What we’ve done, of course, is to review the peer review process as we engaged in it with the Hwang papers. It was a pretty rigorous process.
I hate the use of the word "rigorous" by scientists. Too often it’s nothing more than arm-waving throught the hard part (e.g. Mann’s use of the term in respect to the RE statistic.) The interviewers attempted to get a more operational description of the actual peer review process at Science from Kennedy, but without much success. Kennedy did distinguish peer review from an audit as follows:
What we can’t do is ask our peer reviewers to go into the laboratories of the submitting authors and demand their lab notebooks. Were we to do that, we would create a huge administrative cost, and we would in some sense dishonor and rob the entire scientific enterprise of the integrity that 99.9 percent of it has….
I am trying to explain to everybody that, as Dr. Scadden said, it all depends on trust at the end, and the journal has to trust its reviewers; it has to trust the source. It can’t go in and demand the data books. What it can do is to make sure that its process is as good as it can be.
OK, why not? Let’s apply this to the paleoclimate studies that I’ve considered carefully. Here Kennedy’s comments don’t make any sense at all. Doesn’t Science already have theoretical policies requiring authors to archive their data? The equivalent of "data books" or "lab notebooks" for multiproxy paleoclimate studies can be easily archived. Part of the problem at Science is that it doesn’t administer its existing policies effectively. I’ve reported from time to time on my efforts to get Science to archive data from various paleoclimate authors – Thompson, Esper for example. This has been going on for months and virtually nothing has been accomplished. (My experience with Nature in respect to MBH is also instructive – they complied partially with some requests, but were totally unresponsive on some completely reasonable data requests. It might be interesting to post up my correspondence file, now that this is in the news).
So why can’t Science demand that paleoclimate authors archive these? (I don’t know whether there is an equivalent archive that would be applicable to stem cell research, but I’ll bet that there is something that could have been readily archived and which would have been useful to expert readers, which Science did not require the authors to archive.)
Kennedy’s excuse about "huge administrative cost" is simply piffle. There’s a distinction between the journal ensuring that the authors have provided a complete replication archive and between the journal reviewers actually carrying out the replication. All scientists seem to agree that "replication" is part of the scientific process. To the extent that the costs of replication are an "administration cost" – huge or otherwise – they are therefore a cost that eventually has to be borne by someone. In the stem cell research area, according to Scadden, other labs rush out to try to replicate results using the same methodology. (While this may be the case in medical research, may I observe in connection with paleoclimate that there’s no evidence that anyone had attempted to replicate MBH methodology until we attempted to do so.)
By Science failing to require an archive at the time of publication, they vastly increase the "administrative cost" for other parties, as McCullough has pointed out. In fact, by not requiring an adequate archive, the journals may create almost impossible barriers to exact replication. The obstacles to replication of MBH (or Esper, or Briffa) require virtual litigation to overcome, as my experiences have provided an object lesson and are undoubtedly one of the reasons why noone tried to replicate MBH until we tried.
Is this form of archiving feasible? The process of requiring the authors to confirm that they had archived the data and source code/methodology and checking that the archives were not empty would have a trivial cost. This form of archiving is already practiced by econometrics journals at negligible cost (see my post on the Journal of Political Economy or an earlier post on the American Economic Review).
One last point about MBH, if I may. I’m not sure that I entirely agree with Scadden’s views expressed below, but he connected reliance on results using accepted methods to publication:
But science is really based on trust like most other human endeavors and so when that information is provided as long as it has the credibility of coming from a person of some reputation done with methods that are accepted methods and where the details are sufficiently clear that you feel this information makes sense and has the validity you anticipate for a series of studies that give multiple perspectives on the issue, you then regard it as something that is worthy of publication.
McCullough (as previously posted here) observed of MBH that the principal contribution that they claimed was a "new statistical approach", while archly observing their reluctance to furnish the details of their algorithm:
As expected, high visibility invites replication and tests of robustness. In a series of papers, McIntyre and McKitrick (2003, 2005a, 2005b) have chronicled their difficulties in obtaining the data and program code; the publishing journal, Nature, did not archive the data and code. After some delay, the authors provided the data (see Mann et al., 2004) but have declined, at least as of this writing, to furnish their statistical estimation programs despite their statement that the statistical method is the principal contribution of their article, specifically, to “…take a new statistical approach to reconstructing global patterns of annual temperature back to the beginning of the fifteenth century, based on calibration of multiproxy data networks by the dominant patterns of temperature variability in the instrumental record.” (Mann et al. 1998, p. 779).
Bradley credits Mann with originating new mathematical approaches that were crucial to identifying strong trends.
I simply point out the irony of Scadden’s applying "accepted methods" as a criterion for assessing results with MBH claim to use "new" methods and their inaccurate documentation of these methods, refusals to document the methods and the ultimate acquiescence of Nature in this performance. I’ll post up on this another day.
For now, I want to keep hitting on the point that, in my opinion, there is no acceptable reason why the franchise science journals, Science and Nature, – for paleoclimate articles at least – cannot insist on comprehensive archiving of data sources, data as used and source code. I don’t know how this would exactly translate into policy for medical research, such as Hwang’s, but I’m sure that the same philosphy applied to the altered circumstances would much improve the situation and no one has ever provided a reason not to do it.