How can peer-review be accomplished without the reviewers looking at the original data?

…is a question we’d like to know the answer to. The answer is that peer review cannot bear the weight of the expectation of due diligence that people believe when they hear about it.

Peer review can catch obviously wrong statements that contradict known scientific principles. It cannot catch data fraud. That can only be done by people replicating and/or auditing the work.

One of the things you learn as you read this blog is that while you might have thought that you can tell a good paper from a bad one by just reading it, that turns out not to be the case.

Merely because a paper looks good, follows the rules for good papers and agrees with what you thought was the truth where you can check without difficulty does not mean that what’s new about the paper you’re reading is correct.

OTOH, I’m not sure that peer review is supposed to do so. That’s supposed to be the people who read the paper and respond.

Perhaps they should call it pier review since it’s so a dhoc.

Or am I completely misunderstanding the process (I am an electrical engineer)

Thanks
JK

Steve,

If they had learned anything from the Hwang affair, it’s that the data should be archived as a precondition of acceptance for publication, not as a request after the fact some time later by which time the authors are on their next project, their next funding meeting, their next academic sinecure. The authors have no particular compulsion to respond to requests for data after publication, unless and until the journals start taking their archival policies seriously.

Have they actively started to do this? No. The last thing we heard from Kennedy was “Scientific fraud happens and there’s nothing we can do to detect it”

If someone’s data is questionable, then their argument falls apart and their reputation suffers in the future “¢’‚¬? nothing more nor less.

I agree.

The problem is to get the data from people who:
1. lost it
2. missplaced it
3. are too busy to bother
4. do not want to give you their data.

This intransigence makes it very difficult to evaluate whether the data is questionable.

If I were Hanson, I would really lay down the line with these guys. I’d tell them that I have other things to do besides babysit the production of data. I’d say – here’s the list, here’s a deadline, if you don’t have done it done by then, we’re retracting the paper. We’re in a post-Hwang era; get with the program.

The one thing that the editors of Science and Nature have learned is that they have learned nothing from the Hwang woo Suk affair. Precisely nothing has changed.

“I find it hard to believe that scientists who write peer reviewed papers are so dissorganized that they can not find the data which they used in this decade.”

I must confess that in computer science departments I do not see especially good archival of data. It is something each person does on their honor, and there simply is no one interested in cracking down on people who do it poorly.

It is not Science 101, but it actually does all works out okay in the end. The ins and outs of it are rather complicated, and between-scientist politics is a big part of it, but in the end, computer scientists of today know a lot more than they did a decade or three ago.

Incidentally, here’s a better way (IMHO) to think about the situation. The goal is not to mark who is a Good Scientist and who is a Bad Scientist, and neither is the goal to keep some kind of score in a great big scientist competition. The goal is to find the truth. If someone’s data is questionable, then their argument falls apart and their reputation suffers in the future — nothing more nor less.