http://www.scn.ucla.edu/Voodoo&TypeII.html

Any chance of some constructive feedback on a video I put together about the above study?

Regards

Justin

“There just aren’t the resources now to do really large, well-powered mouse studies”.

And there’s also a trend to limit the use of laboratory animals or skip some levels of animal testing in the experiments, because of all this “green” legislative, especially in the EU. This, of course, can lead to half-baked experiments with totally insufficient controls.

http://en.wikipedia.org/wiki/Rosenhan_experiment

Researchers claimed to hear voices and were admitted to institutions. When the phony patients wrote their daily research notes, genuine patients quickly caught on that the researchers were sane. Staff, on the other hand, interpreted the note making as compulsive behaviour, a further symptom of insanity. The only way some of them could get released was to admit that they were insane. Irony²!

There’s a parallel here, somewhere…

Little bit of poor control of variables, bit of chucking out data you don’t like, and the use of small group size; and gee whiz, you have 50 papers. Small group sizes are key, because you can get a big effect size by random chance

Sounds like the Team multiproxy literature with a bunch of studies with 6-15 “proxies”, all with bristlecones and/or Yamal, plus Tornetrask, ….

I saw something similar when I was in grad school. People use poor data sets, or limited experimental design. You point out their error – carefully, because you can’t make enemies out of faculty – and the respose is “It’s not optimal, but it’s all we’ve got in this field, so we need to use it to solve the problem.” That’s the good careerist answer, but of course the truth is that if you are unable to do it right, then you have to right to do it. Wrong is wrong, and if that means you have to find some other way to get published, so be it.

Sounds like the old quality management saw: “There’s not enough resources to do it right the first time, but there’s always enough to do it over again.” Poorly designed trials can keep a project alive for years.

It is probably germane to note that this paper, and the story, are not unique. This one has got a lot of press, perhaps due to the title, which is snappy.

However, this is a common problem. See:
Nature 454, 682-685 (2008)
Amyotroph Lateral Scler. 2008;9(1):4-15 Scott et al.

Scott et al followed up a mouse model used in over 50 publications, all of which purportedly showed statistically significant effects on prolonging lifespan in the mouse. Little bit of poor control of variables, bit of chucking out data you don’t like, and the use of small group size; and gee whiz, you have 50 papers. Small group sizes are key, because you can get a big effect size by random chance; you don’t publish the uninformative experiments but you do get a decent number of false positives. When you do the experiment properly, you cannot repeat the “positive” studies.

One of the interesting aspects is the Nature overview, which quotes one researcher “There just aren’t the resources now to do really large, well-powered mouse studies”. That’s interesting, ‘cos there was all the money spent on enabling 50 different publications (plus unpublished failed replications), plus the resulting clinical trials in humans, and that has all gone down the drain because the mouse studies are (according to scott’s study) not replicable. So 50 different publications is going to be ~~ $5 million, plus the clinical studies, which will easily double that.

there is also the rather invidious issue that many of these researchers who got these fantastic papers got follow-on grant money, prestige, etc.

per