Dr. Ioannidis, an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass, has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye. He writes in the WSJ as follows:
Most Science Studies Appear to Be Tainted By Sloppy Analysis
We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong. …
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. “There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims,” Dr. Ioannidis said. “A new claim about a research finding is more likely to be false than true..
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Sounds to me like he’d support checking for “miscalculation, poor study design or self-serving data analysis” in climate science as well.
Climate Audit 
58 Comments
Gee, I posted about this yesterday in the “Should NASA climate accountants adhere to GAAP?” thread (comment 125), but I guess everybody was too buy debating Boris to take notice…
Dr. Ioannidis says “A new claim about a research finding is more likely to be false than true.. ”
except, of course, his new claim about a research finding.
RE: #2
It does sound a little self serving, now that I think about it. Meta-Science degrees coming soon?
Reid,
I suppose you could audit his research and find out. Nah, it’s probably easier to just snipe at it on a blog.
One should be particularly wary of studies where the author originally refuses to release his data and methodology and then changes the data and methodology after being pressured to comply with the most rudimentary standards of scientific inquiry.
Re #4
I agree with Dr. Ioannidis. However, I found the irony of his claim humorous. Monty Python fodder.
With Steve’s permission I’ll post some other gems from the same article (can’t link it, it’s subscription):
\\”Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. “People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,” Dr. Ioannidis said.//
John Brignell has written a lot on the subject of weak
epidemiology,
http://www.numberwatch.co.uk/
He is fequently attacked by the usual cadre.
That clearly stands to his credit.
Reid,
Sorry, I misunderstood how your comment was meant. Sometimes I should just keep my big mouth shut.
So to speak…
Any good scientist knows that to study something that is variable, you have to hold the other variables that affect that variable constant. However, it seems that all to often studies are funded knowing in advance that they do not hold to this basic principle even when they could have. If the general person on the street can recognize a flawed study, why can’t scholars and journalists? This tells me that these studies are done this way on purpose and are NOT mistakes or the result of sloppiness. These studies are make-work for funding, profile, and politics.
Which is often way more fun, sort of a stress-relief exercise.
I think the way Dr. Ioannidis’ paper should be viewed is as confirmation that scientific claims should always be viewed with skepticism. The more outstanding the results, the more outstanding the evidence should be. Unfortunately, mainstream media, and the public at large, do not do so.
Mark
From the same article by Robert Lee Hotz (!):
This article is really a statement of the obvious. How many times, especially when medicine or biotechnology are involved, have we read a story that goes something like “A causes B to rise”, only to see another one six months later that says “A cause B to fall”, only to see another “A causes B to rise” story in another six months?
If you’re not skeptical of science in the news, you’re either not paying attention, or your one of those people born every minute.
theDuke (#13)
Replication is the key. Replication requires the release of data and code.
#14
As numerate skeptics I think tyou are correct. The value of the article is not that it does happen but the scope of the problem. It also serves to justify the mantra of this site: Show me the data, show me the code.
#15
Replication is certainly critical, but there also needs to be enforcement. Somehow the science community must make it more painful to publish erroneous results than to do research that produces little of importance.
Larry,
What people?
Now we know why some “climate scientists” do not want their work audited.
Case in point:
http://www.time.com/time/health/article/0,8599,1662484,00.html
Now, I don’t claim to know if they’re on to something or not in this article, but if the science on the relationship between fat and cancer isn’t “settled” by now, how on God’s green earth do they expect us to believe that the science of climate change is “settled”?
Mark T. says:
September 19th, 2007 at 11:21 am
but I guess everybody was too buy debating Boris to take notice..
Boris is the Muppet, Beaker.
On the internet, no one knows if you’re a Muppet.
Re#14
There were concurrent publications in different medical journals maybe 6 months ago about caffeine and a heart/pulmonary issue (can’t rememeber if it was heart attacks, heart disease, strokes, high blood pressure, or what – but it was the same isse for both article). One publication said caffeine reduced risk, while the other said it increase it. Those were to strange press releases to read abck-to-back.
“57.2% of all statistics are made up”
Steven Wright
comedian
A scientist will never show any kindness for a theory which he did not start himself. (Mark Tawin)
Bernie, #15: on the subject of replication from the same article:
Is debating now defined as posting pictures of Beaker? Or saying “Goerbal warming”?
Oh, how’s the comparison between CRN1/2 and CRN5 sites going?
27, not debating Boris, baiting Boris…
I agree with Larry, pretty much stating the obvious.
Re #27
Boris,
Speaking of results, any progress on your analysis of microsite thermal impacts of air conditioning units?
Cheers!
Earle
Larry in #14 about the article being rather self evident.
29, 32 – right. If the media had a story about how bad the media is, that’s a “man bites dog” story. If the a scientist writes about how bad science is, that’s a “dog bites man” story. I had thought that most people had already become jaded about both media reporting and scientific research.
Per an old study in Science in the 60s, 35% of published peer reviewed papers are never cited in the literature, 49% are cited only once, and just 1% are cited six or more times. Suggesting that most research is hardly looked at, let alone replicated or audited.
Link to study: http://www.garfield.library.upenn.edu/papers/pricenetworks1965.pdf
Oh, hows the comparison between CRN1/2 and CRN5 sites going?
Boris, how’s the investigation of continuity of GISS data manipulations of monthly/daily USHCN station data going? Corelations with precipitation? Posted any data or analysis lately? Didn’t think so.
Re: #27
Or saying Goerbal warming?”
It’s actually Gorebal Warning. Too subtle for some, I suppose.
Re: #27
Posting images of Beaker?
Actually only a link. I’m not sure how to post the image or if hot linking here is allowed. It’s still funny to me though.
RE 28. LARRY……………………………………………..
Stop feeding the Dung beetle
More from Robert Lee Hotz:
Did you know that Climate Change causes the North Atlantic to increase in salinity?
http://powerandcontrol.blogspot.com/2007/09/global-warming-causes.html
It also causes the North Atlantic salinity to decline
http://powerandcontrol.blogspot.com/2007/09/global-warming-causes.html
Yep. Same url for both (I’m a collector).
M. Simon, #40: We’re experiencing the perfect propaganda storm: publishers and editors want material that examines global warming because it sells, and the “scientific” (I use the term loosely) AGW crowd is only too happy to provide it.
Re #11: Ah, yes, the joys of variables. Michael Crichton spoke often about the Drake equation N=N*fp ne fl fi fc fL, which, according to him was a useless formula. Seems like the AGW crowd is using a variation of the Drake equation to come up with their outlandish projections:
http://www.michaelcrichton.net/speech-ourenvironmentalfuture.html
#42 : Ah, well, I would in first place enjoy a consistent notation. If my guess is correct that the terms in the equation are all to be multiplied, why is there an asterisk in one place and spaces in the rest ? Talking about science….
I thought that was supposed to be de-baiting Boris…you know…hook, sinker, Waldo.
Goerbels, Goebbels, Gorebells, Gorepals warning…there’s a difference?
Did you know the consensus scientific opinion has found that eating meat causes Global Warming? Will eating meat be banned at the next meeting of the IPPC?
ABC News: Meat-Eaters Aiding Global Warming?
abcnews.go.com/Technology/story?id=1856817&page=1
GoVeg.com // Meat and the Environment // Pollution // Fight Global …
… would if you just gave up eating meat and other animal products. … individuals to avert global warming is to stop eating meat, eggs, and dairy products. …
http://www.goveg.com/environment-globalwarming.asp – 29k – Cached
YouTube – Al Gore ignores eating meat global warming
Glenn Beck on how Al gore is a hypocrite. This shows how rotten Gore is. Gore is all about grabbing taxes. Do not eat meat if you want to stop global warming…
http://www.youtube.com/?v=VwkbDubF2qM
Eating meat worse for planet than driving, animal rights groups say …
http://www.iht.com/articles/2007/08/29/business/climate.php
On meat eating and global warming
grist.org/advice/ask/2007/09/17/index.html
Eating Meat Contributes to Global Warming
http://www.chooseveg.com/global-warming.asp
Another Inconvenient Truth: Meat is a Global Warming Issue (By Dan Brook)
http://www.emagazine.com/view/?3312 – 67k – Cached
Cut global warming by becoming vegetarian
PhysOrg news: Cut global warming by becoming vegetarian … Global warming could be controlled if we all became vegetarians and stopped eating meat. …
http://www.physorg.com/news4998.html
Yahoo! Canada Answers – Is eating meat causing global warming?
ca.answers.yahoo.com/question/index?qid=20070913145620AAqIFgv
A New Global Warming Strategy (PDF)
Global warming poses one of the most serious threats to the global environment ever faced … most effective strategy for reducing global warming in our …
http://www.earthsave.org/news/earthsave_global_warming_report.pdf – 323k – View as html
Let me relate this to my current field of study – Nuclear Fusion.
The big money is going into projects like ITER (the US is spending something like $200 to $400 mil a year on this project). All the scientists involved say we are at least 30 years away from a net power reactor delivering watts to the grid. When that net power device is built it will be too big 17GW (most power plants built today are under 100MW and the largest are in the 1 GW range), too expensive (at 20X to 30X the current cost of electricity), and too late. All this is inherent in trying to get fusion by heating things up. And yet funding rolls on. Grant money is relatively easy if there is an ITER angle.
Contrast this with IEC fusion. In the US there are 5 to 10 projects going on at a funding rate that is probably on the order of $20 million or less total. The thing about IEC Fusion is that instead of heating up a mass of gas to get fusion in the high energy tail, particles are accelerated directly to fusion speeds. This makes the devices much smaller, less costly, and quicker to develop. So who is doing IEC Fusion? Basically a bunch of old cranks who see ITER and the Tokamaks as useless except as science fair projects. Let me quote Plasma Physicist Dr. Nicholas Krall who said, “We spent $15 billion dollars studying tokamaks and what we learned about them is that they are no damn good.”
And yet the money rolls on.
If I was in charge of science I would see that in any discipline 70% went to mainstream and 30% to dissenters. That would tend to keep everyone honest. Does it mean some money would go for stupidity? Sure. As Murray Gell-Mann says – there is a reason most new stuff ought not get funded, most of it is flat wrong. However, if we do not encourage dissent from orthodoxy we will never learn anything new.
Our current ratios are out of balance.
Let me add that a significant part of the 30% should go towards replication by dissenters.
If we are really going to do good science we must encourage a climate of dissent and replication.
Let me add that we see this in Cold Fusion. The mainstream derided it because at first replication was difficult. Now at least the laboratory aspects are better under control and replication is the norm. We still do not understand what is happening or why. However, finally progress is being made. So far it seems to be a low energy process. Heat is created. Just not enough to even boil the water (actually D2O) in the experimental apparatus. It is being researched. We will find out why. We lost 10 years of useful work because of clinging to orthodoxy.
In many way science is like religion. Woe be unto him who strays from the canon.
Interestingly enough the US Navy is funding IEC Fusion and Cold Fusion. Why? They don’t look at it from a right/wrong basis. It is all about risk vs reward. They are not crazy. They do require at least a minimum of results before funding. They come at it from: “we don’t know everything” and “mathematics can be helpful but is not definitive. Only real world results count.
Why not more dependence on math? Because with math – if you pick the right assumption – you can prove anything.
Think of the progress this article represents for science! And think of how much time and money we can all save, knowing as we now do that we can remove everyone else from scientific editorial boards and peer-reviewer lists, and replace them with one John Ioannidis.
Those reviewers can then return to performing useful scientific research — or dare I say it, the breadline! — and our science will be several times more accurate to boot.
#46. In climate science, the best way to improve things is to require authors to archive their data as used and methods at the time of publication so that you can see what they did exactly without having to run a guantlet. Of course, Team apologists object to this.
Does anyone know where comments like 48 come from? Is it some sort of bot?
#49 Larry, I looked at the link and it seems that Boris is a poster, and it seems to be a “we don’t like the “denialists” and we call them “cranks” rant blog: scienceblogs/denialism
They are talking about SteveM personally too, and calling him a “crank” . (Sheesh just about everybody is one if they don’t agree with these two men)
The owners of this blog are listed as : “Mark Hoofnagle is a MD/PhD Candidate in the Department of Molecular Physiology and Biological Physics at the University of Virginia. His interest in denialism concerns the use of denialist tactics to confuse public understanding of scientific knowledge.”
And “Chris Hoofnagle is an attorney with experience in consumer protection advocacy in Washington and Sacramento. His interest in denialism concerns the use of rhetorical tactics by various industries in dumbing down policy debates. He is the author of The Denialists’ Deck of Cards “
50, it sure looks like spam to me, complete with link to blog. I don’t know why Steve lets that kind of stuff stand.
51, is it called a “ping back” or something like that? …under the article on that blog there are these little icons to click to send or link “this article” up to a search engine or to “digg” it? …I don’t understand all that stuff. but it is like spam, yes.
(And the comments to this article there are really kinda insulting to read too-like SteveM said “Team apoligists” object to a discussion like this-AND they are exaggerating even further; saying that CA and SteveM think ALL science is “bunk”. Which ironically kind of proves to me that the “self serving, mundane behavior” is alive and well and may be affecting science accuracy. It reminds me of the child’s game “Telephone”. lol)
re 50.
I think they are from Rio Linda.
I especially liked the “we dont argue with Denialist” slant. True, even when you attempt to
you don’t.
Here is a especialy rich comment on fake experts, after invoking the Frye rule from the legal sphere.
” Therefore I would say a fake expert is usually somebody who is relied upon for their credentials rather than any real experience in the field at issue, who will promote arguments that are inconsistent with the literature, aren’t generally accepted by those who study the field in question, and/or whose theories aren’t consistent with established epistemological requirements for scientific inquiry. Sheesh. I just described Michael Egnor, Bill Dembski, Michael Fumento, Patrick Michaels, Steven Milloy, Richard Lindzen…”
Hey Expert on “fake experts” forgot to mention the MANN.
So, How to recognize a fake expert on fake experts. Does he list Mike Mann as a fake statistician.
It’s a bright line test
Lindzen has no experience in climatology? Or is the “field at issue” cooking data?
54, good gravy yes, what a good point. Too funny!
As far as that goes, how on earth are we supposed to evaluate the expertise of Rabett and Tomino, when they won’t even self-identify? That’s beyond “fake expert”, that’s “stealth expert”.
Not that I accept the basic premise that membership in the club is all that important; I’m just saying…
i promise you, i can hire a PhD “expert witness” to take any side of any argument you want to name. some such experts have even testified of both sides of some arguments. we called them conslutants ….
RE 55.
Bloom vouched for them. What more do you need to know.
It would seem that Dr. Ioannidis would disagree with your reading of his research:
[snip – you’ve used some religion-charged words that are not permitted on this blog.]
I quoted Dr. Ioannidis. Sorry if he offended you.
#59. Boris, there are certain religion-charged words that I do not permit here, because I don’t want the matters discussed here. IF you wish to discuss such topics, you can do so elsewhere. If Dr Ionnadis disagrees with any views that I’ve actually expressed, I would of course be quite happy to discuss specific matters.
2 Trackbacks
[…] published last month in the Journal of the American Medical Association …Original post by theduke delivered by Medtrials and […]
[…] Via Climate Audit […]