"Miscalculation, poor study design or self-serving data analysis"

Dr. Ioannidis, an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass, has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye. He writes in the WSJ as follows:

Most Science Studies Appear to Be Tainted By Sloppy Analysis

We all make mistakes and, if you believe medical scholar John Ioannidis, scientists make more than their fair share. By his calculations, most published research findings are wrong. …

These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. “There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims,” Dr. Ioannidis said. “A new claim about a research finding is more likely to be false than true..

The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.

Sounds to me like he’d support checking for “miscalculation, poor study design or self-serving data analysis” in climate science as well.


58 Comments

  1. Wayne Holder
    Posted Sep 19, 2007 at 10:03 AM | Permalink

    Gee, I posted about this yesterday in the “Should NASA climate accountants adhere to GAAP?” thread (comment 125), but I guess everybody was too buy debating Boris to take notice…

  2. Reid
    Posted Sep 19, 2007 at 10:04 AM | Permalink

    Dr. Ioannidis says “A new claim about a research finding is more likely to be false than true.. ”

    except, of course, his new claim about a research finding.

  3. CO2Breath
    Posted Sep 19, 2007 at 10:20 AM | Permalink

    RE: #2

    It does sound a little self serving, now that I think about it. Meta-Science degrees coming soon?

  4. Mike
    Posted Sep 19, 2007 at 10:22 AM | Permalink

    Reid,
    I suppose you could audit his research and find out. Nah, it’s probably easier to just snipe at it on a blog.

  5. Lance
    Posted Sep 19, 2007 at 10:23 AM | Permalink

    One should be particularly wary of studies where the author originally refuses to release his data and methodology and then changes the data and methodology after being pressured to comply with the most rudimentary standards of scientific inquiry.

  6. Reid
    Posted Sep 19, 2007 at 10:32 AM | Permalink

    Re #4

    I agree with Dr. Ioannidis. However, I found the irony of his claim humorous. Monty Python fodder.

  7. Posted Sep 19, 2007 at 11:01 AM | Permalink

    With Steve’s permission I’ll post some other gems from the same article (can’t link it, it’s subscription):

    \\”Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. “People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual,” Dr. Ioannidis said.//

  8. Geoff
    Posted Sep 19, 2007 at 11:02 AM | Permalink

    John Brignell has written a lot on the subject of weak
    epidemiology,

    http://www.numberwatch.co.uk/

    He is fequently attacked by the usual cadre.
    That clearly stands to his credit.

  9. Mike
    Posted Sep 19, 2007 at 11:05 AM | Permalink

    Reid,
    Sorry, I misunderstood how your comment was meant. Sometimes I should just keep my big mouth shut.

  10. Larry
    Posted Sep 19, 2007 at 11:10 AM | Permalink

    The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.

    So to speak…

  11. David
    Posted Sep 19, 2007 at 11:15 AM | Permalink

    Any good scientist knows that to study something that is variable, you have to hold the other variables that affect that variable constant. However, it seems that all to often studies are funded knowing in advance that they do not hold to this basic principle even when they could have. If the general person on the street can recognize a flawed study, why can’t scholars and journalists? This tells me that these studies are done this way on purpose and are NOT mistakes or the result of sloppiness. These studies are make-work for funding, profile, and politics.

  12. Mark T.
    Posted Sep 19, 2007 at 11:21 AM | Permalink

    but I guess everybody was too buy debating Boris to take notice…

    Which is often way more fun, sort of a stress-relief exercise.

    I think the way Dr. Ioannidis’ paper should be viewed is as confirmation that scientific claims should always be viewed with skepticism. The more outstanding the results, the more outstanding the evidence should be. Unfortunately, mainstream media, and the public at large, do not do so.

    Mark

  13. Posted Sep 19, 2007 at 11:45 AM | Permalink

    From the same article by Robert Lee Hotz (!):

    In the U. S., research is a $55-billion-a-year enterprise that stakes its credibility on the reliability of evidence and the work of Dr. Ioannidis strikes a raw nerve. In fact, his 2005 essay “Why Most Published Research Findings Are False” remains the most downloaded technical paper that the journal PLoS Medicine has ever published.

    “He has done systematic looks at the published literature and empirically shown us what we know deep inside our hearts,” said Muin Khoury, director of the National Office of Public Health Genomics at the U.S. Centers for Disease Control and Prevention. “We need to pay more attention to the replication of published scientific results.”

  14. Larry
    Posted Sep 19, 2007 at 11:53 AM | Permalink

    This article is really a statement of the obvious. How many times, especially when medicine or biotechnology are involved, have we read a story that goes something like “A causes B to rise”, only to see another one six months later that says “A cause B to fall”, only to see another “A causes B to rise” story in another six months?

    If you’re not skeptical of science in the news, you’re either not paying attention, or your one of those people born every minute.

  15. bernie
    Posted Sep 19, 2007 at 11:53 AM | Permalink

    theDuke (#13)
    Replication is the key. Replication requires the release of data and code.

  16. bernie
    Posted Sep 19, 2007 at 11:58 AM | Permalink

    #14
    As numerate skeptics I think tyou are correct. The value of the article is not that it does happen but the scope of the problem. It also serves to justify the mantra of this site: Show me the data, show me the code.

  17. allanj
    Posted Sep 19, 2007 at 12:09 PM | Permalink

    #15
    Replication is certainly critical, but there also needs to be enforcement. Somehow the science community must make it more painful to publish erroneous results than to do research that produces little of importance.

  18. A. Sucker
    Posted Sep 19, 2007 at 12:14 PM | Permalink

    Larry,

    What people?

  19. Gerald Machnee
    Posted Sep 19, 2007 at 12:18 PM | Permalink

    Now we know why some “climate scientists” do not want their work audited.

  20. Larry
    Posted Sep 19, 2007 at 12:21 PM | Permalink

    Case in point:

    http://www.time.com/time/health/article/0,8599,1662484,00.html

    Now, I don’t claim to know if they’re on to something or not in this article, but if the science on the relationship between fat and cancer isn’t “settled” by now, how on God’s green earth do they expect us to believe that the science of climate change is “settled”?

  21. CO2Breath
    Posted Sep 19, 2007 at 12:24 PM | Permalink

    Mark T. says:
    September 19th, 2007 at 11:21 am

    but I guess everybody was too buy debating Boris to take notice..

    Boris is the Muppet, Beaker.

    On the internet, no one knows if you’re a Muppet.

  22. Michael Jankowski
    Posted Sep 19, 2007 at 12:44 PM | Permalink

    Re#14

    This article is really a statement of the obvious. How many times, especially when medicine or biotechnology are involved, have we read a story that goes something like “A causes B to rise”, only to see another one six months later that says “A cause B to fall”, only to see another “A causes B to rise” story in another six months?

    There were concurrent publications in different medical journals maybe 6 months ago about caffeine and a heart/pulmonary issue (can’t rememeber if it was heart attacks, heart disease, strokes, high blood pressure, or what – but it was the same isse for both article). One publication said caffeine reduced risk, while the other said it increase it. Those were to strange press releases to read abck-to-back.

  23. Reid
    Posted Sep 19, 2007 at 12:46 PM | Permalink

    “57.2% of all statistics are made up”

    Steven Wright
    comedian

  24. Hoi Polloi
    Posted Sep 19, 2007 at 12:56 PM | Permalink

    “A scientist will never show any kindness for a theory which he did not start himself.” (Mark Tawin)

  25. Posted Sep 19, 2007 at 1:07 PM | Permalink

    Bernie, #15: on the subject of replication from the same article:

    Take the discovery that the risk of disease may vary between men and women, depending on their genes. Studies have prominently reported such sex differences for hypertension, schizophrenia and multiple sclerosis, as well as lung cancer and heart attacks. In research published last month in the Journal of the American Medical Association, Dr. Ioannidis and his colleagues analyzed 432 published research claims concerning gender and genes.

    Upon closer scrutiny, almost none of them held up. Only one was replicated.

  26. Boris
    Posted Sep 19, 2007 at 1:50 PM | Permalink

    but I guess everybody was too buy debating Boris to take notice…

    Is debating now defined as posting pictures of Beaker? Or saying “Goerbal warming”?

    Oh, how’s the comparison between CRN1/2 and CRN5 sites going?

  27. Larry
    Posted Sep 19, 2007 at 2:20 PM | Permalink

    27, not debating Boris, baiting Boris…

  28. Sam Urbinto
    Posted Sep 19, 2007 at 2:25 PM | Permalink

    I agree with Larry, pretty much stating the obvious.

  29. Earle Williams
    Posted Sep 19, 2007 at 2:28 PM | Permalink

    Re #27

    Boris,

    Speaking of results, any progress on your analysis of microsite thermal impacts of air conditioning units?

    Cheers!
    Earle

  30. Sam Urbinto
    Posted Sep 19, 2007 at 2:48 PM | Permalink

    Larry in #14 about the article being rather self evident.

  31. Larry
    Posted Sep 19, 2007 at 3:00 PM | Permalink

    29, 32 – right. If the media had a story about how bad the media is, that’s a “man bites dog” story. If the a scientist writes about how bad science is, that’s a “dog bites man” story. I had thought that most people had already become jaded about both media reporting and scientific research.

  32. Ed
    Posted Sep 19, 2007 at 3:26 PM | Permalink

    Per an old study in Science in the 60s, 35% of published peer reviewed papers are never cited in the literature, 49% are cited only once, and just 1% are cited six or more times. Suggesting that most research is hardly looked at, let alone replicated or audited.

    Link to study: http://www.garfield.library.upenn.edu/papers/pricenetworks1965.pdf

  33. CO2Breath
    Posted Sep 19, 2007 at 3:41 PM | Permalink

    Oh, how’s the comparison between CRN1/2 and CRN5 sites going?

    Boris, how’s the investigation of continuity of GISS data manipulations of monthly/daily USHCN station data going? Corelations with precipitation? Posted any data or analysis lately? Didn’t think so.

  34. CO2Breath
    Posted Sep 19, 2007 at 3:49 PM | Permalink

    Re: #27

    Or saying “Goerbal warming”?”

    It’s actually Gorebal Warning. Too subtle for some, I suppose.

  35. CO2Breath
    Posted Sep 19, 2007 at 3:52 PM | Permalink

    Re: #27

    Posting images of Beaker?

    Actually only a link. I’m not sure how to post the image or if hot linking here is allowed. It’s still funny to me though.

  36. steven mosher
    Posted Sep 19, 2007 at 4:03 PM | Permalink

    RE 28. LARRY……………………………………………..

    Stop feeding the Dung beetle

  37. Posted Sep 19, 2007 at 5:28 PM | Permalink

    More from Robert Lee Hotz:

    Every new fact discovered through experiment represents a foothold in the unknown. In a wilderness of knowledge, it can be difficult to distinguish error from fraud, sloppiness from deception, eagerness from greed or, increasingly, scientific conviction from partisan passion. As scientific findings become fodder for political policy wars over matters from stem-cell research to global warming, even trivial errors and corrections can have larger consequences.

    Still, other researchers warn not to fear all mistakes. Error is as much a part of science as discovery. It is the inevitable byproduct of a search for truth that must proceed by trial and error. “Where you have new areas of knowledge developing, then the science is going to be disputed, subject to errors arising from inadequate data or the failure to recognize new matters,” said Yale University science historian Daniel Kevles. Conflicting data and differences of interpretation are common.

    To root out mistakes, scientists rely on each other to be vigilant. Even so, findings too rarely are checked by others or independently replicated. Retractions, while more common, are still relatively infrequent. Findings that have been refuted can linger in the scientific literature for years to be cited unwittingly by other researchers, compounding the errors.

  38. Posted Sep 19, 2007 at 6:10 PM | Permalink

    Did you know that Climate Change causes the North Atlantic to increase in salinity?

    http://powerandcontrol.blogspot.com/2007/09/global-warming-causes.html

    It also causes the North Atlantic salinity to decline

    http://powerandcontrol.blogspot.com/2007/09/global-warming-causes.html

    Yep. Same url for both (I’m a collector).

  39. Posted Sep 19, 2007 at 8:03 PM | Permalink

    M. Simon, #40: We’re experiencing the perfect propaganda storm: publishers and editors want material that examines global warming because it sells, and the “scientific” (I use the term loosely) AGW crowd is only too happy to provide it.

  40. Richard deSousa
    Posted Sep 19, 2007 at 11:00 PM | Permalink

    Re #11: Ah, yes, the joys of variables. Michael Crichton spoke often about the Drake equation N=N*fp ne fl fi fc fL, which, according to him was a useless formula. Seems like the AGW crowd is using a variation of the Drake equation to come up with their outlandish projections:

    http://www.michaelcrichton.net/speech-ourenvironmentalfuture.html

  41. Ulises
    Posted Sep 20, 2007 at 2:11 AM | Permalink

    #42 : Ah, well, I would in first place enjoy a consistent notation. If my guess is correct that the terms in the equation are all to be multiplied, why is there an asterisk in one place and spaces in the rest ? Talking about science….

  42. D. Patterson
    Posted Sep 20, 2007 at 3:22 AM | Permalink

    I thought that was supposed to be de-baiting Boris…you know…hook, sinker, Waldo.

    Goerbels, Goebbels, Gorebells, Gorepals warning…there’s a difference?

    Did you know the consensus scientific opinion has found that eating meat causes Global Warming? Will eating meat be banned at the next meeting of the IPPC?

    ABC News: Meat-Eaters Aiding Global Warming?
    abcnews.go.com/Technology/story?id=1856817&page=1

    GoVeg.com // Meat and the Environment // Pollution // Fight Global …
    … would if you just gave up eating meat and other animal products. … individuals to avert global warming is to stop eating meat, eggs, and dairy products. …
    http://www.goveg.com/environment-globalwarming.asp – 29k – Cached

    YouTube – Al Gore ignores eating meat global warming
    Glenn Beck on how Al gore is a hypocrite. This shows how rotten Gore is. Gore is all about grabbing taxes. Do not eat meat if you want to stop global warming…
    http://www.youtube.com/?v=VwkbDubF2qM

    Eating meat worse for planet than driving, animal rights groups say …
    http://www.iht.com/articles/2007/08/29/business/climate.php

    On meat eating and global warming
    grist.org/advice/ask/2007/09/17/index.html

    Eating Meat Contributes to Global Warming
    http://www.chooseveg.com/global-warming.asp

    Another Inconvenient Truth: Meat is a Global Warming Issue (By Dan Brook)
    http://www.emagazine.com/view/?3312 – 67k – Cached

    Cut global warming by becoming vegetarian
    PhysOrg news: Cut global warming by becoming vegetarian … Global warming could be controlled if we all became vegetarians and stopped eating meat. …
    http://www.physorg.com/news4998.html

    Yahoo! Canada Answers – Is eating meat causing global warming?
    ca.answers.yahoo.com/question/index?qid=20070913145620AAqIFgv

    A New Global Warming Strategy (PDF)
    Global warming poses one of the most serious threats to the global environment ever faced … most effective strategy for reducing global warming in our …
    http://www.earthsave.org/news/earthsave_global_warming_report.pdf – 323k – View as html

  43. Posted Sep 21, 2007 at 6:19 AM | Permalink

    Let me relate this to my current field of study – Nuclear Fusion.

    The big money is going into projects like ITER (the US is spending something like $200 to $400 mil a year on this project). All the scientists involved say we are at least 30 years away from a net power reactor delivering watts to the grid. When that net power device is built it will be too big 17GW (most power plants built today are under 100MW and the largest are in the 1 GW range), too expensive (at 20X to 30X the current cost of electricity), and too late. All this is inherent in trying to get fusion by heating things up. And yet funding rolls on. Grant money is relatively easy if there is an ITER angle.

    Contrast this with IEC fusion. In the US there are 5 to 10 projects going on at a funding rate that is probably on the order of $20 million or less total. The thing about IEC Fusion is that instead of heating up a mass of gas to get fusion in the high energy tail, particles are accelerated directly to fusion speeds. This makes the devices much smaller, less costly, and quicker to develop. So who is doing IEC Fusion? Basically a bunch of old cranks who see ITER and the Tokamaks as useless except as science fair projects. Let me quote Plasma Physicist Dr. Nicholas Krall who said, “We spent $15 billion dollars studying tokamaks and what we learned about them is that they are no damn good.”

    And yet the money rolls on.

    If I was in charge of science I would see that in any discipline 70% went to mainstream and 30% to dissenters. That would tend to keep everyone honest. Does it mean some money would go for stupidity? Sure. As Murray Gell-Mann says – there is a reason most new stuff ought not get funded, most of it is flat wrong. However, if we do not encourage dissent from orthodoxy we will never learn anything new.

    Our current ratios are out of balance.

    Let me add that a significant part of the 30% should go towards replication by dissenters.

    If we are really going to do good science we must encourage a climate of dissent and replication.

    Let me add that we see this in Cold Fusion. The mainstream derided it because at first replication was difficult. Now at least the laboratory aspects are better under control and replication is the norm. We still do not understand what is happening or why. However, finally progress is being made. So far it seems to be a low energy process. Heat is created. Just not enough to even boil the water (actually D2O) in the experimental apparatus. It is being researched. We will find out why. We lost 10 years of useful work because of clinging to orthodoxy.

    In many way science is like religion. Woe be unto him who strays from the canon.

    Interestingly enough the US Navy is funding IEC Fusion and Cold Fusion. Why? They don’t look at it from a right/wrong basis. It is all about risk vs reward. They are not crazy. They do require at least a minimum of results before funding. They come at it from: “we don’t know everything” and “mathematics can be helpful but is not definitive. Only real world results count.

    Why not more dependence on math? Because with math – if you pick the right assumption – you can prove anything.

  44. Posted Sep 21, 2007 at 7:36 AM | Permalink

    Think of the progress this article represents for science! And think of how much time and money we can all save, knowing as we now do that we can remove everyone else from scientific editorial boards and peer-reviewer lists, and replace them with one John Ioannidis.

    Those reviewers can then return to performing useful scientific research — or dare I say it, the breadline! — and our science will be several times more accurate to boot.

  45. Steve McIntyre
    Posted Sep 21, 2007 at 7:49 AM | Permalink

    #46. In climate science, the best way to improve things is to require authors to archive their data as used and methods at the time of publication so that you can see what they did exactly without having to run a guantlet. Of course, Team apologists object to this.

  46. Larry
    Posted Sep 22, 2007 at 3:26 PM | Permalink

    Does anyone know where comments like 48 come from? Is it some sort of bot?

  47. welikerocks
    Posted Sep 23, 2007 at 6:45 AM | Permalink

    #49 Larry, I looked at the link and it seems that Boris is a poster, and it seems to be a “we don’t like the “denialists” and we call them “cranks” rant blog: scienceblogs/denialism

    They are talking about SteveM personally too, and calling him a “crank” . (Sheesh just about everybody is one if they don’t agree with these two men)
    The owners of this blog are listed as : “Mark Hoofnagle is a MD/PhD Candidate in the Department of Molecular Physiology and Biological Physics at the University of Virginia. His interest in denialism concerns the use of denialist tactics to confuse public understanding of scientific knowledge.”

    And “Chris Hoofnagle is an attorney with experience in consumer protection advocacy in Washington and Sacramento. His interest in denialism concerns the use of rhetorical tactics by various industries in dumbing down policy debates. He is the author of The Denialists’ Deck of Cards “

  48. Larry
    Posted Sep 23, 2007 at 8:25 AM | Permalink

    50, it sure looks like spam to me, complete with link to blog. I don’t know why Steve lets that kind of stuff stand.

  49. welikerocks
    Posted Sep 23, 2007 at 9:03 AM | Permalink

    51, is it called a “ping back” or something like that? …under the article on that blog there are these little icons to click to send or link “this article” up to a search engine or to “digg” it? …I don’t understand all that stuff. but it is like spam, yes.

    (And the comments to this article there are really kinda insulting to read too-like SteveM said “Team apoligists” object to a discussion like this-AND they are exaggerating even further; saying that CA and SteveM think ALL science is “bunk”. Which ironically kind of proves to me that the “self serving, mundane behavior” is alive and well and may be affecting science accuracy. It reminds me of the child’s game “Telephone”. lol)

  50. steven mosher
    Posted Sep 23, 2007 at 10:47 AM | Permalink

    re 50.

    I think they are from Rio Linda.

    I especially liked the “we dont argue with Denialist” slant. True, even when you attempt to
    you don’t.

    Here is a especialy rich comment on fake experts, after invoking the Frye rule from the legal sphere.

    ” Therefore I would say a fake expert is usually somebody who is relied upon for their credentials rather than any real experience in the field at issue, who will promote arguments that are inconsistent with the literature, aren’t generally accepted by those who study the field in question, and/or whose theories aren’t consistent with established epistemological requirements for scientific inquiry. Sheesh. I just described Michael Egnor, Bill Dembski, Michael Fumento, Patrick Michaels, Steven Milloy, Richard Lindzen…”

    Hey Expert on “fake experts” forgot to mention the MANN.

    So, How to recognize a fake expert on fake experts. Does he list Mike Mann as a fake statistician.
    It’s a bright line test

  51. Larry
    Posted Sep 23, 2007 at 11:16 AM | Permalink

    Therefore I would say a fake expert is usually somebody who is relied upon for their credentials rather than any real experience in the field at issue

    Lindzen has no experience in climatology? Or is the “field at issue” cooking data?

  52. welikerocks
    Posted Sep 23, 2007 at 11:32 AM | Permalink

    54, good gravy yes, what a good point. Too funny!

  53. Larry
    Posted Sep 23, 2007 at 12:02 PM | Permalink

    As far as that goes, how on earth are we supposed to evaluate the expertise of Rabett and Tomino, when they won’t even self-identify? That’s beyond “fake expert”, that’s “stealth expert”.

    Not that I accept the basic premise that membership in the club is all that important; I’m just saying…

  54. Posted Sep 23, 2007 at 4:02 PM | Permalink

    i promise you, i can hire a PhD “expert witness” to take any side of any argument you want to name. some such experts have even testified of both sides of some arguments. we called them conslutants ….

  55. steven mosher
    Posted Sep 23, 2007 at 7:52 PM | Permalink

    RE 55.

    Bloom vouched for them. What more do you need to know.

  56. Boris
    Posted Oct 2, 2007 at 12:57 PM | Permalink

    It would seem that Dr. Ioannidis would disagree with your reading of his research:

    [snip – you’ve used some religion-charged words that are not permitted on this blog.]

  57. Boris
    Posted Oct 2, 2007 at 1:14 PM | Permalink

    I quoted Dr. Ioannidis. Sorry if he offended you.

  58. Steve McIntyre
    Posted Oct 2, 2007 at 1:59 PM | Permalink

    #59. Boris, there are certain religion-charged words that I do not permit here, because I don’t want the matters discussed here. IF you wish to discuss such topics, you can do so elsewhere. If Dr Ionnadis disagrees with any views that I’ve actually expressed, I would of course be quite happy to discuss specific matters.

2 Trackbacks

  1. […] published last month in the Journal of the American Medical Association …Original post by theduke delivered by Medtrials and […]

  2. […] Via Climate Audit […]

Follow

Get every new post delivered to your Inbox.

Join 3,382 other followers

%d bloggers like this: