There’s an interesting article online here by David Goodstein of Caltech, in which he notices that misconduct problems seem rife in biological sciences administered by NIH and very infrequent in sciences administered by NSF. He identifies three factors as common in problems, noting that exact reproducibility in physical sciences is a major deterrent to fraud. If you look at the ingredients in climate science, I hardly need to editorialize.
Goodstein:
However, the larger number of cases arise from more self-interested motives. In the cases of scientific fraud that I have looked at, three motives, or risk factors have always been present. In all cases, the perpetrators,
1. were under career pressure;
2. knew, or thought they knew what the answer would turn out to be if they went to all the trouble of doing the work properly, and
3. were working in a field where individual experiments are not expected to be precisely reproducible.It is by no means true that fraud always occurs when these three factors are present; quite the opposite, they are often present and fraud is quite rare. But they do seem to be present whenever fraud occurs. Let us consider them one at a time.
Career Pressure: This is included because it is clearly a motivating factor, but it does not provide any distinctions. All scientists, at all levels from fame to obscurity are pretty much always under career pressure. On the other hand, simple monetary gain is seldom if ever a factor in scientific fraud.Knowing the answer: If we defined scientific fraud to mean knowingly inserting an untruth into the body of scientific knowledge, it would be essentially nonexistent, and of little concern in any case because science would be self-correcting. Scientific fraud is always a transgression against the methods of science, never purposely against the body of knowledge. Perpetrators always think they know how the experiment would come out if it were done properly, and decide it is not necessary to go to all the trouble of doing it properly. The most obvious seeming counter-example to this assertion is Piltdown man, a human- skull and ape-jaw planted in a gravel pit in England around 1908. If ever a fraudulent physical artifact was planted in the scientific record, this was it. Yet it is quite possible that the perpetrator was only trying to help along what was known or thought to be the truth. Prehistoric remains had been discovered in France and Germany, and there were even rumors of findings in Africa. Surely human life could not have started in those uncivilized places. And, as it turned out, the artifact was rejected by the body of scientific knowledge. Long before modern dating methods showed it to be a hoax in 1954, growing evidence that our ancestors had ape-skulls and human-jaws made Piltdown Man an embarrassment at the fringes of anthropology.
Reproducibility: In reality, experiments are seldom repeated by others in science. When a wrong result is found out, it is almost always because new work based on the wrong result doesn’t proceed as expected. Nevertheless, the belief that someone else can repeat an experiment and get the same result can be a powerful deterrent to cheating. This appears to be the chief difference between biology and the other sciences. Biological variability – the fact that the same procedure, performed on two organisms as nearly identical as possible is not expected to give exactly the same result – may provide some apparent cover for a biologist who is tempted to cheat. This last point, I think, explains why scientific fraud is found mainly in the biomedical area.
Climate Audit 
25 Comments
I think the name should be Goodstein?John replies: Quite correct. Name corrected
At the risk of being presumptuous I’d like to add one bit to his final point above. I think that the variability of individuals gives the opportunity for cheating in the biomedical area, but money is the dynamic principle. It’s the loot that perverts that area.
========================================================
Steve,
I would add that there is fraud and there is fraud. Hwang’s case is the extreme case where the experiments were never performed. It is also extreme in that the results announced were spectacular and made him a national hero. Like an Australian friend of mine once said, he didn’t just shoot himself in the foot, he deepthroated a bazooka! He would have been caught sooner or later and he probably knew it. That’s a pathological case, and in my opinion very rare.
Then there is the much more frequent sloppy or negligent piece of work, that you publish because you have to publish. All academics are under that pressure. So the ones with less ethics will just publish anything that comes out of the lab, and really not care too much about the validity of the results. They might also tweak the numbers a bit to make the results look better. And this happens not just in the biological sciences, but also a lot in the physical sciences (my personal field). And Goodstein is right: unless you publish spectacular and groundbreaking results, no one is going to repeat your experiment. Most of the science published today is almost totally irrelevant. It’s just unoriginal, incremental stuff that fills scientific journals, and that’s really just because everyone is judged on the number of papers they publish, and not so much on their quality.
Steve, your desire to see science follow the same disclosure rules as public companies is unfortunately unrealistic in the current “publish or perish” context. Furthermore, 99.9% of the time, an erroneous paper has no negative financial implications for anyone. In other words: nobody cares!
On the other hand, I do agree that a journal that claims to be “prestigious” like Nature or Science, and publish groundbreaking results, should be much more careful and diligent. Peer review is probably not enough, at least not in its current form.
I am not much convinced about the case for singling out “biomedical science”. Firstly, there is a substantial grey area between biomedical and biology and medicine.
Secondly, I think the physicist Jan Hendrik Schàƒ had 8 Science and 7 Nature papers withdrawn, so the relative innocence of other areas isn’t clear to me.
Thirdly, I note that “biomedical” science gets a fantastic proportion of current spending in basic science. It may be that biomedical science has more scientific fraud because there is more biomedical science.
You could have quite a strong view on the career pressure issue. To be successful in biomedical, it is arguable that you have to have far more grant income, get better papers, than you do in other fields.
I am intrigued by the rest of the argument presented; Goodstein seems happy to cast the argument in terms of “intent to deceive”. While this is great stuff, and allows them to weed out a lot of minor transgressions, I am sure that the relevant legal eagles would have great fun with institutions finding people guilty of “intent to deceive”.
cheers
per
Francois, I’m not advocating that peer reviewers should necessarily be wading through details like I’ve done. However, it is simple for paleoclimate journals to adopt the “best practices” already in place at some economics journals under which archiving of working code and data is a condition of publication. That immensely reduces the “cost” and trouble of replication and would be a little deterrent to slopy work.
I’m probably more critical of IPCC. They are perceived by the pulic to be bringing an additional level of due diligence, but they neglect to do so. IPCC refused to give me access as a reviewer to data for an unpublished study and threatened to expel me as a reviewer if I asked authors for data. They stated that the job of an IPCC reviewer was merely to see if the IPCC report accurately reflected the journal publication – not to carry out additional review of the article (even if still unpublished).
For policy purposes, you can’t rely on studies which have merely had the cursory due diligence of journal peer review. I’m not saying that you should turn journal policies upside down since most articles are not relevant to policy. But if you relying on an article -especially one that’s hot off the press,as these guys do, because they are always “moving on” as soon as you pin them down on any past article, then surely some additional due diligenece is needed somewhere.
per – I perhaps should have put some comments in. I wasn’t really thinking about a beauty contest between science fields. What caught my eye was how well climate science fit the last 2 criteria. They all are not only convinced that they know the “right” answer; if one avenue is flawed, they argue that they can “get” the answer some other way. Secondly, it seems it never really occurred to them that anyone would ever try to replicate their work exactly; before I materialized into their lives, it doesn’t seem that anyone ever tried to; they find this offensive. Sound like Goodstein’s third point?
Hi steve
yes, my comments about biomed may be tangential.
I think Goodstein’s point 1 is also quite relevant. If you are working in a field where you don’t get headlines, don’t get Nature papers, etc., then it may be true that your career progression is not so good. In the case at hand, it might be interesting to comment on the perception of dendrochronology pre- and post-Mann. Before, was there a perception that this was a “worthy but boring” field ? After MBH98, it becomes clear that this field has the power to help us save the world ! I am sure that MBH’98 in Nature, and the subsequent raft of high-profile publications, are not bad for the MBH career trajectories.
Re: replication. I don’t share Goodstein’s view that fraud/ sloppy science is so very rare. I have come into contact with more examples recently of data that cannot be reproduced; and it leaves me with a foul taste in my mouth. It is axiomatic that scientists must allow replication of their data. If there is a current ethos that some scientists do not have to share their data, I find that disturbing.
yours
per
Another interesting article on the topic.
http://www.independent.org/newsroom/article.asp?id=1666
Amazing how often “Is the report, including all data and methodology, available for examination?” comes at the very top of the list.
ETSV, that’s a good link. We cited that Nature editorial last May when we were objecting to the UCAR press release about Ammann and Wahl claiming that all our criticisms were “unfounded”, which then got used by Houghton, Mann and the European Geophysical Union in evidence to Congress. Funny, I didn’t notice Nature (or any climate scientists) objecting to the UCAR press release.
My point about peer review is more prosaic: if its not there to catch or deter fraud, then what is it for?
To catch mistakes or errors.
Never credit anything to willful fraud if it can be explained by the much more prevalent ignorance or human fallibility.
While I think the cases of fraud (differing levels) is greater than Pat Frank mentioned in his post, I still think that incompetence is #1 with willful fraud a distant 2nd.
Speaking in general, not just about Climate “Science”
Re 10, John A’s questions what is peer review good for.
My father-in-law, Dr AB Hollingshead, a noted social scientist and department head at Yale in the 60s and 70s were discussed the weakness of the peer review processes over a couple of beers in my back yard in 1973. He pointed out that most shifts in scientific theory comes at generational boundaries, as those protecting 20-30 years of academic work die off, allowing the next generation to stake their reputation on new ideas and better information. He saw peer review as nothing more than a job protection mechanism, newly minted academics conformed to the current dogma, or they do not get published. In a publish or perish environments, this could have long term implications for young professors, but it was job protection for the old guard. In dad’s view when the old guard died off, there was a window of opportunity to introduce new ideas. Now, we have outsiders like Steve and Ross, who are not waiting for a generational boundary to identify the errors of the old guard and providing new insight to the problems of calculating past temperature trends from cherry picked tree rings.
The other important difference is the significant policy implications of bad science. How many headlines on have appeared in the past couple of days siting GW as the reason for some impending doom (with the implication that the warming is anthropogenic)? “Consensus” has been achieved, even if it is/was based on problematic science.
To the list above, I’d add #4: Social/political pressure. A scientist has a certain world view and sets about to validated that world view. (Related to #2)
Who’s going to repair the damages caused by policy derived from this bad science?
===========================================================
Re: 12 Excellent points.
Re #12&13
I do not believe that this is quite correct. 1998 to 2003 is not what I would call a generation (in humans anyway).
Prior to MBH98 the MWP and LIA were recognised as historical global climate events. MBH98 presented “evidence” that the MWP and LIA were not climate events in the Northern Hemisphere and were probably regional climate events limited to parts of Europe and the North Atlantic where they registered qualitatively in written historical records.
It should be recognised that the MBH98 results were (and continue to be) very, very politically correct on at least two counts:
1. They supported the IPCC notion of something unusual happening climate wise, and
2. They supported the prevailling believe that a Eurocentric view of history has skewed global perceptions of everything that has happened up till now.
Re #11.
At what point does perpetuating an error become fraud? let’s say a person’s ego demanded that they never make a mistake…
Re: #16
Jeff:
But, if Steve and Ross had not come to the fore, it might be 20 or more years before the next generation could fix the problem. Let’s not wait that long.
It’s still not willful if ego driven, as the person does not think they have made an error.
To be perfectly clear I do not think this is the case (that it is an error) in MBH98 or within the AGW comunity at large, hence my “speaking in general” comment.
There is just no way this is a simple error. The total lack of cooperation, secrecy, and malicious comments prove that to me.
RE: “They all are not only convinced that they know the “right” answer; if one avenue is flawed, they argue that they can “get” the answer some other way.”
As an ex-Gaia worshipper and ex-political far Leftist (some might call me an ex-Trotskyite) I can speak to a possible aspect of this. I have spent some time at Real Climate and other venues of the warmers. There is an overt thread of Ecotopia / Gaia worship / quasi-Marxist “Green” polity woven through what I see there. Part of the belief pattern is a natural extension of things which were going on 1965 – 1985 in the realm of Whole Earth Almanac, Utne Reader, and the writings of Amory Lovins, David Brower and others of that genre. I was in the thick of it all during the early 1980s. I recognize certain world views. Now, if we surmise that at least some of those who were similarly affected, and, quite unlike me, kept their youthful views and went into the “climate science” field, I can well imagine a strong urge to cling to preconceived notion which would render cultural and political support to the Gaia / Ecotopia ideal.
I suspect it started with simple error, a mistaken believe in the proxy of tree rings, then combined with poor data, and a credulous world, it has simply become too embarassing to renounce, when it seems that with a possibly warming world they may never be revealed as the charlatans they are. That’s a simple error, mildly sinful. What may be irretrievable are policy changes. We’re getting beyond sinful here.
=======================================================================
RE: “I do not believe that this is quite correct. 1998 to 2003 is not what I would call a generation (in humans anyway).
“Prior to MBH98 the MWP and LIA were recognised as historical global climate events. MBH98 presented “evidence” that the MWP and LIA were not climate events in the Northern Hemisphere and were probably regional climate events limited to parts of Europe and the North Atlantic where they registered qualitatively in written historical records.”
As I see it, the generation / old guard in question is comprised of Baby Boomers as well as some from the head end of “Generation X” whom they indoctrinated. Conveniently, the Boomers are the generation most likely to embrace the Malthusian notions inherent in the biases we see such strong evidence of among the Hockey Team and those who support and follow them.
#5 Steve, I was just saying that for most of scientific publications, it’s hard to do anything more than peer review, and even peer review is mostly flawed as a process.
In your case, since the MBH paper dealt mostly with tabulated data and computer code, it is quite easy to replicate, and I also agree that you should have been able to get both data and code without any problem. But the problem came from the authors themselves. It’s very hard to understand their position, other than that they realized that their result was either very weak or plain wrong, and the implications for their own career and status suddenly became enormous (remember Mann was named a “visionary”).
On the other hand, experimental science can be very hard to reproduce: some experimental setups require expensive, specialized pieces of equipment, and often take months or even years to build and refine, until all sources of errors or artefacts have been eliminated, and the accuracy is good enough to measure what you want. Think particle physics, for example, where there is sometimes only one accelerator in the world. There’s no way a reviewer can check the results: you have two weeks to review, and that’s all volunteer work!
The IPCC is a very different beast. If it’s going to be used to determine public policy, there should be a very careful and independent review process. This is clearly not happening: it’s just a collection of papers put together by the very same people who wrote them. I think the politicians put too much faith in the objectivity and independence of scientists! The whole process is based on the “consensus” idea: if enough scientists agree on something, then it must be true! Over time, I think politicians will realize how flawed this process was. Unfortunately, the credibility of science will suffer from it.
There will be damages, and not just to the credibility of science. Mitigating those damages is the urgency.
=========================================
Re: #23
I think that is a very good point about the nature of what the IPCC is.
I am aware of praxis on (some) government regulatory committees. They are supported by civil servants, who have a (sometimes) clear view of what should be addressed and why. As scientists, data is interpreted, and although there is some leeway for judgement, it is not a place for wild speculation or theory. The science has to be well founded and justifiable, and it just isn’t done to argue only one side of a hypothesis. The consequences of making a regulatory decision weigh heavily on all concerned.
It is not clear to me what sort of beast the IPCC is, and whether it sees itself as a crusading organisation, or with any power, and hence, accountability. If you can make recommendations, and you are not responsible for the consequences, that is an interesting situation.
cheers
per