Lots of people have criticized me for the mere idea of auditing scientific articles. Think of the many blog-posters who have ridiculed this as a total waste of time for scientists, who should be getting on with more "productive" work. Here’s an endorsement for the concept reported in yesterday’s Washington Post.
The Washington Post stated here.:
There is tremendous pressure today to be first. If you do something first, all the money and fame will come to you," said Adil E. Shamoo, a biomedical ethicist at the University of Maryland. "All that is an obvious seduction for doing something like this."…
"Unfortunately, research institutions and universities have not taken this issue seriously," said Shamoo, who edits the journal Accountability in Research. "They pay lip service to responsible conduct and research integrity." "It’s unconscionable" they don’t do more, he said. Beyond education, independent auditors could make spot checks in labs to verify data in much the same way the Internal Revenue Service polices taxpayers. If the IRS never conducted audits, what do you think would happen to people paying taxes in this country? If the public knows the IRS never conducts audits, don’t you think there will be an increase in problems?" Shamoo said.
Climate Audit 
23 Comments
Which basically implies that the current climatology hinges on unaudited paleo climate hypotheses from the past. Who audited Agassiz when he invented the Ice ages?; or Arrhenius when he overstated the effect of CO2 forcing considerably?; or Dansgaard when he invented that isotopes in ice are temperatures?; or numerous carbon dated events that have been mistaken for real calendar dates before the calibration with counted proxies was possible?
So everybody is happy with that? continuing climate assumptions on highly misunderstood past paleao climates
“Negative work” is exactly as important as “positive work”. When learning the truth, having a “no” answer has the same value as having a “yes” answer. This is easy to see – simply because for every question, there exists a negated questions whose answers are reverted. 😉
People in science have become too much “synergetic”. This kind of “scientific consensus” – or “consensus science” as Michael Crichton would call this pernicious development – is everywhere around us. Not only in climate science and stem cell research. We can see it in all other fields, and theoretical physics is no exception. It’s completely wrong.
Incidentally, this Hwang consensus has had a political flavor, too. Many people (journalists etc.) in the U.S. liked the idea that the Koreans are better and the U.S. science is falling behind – because it’s another new slogan that they can use to criticize George W. Bush. Of course that such a criticism for Korean successes – even if they were real – is completely moronic, but they don’t care.
I don’t think that the critical evaluation and verifications should be made by some kinds of bureaucrats who would be intimidating the scientists. It must be done by scientists who are equally educated and skillful in doing their business. (Of course that this includes Steve McIntyre in the case of the climate.)
What should happen is simply that the fractions of positive and negative results should converge. I am inclined to believe that a scientist who “believes” significantly more than 50 percent of the claims made around is not doing her or his job right. This must be eliminated.
There must exist some mechanisms to revive the natural criticism and skepticism of the scientific community. It’s not easy. You must understand that many scientists are applying for grants and submitting papers – and they are often refereed and judged by their colleagues. So no one really wants to make his or her colleagues upset because this could negatively influence his own grant applications and the fate of his own papers.
This looks like a bad by-product of the current establishment system. I don’t know of any easy and universal cure – but there is definitely something that should be cured. Maybe the scientists should be divided to positive and negative ones who would be asking for separate grants etc. 😉 The negative ones would be systematically looking for errors/problems in the positive papers.
Some comments about the Hwang affair (plus evolution):
http://motls.blogspot.com/2005/12/evolution-and-genome.html
I agree with Lubos: a large part of the problem is that the reward structure for academics is defective. Career advancement for academic scientists seems to depend mostly on publications, but the things you would do to get publications are not always the things that are best for advancing human knowledge. For instance, most fields need people to do lots of relatively boring measurements (eg., nuclear reaction cross section measurements for nuclear astrophysics, tree ring sampling for paleoclimate) which have a low publication/year payoff. Lubos is (as usual) completely right about the value of “negative work”, but I suspect it doesn’t get you as much publications or grants as the positive stuff.
Another factor is the increasing numbers of academics, all desparately seeking publication. The old-fashioned system of peer-reviewed printed journals is becoming overloaded. Presumably we will move to distributing research via the Internet, with some form of filtering (including peer review or something like it — a ratings system?) and aggregation paid for by subscription fees or grants (or even advertising?) It would be a smart move to make “auditability” part of this.
Re #3
There’s a further problem: the star system that has developed whereby the author of a lauded paper becomes a referee for other papers including those that may criticize or contradict that star author’s paper.
This is what happened to Michael Mann. As John Daly pointed out:
There ain’t no Nobel Prize for Skepticism
Lubos,
I’m referring to your comments on Evolution and the Genome.
In fact, it took us over a hundred years to counter Auguste Comte’s remark that the chemical composition of the Sun would forever remain a mystery, assuming that we want element abundances accurate to a few percent. We needed the development of the laser in order to get sufficiently accurate transition probabilities for spectral lines measured in the laboratory. Prior to that, we had inconsistencies on the solar iron abundances as determined for the corona and photosphere. We also had inconsistencies between the abundances measured in meteorites and those measured in the photosphere.
The strength of a spectral line depends on the ratio of the line absorption coefficient to the continuous absorption coefficient. In cool stars, like the Sun, H minus (a proton and two electrons) is a significnat absorber – that had to be calculated to an accuracy of about 1%.
Finally, the helium abundance of the Sun is found in a roundabout way. We make models of the solar interior, adjusting the hydrogen/helium ratio until the Sun has its present day radius at the age of the Earth, determined by Brent Dalrymple to be about 4.55 billion years.
Dear Chris,
you are of course completely right that the people with positive career plans do focus and have to focus on “positive work” which involves a large number of generated articles – but it’s partly because of the checks and balances that are set to this “standard” today.
(Indeed, if you care about me, I never cared about the career and considered the word “careerist” to be an insult – especially because when I was a kid, it was typically connected with collaboration with the Communist Party – but I also know various people whose opinion is superficial identical who are really obnoxious.) 😉
I also believe in “positive work” in the sense that it should be answering particular problems, instead of just being generally bitter about the activities of others. But answers to particular problems can still be “yes” or “no”. Some of the biggest science revolutions gave the “no” answers to many hopes and alternative claims. In some contexts, the “yes” answers are oversold – it was definitely the case of the stem cell breakthroughs. There are probably contexts in which the “no” answers are oversold, too.
In some sense, I believe that in the long term, the average price of the answer will be realistic. To achieve this point, however, some scandals may be needed. From the viewpoint of eternity, such scandals are a healthy thing.
All the best
Lubos
Dear Roger,
thanks for your clarifications. (Sorry, John and Steve, for replying to a slightly off-topic comment with a similar off-topic comment.) I am not aware that Auguste Comte wanted a one-percent accuracy for the abundances. In fact, his quote was qualitative and it is the following:
“To attain a true idea of the nature and composition of this science [astronomy], it is indispensable…to mark the boundaries of the positive knowledge that we are able to gain of the stars….We can never by any means investigate their chemical composition.”
I mostly agree with your description of the factors that affect the intensity of the observed spectral lines. Do you agree that today, we can calculate all these things, at least in principle, and having computers, even numerically? Even without these current observations, people could have done reasonable guesses and compare the spectral lines seen in the Sun with the spectral lines observed in the lab – and they could even get some numbers by this comparison.
The ratios of stimulated emission and absorption etc. have been related around 1920. Quantum mechanics allowed to calculate the frequencies in the mid 1920s and the methods to see the transition rates were discovered abruptly, in the 1930s. And I hope that my students will be ready to calculate them at the final 1/19 exam better than at the midterm! 🙂
All the best
Lubos
The pressures to publish that can lead to iffy work would seem to apply much more to young researchers than to older researchers. After all, tenure is a tough nut to crack and you need to grab people’s attention early with something snazzy and new.
In the climate debate, is there any evidence that the work by the younger researchers tends to overstate results more than work by more senior researchers? Do the skeptics tend to be more senior researchers who fell they can comment freely on what they see without fear of the consequences to their reputations or tenure?
Re: #8
This was Michael Crichton’s comment about the suppression and censorship happening in climate science:
After considering two other scientific consensuses where science itself was traduced by politics, eugenics and Lysenkoism, Crichton makes the following observation:
– State of Fear, p 579
Re #2 Luboà…⟬
As far as a researcher allowing the chips fall where they may, I see what you mean, but “positive work” by it’s nature, often contains an infinity of negated “negative work”, whereas “negative work” has no such expansive property. Thus “positive work” can contain more information and can be more valuable. If journal space is limited*, “positive work” should get the greater volume of paper. If the “negative work” is one of a limited number solutions it is correspondingly more important.
* Is journal space growing faster than the ability to write valuable articles :)?
oops, infinity of “negative work” or perhaps plethora of “negative work” is what I meant to say in #10.
re: #12
Nice to see someone normalizing negative infinity.
Re #12 Dave,
What I’m trying to say is that one where Osama Bin Laden is, say Gujrat, Pakistan, a positive result, also describes an infinity of negative results: where Osama is not. Obviously finding that Osama is not in Zhob, Pakistan, a negative result, is not so valuable.
Now if we had it down to a few places, striking Zhob off the list might be a big deal.
I’m not as bright as Hofstadter. Langsam bitte?
re #13
Well I know what you mean, but it’s always possible to to state anything in a positive way, e.g.: “Is Osama Bin Laden absent from Zhob, Pakistan?” and you get a positive finding.
And even if we assume that some particular findings are interesting, many aren’t. Finding any number of particuar particles traveling less than the speed of light would be of little interest, but finding one traveling faster than the speed of light would be of surpassing great interest. OTOH, particular findings can sometimes be the source of problems. Finding a cluster of a particular cancer in a small town can lead to much wasted effort to explain it, and calm the townspeople, whereas the actual solution may simply be that it was blind chance aided and abetted by an overzealous researcher.
The notion that the auditing of key scientific articles by persons of ability is an unproductive use of time is nuts. It is a critical necessity. Steve is doing a great public service at some cost to himself.
Don’t read me wrong. I probably just don’t understand what Luboà…⟠means by “negative work”. Perhaps it is a term for an audit in the scientific community.
I think “negative work” commonly means things like testing new cuprates for superconductivity and not finding it. I would argue that you should still publish, just that your work should not be as played up as if you had found something useful or surprising. You are still adding information in terms of ruling out a possiblity. In addition, there may be some positive results hidden in the generally negative result. IOW, while prospecting for new High Tc, you found some new crystal structures. I think important to catalogue such stuff so it does not have to be repeated. Not publishing is a waste of taxpayer dollars. Also, your studies may be useful for something unrelated to what you were going after. In addition, publishing some negative results may get you to think more deeply about the issues…to move to something that does start to build a basis for understanding the issue in a better method than those researchers who just chase the headlines. I have seen this and can explain.
Among prominent “negative work” would be Lord Rayleigh’s “much puzzled” results, publication of which led to the discovery of Argon. Also, the Michelson-Morley negative result comes to mind as a biggie.
Re #17
I think the most obvious form of negative work would be Weiner, Le Gros Clark, and Oakley for exposing Piltdown Man as a fake.
Michelson-Morley is the perfect example. What they did was share all the data including the strange parts rather than glossing over it, skewing it, redoing the experiment til it came out “right” etc. They did not have the correct interpretation, but because they did their work with utmost integrity and candor, someone was able to use their work later to get the right interpretation.
This is exactly what Wilson urges in his famous text. And Steve still needs to read the book. Heck I’ll buy him an Amazon copy…)
And Michelson did it at the Naval Academy. BEAT ARMY!
re: #16
Well, I think that in chemistry much of this ‘negative’ info is quite publishable since all you need to do is title it something like “Survey of Superconductivity in layered mixed Copper-Barium-Lithium oxides”. Since in such a case people will expect tables of various findings, and not some new thing there will be no taint of the negative about it. The same thing could be done in Climate Science if people were told to do it. Publish an article like “Survey of Bristlecone Pine cores on Bald Mountain” listing the location, age, and so forth of each tree and all the ring widths, densities, etc. with the details stored in files on-line. Then publish an article (probably at the same time) like, “Bald Mountain Pines: Climate Change implications” and store the meteorological data used to extract climate indications in other files. Finally the same people or others can gather up a bunch of such Proxies and do the sort of thing MBH tried to do without hiding their methods or the proxies used and when someone like Steve comes along, he wouldn’t even have to send an e-mail to be able to reproduce the findings. I’m sure that’s what he thought would be possible, and it’s what should be made possible ASAP.
crystallographic data is much better handled than paleoclimate data. this despite the huge amounts of funding going into paleo.
Re #7 by Lobos.
One person has spent many years producing an extensive list of data for atomic lines seen in the spectra of stars. The agreement between the solar spectrum calculated using his data gave much poorer agreement with the observed spectrum than did one using the latest laboratory data.
We now get results for the ratios of element abundances in the Sun which are much closer to the meteoritic results. This is all relatively recent work, it certainly could not have been done in the 1920’s or 1930’s. One reason for this is that the solar lines are senn against a “continuous spectrum”.
This continuous spectrum, given in apostrophes because the line spectrum of the Sun is so rich in many wavelength intervals, ia not that of a black body but is affected by absorption by atomic hydrogen and the negative hydrogen ion. The continuous absorption coefficients of the latter weren’t known very well untill about 1980 for free-free absorption.
Redaers may feel that I’m belaboring this point in a response to Lubos but are there comparisons of water vapor and CO2 and CH4 line absorption coefficients with experimental data? Or with telluric H2O vapor lines seen in the solar spectrum?
Roger
The 24 December 2005 issue of New Scientist has an editorial entitled “Breach of trust: events in South Korea have highlighted a deep-seated problem”. The two closing paragraphs make especially important points.