It has been brought to my attention that Science has formal policies on data archiving. The author of the email, who requested confidentiality, argued that this disproved my statement:
Having acknowledged that, the underlying issue is that Science does not seem to either have policies that require authors to archive data or administration practices that ensure that their policies are applied. Since NSF then relies ( a reliance which seems to me to be an abdication of their own separate responsibilities) on journals like Science, with either inadequate policy or inadequate administration, there’s a knock-on effect.
Given that Thompson had not archived anything on Dunde, Guliya or Dasuopu until I started pressing last year, I fail to see how the existence of stated policies at Science contradicts my claim. For unique data sets like Thompson’s (for the collection of which Thompson has been justly praised), nothing but complete archiving of all sample measurements will be adequate. Problems with archiving with climate articles published in Science by Cook have been reported here before; Hans Erren has experienced problems with Luterbacher as well.
For what it’s worh, Science’s data archiving policies say the following here:
When a paper is accepted for publication in Science, it is understood that: * Any reasonable request for materials, methods, or data necessary to verify the conclusions of the experiments reported must be honored.
Elsewhere, they say:
Science supports the efforts of databases that aggregate published data for the use of the scientific community. Therefore, before publication, large data sets (including microarray data, protein or DNA sequences, and atomic coordinates or electron microscopy maps for macromolecular structures) must be deposited in an approved database and an accession number provided for inclusion in the published paper. Large data sets with no appropriate approved repository must be housed as supporting online material at Science, or when this is not possible, on the author’s web site, provided a copy of the data is held in escrow at Science to ensure availability to readers. For answers to questions regarding allowable supporting online material, please see our guidelines; further questions can be directed to Stewart Wills, Online Editor.
Finally their FAQ for authors says:
What’s the policy on publication of data sets? As a condition of publication, authors must agree to honor any reasonable request for materials and methods necessary to verify the conclusions of experiments reported, and must also agree to make the data upon which the study rests available to the scientific community in some form for purposes of verification and replication. As a practical matter, for large data sets such as DNA and protein sequences, microarray data, and crystal structures, this generally means deposition of the data before publication in an approved public database such as GenBank, SwissPROT, or PDB, with the accession numbers provided for inclusion in the published paper. (Coordinates must be released at the time of publication.) A list of acceptable databases can be found here. Other data needed to demonstrate or support conclusions of a paper can be posted on Science Online as supporting online material, where they can be viewed free of charge by all visitors to the site.
I guess the problem pertains to administration rather than the lack of stated policies. Pretty policies mean nothing if you either can’t or don’t administer them. If Science is unable to administer as simple a protocol as the above policy on data archiving, I hope that they refrain from criticizing other people’s attempts to administer more complicated systems until they get their own house in order. If they write any more letters to Barton or hysterical editorials, I hope that there is an accompanying letter stating how they plan to get their own mess in order.
Climate Audit 
25 Comments
Typo alert: In your 2nd paragraph above (beginning “Given that Thompson…”) I think you meant to write “…nothing *but* complete archiving…” instead of “…nothing like…”.
I agree with the fundamental desire for archiving data behind your post, but feel you would actually do more (vice having something to complain about) if you spoke in a more temperate manner. In an idea owrld, IT shouldn’t matter. However, in reality, scientists and scientific gatekeepers (in general) are twidgety types, put off by rough banter. You might also for a second remember that in addition to all the silly academic politics and people publishgint to get ahead, the jouirnals do often do some good things and push things to higher standards then otherwise. IOW, act as if you expect behavior in accordance with valued and refer to them calmly.
On the topic of data saving and philosophy, take a look at the book, On Scientific Research by Wilson. It has lots of good philosophy relevant to the current brouhaha. (And mostly on your side…except for the intermperate attitude.) WRT data, he makes the interesting point that original data should be saved (and in many cases PUBLISHED) because when errors are found later in the work, all is not lost. Someone can still come in and get something useful out of the experimental work done and use it for future analyses. An example might be recording a bulk resistivity vice the resistance (or even voltage drop) that is the fundamental sample measurement. Appealing to things like this famous book, will help you to actually accomplish things. Byt appealing to the highest ideals…and as expressed by someone who is a famous part of the inner preisthood.
These are a bunch of college professors. not burly mining engineers, fighting off small Bedouin bandit armies as they spud in…or keeping roughnecks from raping the 12 year old townies daughter.
TCO, I take your point. However, I’ve been pretty much completely embargoed by the Hockey Team since my first criticism of Mann. So playing nice wouldn’t make any difference at this point (or for a long time). The embargo started long before climateaudit. At this point, the only way that I can get data is through requiring compliance with archiving obligations. Will a harder-line approach get results? I guess time will tell. With non-Hockey Team people, I’ve had some cordial emails when I suggest that they archive any non-archived data. In most cases, the authors promptly archived their data (e.g. Kameda, Hughen.) It’s the Hockey Team that’s the core of the problem.
I have no problem with upping the ante and screaming louder for the data (when you’ve exhausted the nice route). Bang the table…fine (for the data). But the snide comments won’t help you. If anything, they smack of defeatism. Or they give the impression that you’re more interested in the little tit for tat personal stuff than in actually pushing for understanding.
BTW, the whole field seems pretty pathetic. If someone in crystallography tried this kind if crap (my secret data crap), they would get their ass ripped apart. (It’s ok for me to say that…I’m a burly towny’s 12 yo daughter defender.)
And you still have a homework assignment to get the Wilson book. Can use interlibrary loan…
TCO, I’m interested in processes. What is the mechanism whereby this ass-ripping takes place ? Is it just peer-reviewed literature ? In a general magazine like Science or a specialised crystallographic publication ?
Science and Nature tend to have a lot of flawed papers because of how high on the food chain they are and because of the emphasis on groundbreaking results. (and that’s in normal physical science fields…in fields like ecology or temp proxy or such that are way more complicated it’s likely to be even worse.) Basically people rush to get stuff out and overblow results…and you get bad papers (it’s usually a very subtle or slight form of fraud that has a lot to do with self-promotion and deception.)
It used to be that reconstructions were done by others in specialty journals (Acta Cryst C). There was someone who became famous for doing hundreds of them (his name became a verb like Borking). Finally, he was prevailed upon to contact authors and ask them to publish their own corrections (with acknowledgement of him). This is generally considered the civilized society way of doing corrections btw. Of course if an author refused or disagreed, one would be very much justified in publishing the reconstruction. While the whole issue is embarrasing and has some human dynamics it has led to a higher standard in the work of crystallographers and in the peer review of their articles. BTW, crystallography is a feild where one can easily make mistakes, where it is rather abstract. Maybe not as bad as statistics in sociology or climate science. But not like doing a 1/T plot for reaction rate either. It is also an area where people look at all kinds of telltale signs (“thermal paramaters”) that are like statistical tests. Also, where peer reviewers will (sometimes) check the actual math/computer work even by doing a reconstruction with original data (i.e. reworking the problem). however, they don’t always do it and it depends on the journal or the individual how tough they are. But in general it is more hands on review than the rest of chemistry.
But not sharing the data is a no no. You are required to dump it at a database that has public access. Cambridge International structure thingie something or other. Might have to pay to use it…but any company or research uni has a subscription. It’s basically like climatology is supposed to be. you don’t archive the data, they don’t publish your papers. It’s just expected.
Science and Nature are not as bad as Appl Phys. Lett. though. I’ve taken to calling that Applied Marketing Letters.
TCO, I get more prone to being snide when there’s been some kind of public trashing e.g. the ES&T article. I realize that this is not an excuse, but I offer it as an explanation. When I look at the article above, it’s only the last paragraph that’s snide. But some of these points are pretty hard to make without being snide. I really don’t think that Science should be pontificating on data issues when they are one of the major problems. It’s not like I wrote about Holocene or Quaternary Science Reviews, who are minding their own business. Science stepped into the fray.
No sweat. You are doing way better than I would. I admire all that you’ve done. Still…I think there can be some tendancy to defeatism in being too impassioned in the back and forth on some of this stuff. You’re in the right on the stats (I think) and on the philosophy of science (I know). And should just camly keep pushing forward. I’m not saying to stop pushing your points or to assume that the system will take care of you…but better to keep calm (I could imagine it being incredibly wearying over time, but still…), but at same time, no reason to put yourself in box of being a rabble-rouser.
And why the name change?
There is an admin account as ClimateAudit which I was using to put categories on old posts. I hadn’t noticed that I was still in the admin account.
Hey…I have a comment on the Annan-related post (not sure which one it is…) I liked his discussion of Bayesian estimation (bets) for global warming. And that it summarizes the growing consensus that man is warming the world. (Now…that does not mean it is catastrophic…I wish it would get warmer…I want alligators in Virginia. global warming is so slow!!!)
BTW, if we had a time machine, we could take Bayesian bets on the MWP. I wonder if Mann would win the auction of McMc?
TCO, #7
So this one guy became in effect the crystallography auditor, and doubtless kept the rest of the field on their toes. I hope he isn’t the only one doing it.
What you describe is pretty much how one assumes science properly works. You have to wonder what has gone wrong in the case of climatology, and what it will take to get it back on track.
Re: #15
I think this will happen one journal at a time and one scientist at a time. I think that in questions of science, it’s astonishing to me that the response to Steve’s detailed mathematical work has been to denounce his work as heretical and to write embarassingly one sided articles praising Mann for his fortitude in the face of “adversity”.
Mann saying that Steve’s work is garbage is not an adequate response.
I guess so. It’s just that that sounds like a terribly slow process. Particularly journals – from what Steve has been finding, it sounds like they are going to need some fairly senior level changes before they start doing the job properly.
John, you’re in the UK, aren’t you ? You know that ridiculous Carbon Trust advert that tells us we are all in the same position as Robert Oppenheimer ? If I see that damn thing any more, I swear I’m going to find out what an apoplectic fit is, from the inside.
Re: #15 “You have to wonder what has gone wrong in the case of climatology”
There is vastly more money for the climate science field selling the AGW hyposthesis. This is transparently obvious to me. It’s not only capitalist corporations that seek to maximize income. Everyone seeks to maximize income and resources and climate scientists are no different. Anyone have a better explanation?
Re #18. So oil compainies work the same? And would therefore try to stop anything threatening them maximising their income? So they would try just have their version of the truth about AGW being told? “Oh no, they’re different” you’ll cry? OK, do you have a better explaination for all the monies ( http://www.exxonsecrets.org/ ) being put towards selling the ‘AGW no problem’ hypothesis?
Re #16
But, it seems, judging by your many acerbic posts, the reverse is? LOL.
Re: #19 “OK, do you have a better explaination for all the monies ( http://www.exxonsecrets.org/ ) being put towards selling the “AGW no problem’ hypothesis?”
I think we all agree what Big Oil motives are. What we don’t agree on is what is the motive for AGW scientists not adhering to time honored scientific principles of transparency and peer review. It is because of the weakness of the AGW hypothesis. But why stick to a weak hypothesis? The climate science financing bubble will burst if AGW is discredited.
Re #19 and #20: Actually, the motives of “big oil” are a little more nuanced that you two seem to believe. The oil companies (like BP and Shell) with relatively more natural gas reserves are supporting the AGW hypothesis. All energy economists I know think that natural gas is the big short-term gainer from CO2 emission taxes. There also are other corporations on the pro-AGW hypothesis side, particularly the nuclear industry, which early on saw AGW as its “trump card” to counter environmentalist opposition — recent examples being GE (which makes nuclear reactors) and Duke Energy (which I think is the largest generator of nuclear electricity in the US and wants to build more nuclear power plants). The nuclear industry in Japan is also a big supporter of the AGW hypothesis.
Yet again Hearnden wheels out the propaganda. The site “exxonsecrets.org” is a classic example of the game “Six Degrees” in which everything is related to everything else by six degrees.
Thus Steve McIntyre is on there because he’s written papers with Ross McKitrick who is an unpaid associate of the Frasier Institute which received money from Exxon (some trivial amount like $100,000) three years ago. Thus I am also infected with the dread Exxon disease since I setup the weblog for Steve McIntyre….and so on.
It’s a trick called “Poisoning the Well” a pre-emptive ad hominem attack on someone because YOU CAN’T FAULT THEIR CLAIMS ON A SCIENTIFIC BASIS. If you care to calculate if Exxon’s money was distributed equally, you’ll find that Steve has somehow been turned to “climate skepticism” to the tune of a few pennies. Believe that? Of course you do. It’s easier than proving Steve is wrong, or even that “the climate will heat up by 1,2,3,4,5,6,7, 8 or more degrees in the next century which you decline to prove either.
The top 16 environmental organization raked in last year over $2 billion. You’d think with all that money they’s be able to buy a single skeptic like Richard Lindzen or Patrick Michaels.
This would be difficult enough to take were it not for the fact that you are paid a large amount from the EU to not produce food and even the fossil fuels that you use to not produce them are subsidized by UK taxpayers like me through the “red diesel” scheme. I don’t see you retiring the tractor and breaking in the oxen for the plough, so perhaps you’re on the take from Big Oil yourself.
You have yet to show that AGW is detectable let alone significant. Instead you spend too much time on a computer telling everyone else your crazy beliefs without bothering with so much as a calculator.
It’s one thing if Peter Hearndon is retailing Environmenal Defense Fund piffle about exxonsecrets.org, but it is singularly demeaing to Mann that he should have sent Environmental Defense Fund piffle to journals to try to block publication of our material, together with slanderous allegations of “dishonesty”.
Enron was a big supporter of AGW. That doesn’t disprove AGW.
The same could be done for RealClimate which was set up by Environmental Media Services. They are in turn connected to a long list of environmental groups whose lifeblood (read donations) rely on scary scenarios.
One Trackback
[…] discussed Science in particular in a number of posts, including the following Science Editorial, Science Editorial #2, Letter re Esper, Kilimanjaro Data and others. To date, Science’s performance in ensuring […]