The GRL leak may have been plugged up now w/ new editorial leadership there

I merely point out that full, true and plain disclosure is a humoungous amount of work. For example, MBH is a relatively simple area to audit, and it is only recently that you have got the complete listing of data and the computer programme. That doesn’t mean it is in working order, nor does it cover the steps that were implemented to get the whole programme to that final stage.

While I am appalled about the tardy disclosure in MBH, I note the severe difficulties in implementing improved audit. For example, implementing the audit standard of “GLP” in the pharmaceutical industry has been suggested to increase the cost of studies by ~50%. Even so, the more imaginative parts of the pharmaceutical industry (drug discovery) are generally excluded from this regulation; I don’t even know it is a feasible proposition. So that is a massive increase in costs, without even considering the costs of audit, or of replication, or of maintaining data archives, etc.

As a mere side note, I believe that Wakefield was a medic (and not primarily a researcher), and that he moved on from his UK position.

More substantially, the issue of how or even whether you “audit” cutting-edge science remains extremely difficult, given the scope breadth and complexity of what is there.

Regarding journal publication and advancement, you are now conflating this with publishing scare stories to push your issue up the agenda. It is a different issue. I don’t see that you are making a convincing argument here, and while I may sympathise with your general position, it doesn’t address the issue that high impact publications gives recognition and grant income.

cheers
per

Right now one of the major desiderata in climate science seems to me to be a really thorough outside “audit” of at least one of the big climate models by completely independent engineers or engineer-equivalents. This would be a pretty big project and would need some real funding – but why not?

Archiving data and methodology seems like such a simple thing to do (and it is already required in econometrics showing that it is a reasonable best-practices target) that there’s no reason not to do it. Sometimes it’s a good idea to start first with the easily implemented measures and see what they accomplish.

I always admire the caveat of “with all due respect..” which means “you’re completely wrong”

I think the full disclosure idea is inherently problematical. Sure, journals could make you sign a form requiring you to make full, true and plain disclosure, but where would you disclose all the data to the relevant level of detail ? Certainly not in paper journals, with their tight page limits. Maybe this will change with the rapid move to e-publication.

No, not what I meant. What I meant was that prior to journal submission, the data, methodology and all relevant materials (including adverse data) should be archived and made available for inspection. At first, that key information should be available for the editors and reviewers to refer to, otherwise what reviewers are doing is little more than litmus tests? Then after publication is granted, THEN full public disclosure to permit replication should be made.

The journals should insist on this as a condition of submission – why don’t they?

The funding authorities (such as the NSF) should insist on this a condition of funding – why don’t they?

The IPCC which allowed such a situation to develop where one study could bring down the entire operation, did not and still does not independently audit the studies submitted – why don’t they?

The US and UK Governments, in particular, have funded studies whose results are seriously affecting the present and future prosperity of their own nations, if not the world, and yet there is no mechanism of enforcement when scientific reports seriously mislead. Dr Andrew Wakefield, after producing an entire scare on the MMR vaccine which turned out to be false, for example, still carries on in his job. There appears no independent audit function that government can turn to to check the reliability of the science fed to it.

There is already a strong scientific ethic against bending the rules. I suggest that is one reason why this site gets so much interest. How you force the implementation of perfect ethics everywhere is however a substantive problem.

That’s correct. But when money is at stake, as in business and investments in the stock market, everyone is forced to take the issues of ethics and full disclosure very seriously. There is enforcement through laws like Sarbanes-Oxley and agencies like the SEC, to make fraud difficult to achieve and easy to prosecute. Everyone knows that when money is involved, ethics can get pushed into the background, because money corrupts, right?

Which brings me neatly to the next bit.

“de-coupling the link between journal publication and advancement”; this is verging on bizarre. Why is this even desirable?

Because the institutions that people work for, make journal publication and not quality or accuracy, the key metric that decides who gets preferment and who not. The institutions would also be working on stuff that scares people rather than on stuff that reassures. So its not in their best interests to turn around to government through the funding agencies, saying “we took your money, investigated it thoroughly and found there was little to worry about”, because the funding agency may think the institution may not be doing a good enough job. Much better to produce a report that includes such phrases as “much worse than previously thought” and “this requires much more detailed work to assess the scale of the problem”.

It’s this fundamental asymmetry that is driving science of all kinds to push the “Panic!” button rather than “Don’t Panic”.

I think there is a tendency to invent more rules to cure problems, but this will bring its own crop of problems. Science is a self-righting process, so I am quite sure that the message here is being absorbed and slowly mobilised. For me, the messages are to do with having an auditable data collection, and full and open disclosure of that information. It is immediately evident that these are essentials for the scientific process.

You are correct. Science is a self-righting process. But my reading of history is that for science to right itself usually takes a very long time. I believe once scientist described the progress of science as “one funeral at a time”.

I also think that a catastrophic failure in confidence in science will cause a resistance to science for a generation. Consider what happened in the nuclear industry after Three Mile Island: only now, a generation later, a younger generation that didn’t go through that experience is now beginning to embrace nuclear power as a solution to future energy needs.

I think the full disclosure idea is inherently problematical. Sure, journals could make you sign a form requiring you to make full, true and plain disclosure, but where would you disclose all the data to the relevant level of detail ? Certainly not in paper journals, with their tight page limits. Maybe this will change with the rapid move to e-publication.

There is already a strong scientific ethic against bending the rules. I suggest that is one reason why this site gets so much interest. How you force the implementation of perfect ethics everywhere is however a substantive problem.

“de-coupling the link between journal publication and advancement”; this is verging on bizarre. Why is this even desirable?

I think there is a tendency to invent more rules to cure problems, but this will bring its own crop of problems. Science is a self-righting process, so I am quite sure that the message here is being absorbed and slowly mobilised. For me, the messages are to do with having an auditable data collection, and full and open disclosure of that information. It is immediately evident that these are essentials for the scientific process.

yours
per

I don’t believe there’s an intellectual vacuum, so much as an ethical vacuum. In the political sphere, there are laws of conduct in regard to what can and cannot be done and by what means – with checks and balances which are independently enforced (this is what’s happening to Tom deLay, for exmaple).

In science, I think there is a vital need for proper checks and balances to be instituted to prevent poor work, overreaching claims, announcing scientific results by press conference and other such abuses. In that sense government can help by making full disclosure of data and relevant materials so as to permit replication before journal submission can even begin, as a mandatory requirement of funding. There also needs to be some cultural change in scientific bodies to arrest the current abuses of the journal publication system, in particular by de-coupling the link between journal publication and advancement. Additionally there must be a requirement to disclose conflicts of interest and to recuse oneself from even the appearance of such a conflict.

In all of this, a strong ethic against bending the rules needs to be established. I don’t know where this is going to come from, because science is by its very nature a rather fragmented intellectual enterprise. But in the realm of public policy, the need for full and plain disclosure of all relevant facts, as is currently the standard for raising finance from investors, helps reduce the possibility of fraud. In public policy, there must be full and independent audit and open review of scientific evidence, because the stakes for democracy and market economies are that much higher. In the case of funding, scientific endeavor should be done through blind trusts so that government and the private sector can fund science without compromising the integrity of the research or the reputations of the scientists involved.

I don’t know where all of this is going, but the incentives to cheat are too great and the incentives for ethical behavior are too poor. In that regard there must be detection, audit, enforcement and disqualification as a final resort, to arrest the abuses of the scientific method.

In a private economy, such an official would be provided by philosophy. Sadly, today’s university departments of philosophy are mired in a swamp of irrationality thanks to guys like Hume and Kant et al. Nevertheless, they are the ones making the rules by which you gentlemen are supposed to discover truth. “You can’t be certain of anything”,”reality is an illusion”, “reason is flawed”, “what’s true today won’t be tomorrow” they tell us. Is it any wonder science is suffering?

Under such an anti-intellectual barrage, it becomes clear to me at least, that the late philosopher Ayn Rand was right on in her essay “The Establishing of an Establishment” when she wrote: “Governmental encouragement does not order men to believe that the false is true: it merely makes them indifferent to the issue of truth or falsehood.” She’s right you know.

So that is what science is facing today. Not some powerfull evil bent on destroying it, but an intellectual vacuum of irrelevance. I don’t know how to go about filling it but I think reducing government funding of science would be a good goal. Perhaps the next time you testify before congress you could not suggest but demand some of the ideas on this post be adopted. Science doesn’t have any rules officials right now except,maybe, you.