Geoffrey Boulton, who did an execrable job on the Muir Russell “inquiry”, has written a good editorial in Nature here reporting on the recent Royal Society report that he chaired.
There have been a number of reports over the years, urging improved data archiving, and yet the problems persist. Boulton’s report is merely one more. Whether it will have an impact when past reports have failed remains to be changed. In the U.S., there are quite sensible high-level senior policies on data archiving, but these are flouted in paleoclimate by the relevant NSF division. The AGU has sensible policies, but these are ignored by editors and journalists. In the past, as evidenced in Climategate emails, members of the climate “community” have sneered at my efforts to ask AGU editors to enforce these policies, confident in the solidarity of the editor, and such efforts have proved fruitless.
Boulton’s report and editorial merely add one more editorial, but one more editorial isn’t going to affect someone like Lonnie Thompson.
The missing link in Boulton’s report is putting teeth into such recommendations. Enough teeth that recalcitrant journals like The Holocene (which proudly has no data archiving policy) have to adopt and enforce policies and recalcitrant scientists like Lonnie Thompson have to pay attention.
I’ve observed for many years that the funding agencies, e.g. some divisions of the US. National Science Foundation, have much of the responsibility. One of the problems is that some NSF divisions have failed to ensure compliance with existing federa policy, instead leaving matters up to academic journals (which have no obligation to enforce US federal policy.)
As sometimes happens with regulatory agencies, the NSF climate agencies have become a cheerleader for the industry that they are supposed to regulate (in the sense of at least requiring compliance with data archiving). When challenged in the past, NSF administrators have said that authors have met journal standards and that was the end of the story. Similarly with IPCC, which should also have standards for articles cited in IPCC assessment reports. I challenged Susan Solomon on this in 2005 and she said that the establishment of data policies by IPCC would be interfering with academic journals.
However, both funding agencies and IPCC have different obligations than academic journals. If they wish to rely on academic journal policies, then they need to ensure that such journals have data policies that are sufficient to ensure compliance with the funding agency and/or IPCC obligations. To put some teeth on this, if the journal (e.g. The Holocene) does not have such standards, then its eligilibity for citation as a publication of record for an NSF funded project or by IPCC should be withdrawn. If The Holocene, for example, was reduced to grey literature for NSF and/or IPCC purposes, maybe editor John Mathews would pay attention.
Similarly, the funding agencies need to pay some attention to their obligations to ensure data archiving. While they enjoy being cheerleaders for the scientists that they are funding – and are entitled to take some satisfaction in their accomplishments – they cannot become so close to the science industry that they abnegate their regulatory responsibilities.
In passing, Boulton endorses a longstanding issue at Climate Audit – that scientists should archive all the data, not just the data “used” in the final calculations.
Too often, we scientists seek patterns in data that reflect our preconceived ideas. And when we do publish the data, we too frequently publish only those that support these ideas. This cherry-picking is bad practice and should stop.
For example, there is strong evidence that the partial reporting of the results of clinical trials, skewed towards those with positive outcomes, obscures relationships between cause and effect. We should publish all the data, and we should explore them not just for preconceived relationships, but also for unexpected ones. Without rigorous use and manipulation of data, science merely creates myths.
I raised this issue originally with Jacoby at Climatic Change and my requests were rebuffed. Little has changed over the years and my recent similar request with Gergis to four different journals (Journal of Climate, Climate Dynamics, Holocene and GRL) likewise had no success.
This is, of course, the issue that was at the heart of the ongoing Yamal controversy. Gavin Schmidt and Real Climate sneered at the idea that CRU should have any obligation to do anything other than what Boulton describes as “partial reporting”. Perhaps Boulton’s new report will help change perceptions on this point. It’s too bad that Boulton’s epiphany came after his participation in the Muir Russell whitewash.
Climate Audit 
33 Comments
Boulton for cover
Run rabbit, fox run rat.
Yoicks & Tally Ho!
==========
The position is noble, and one you’ve long stated
The data and methods must be in the open
But Boulton’s own dealings with truth? Complicated
A rift in protecting the Teammates? Here’s hopin’
===|==============/ Keith DeHavelle
This is really just another example of “weather-vaning”.
Many people in science/high tech simply follow the prevailing wind.
The winds shift so …
Boulton reportedly has had a good reputation amoung his students. I think he’s been hurt by the cogent critique of his role in the Russell Inquiry at CA and elsewhere and has wanted to restore his credibility. It’s not a bad motive. Whether the force of his report and Nature editorial has an impact on the more immovable objects of Climate Science remains to be seen.
“We find that there has been a consistent pattern of failing to display the proper degree of openness, both on the part of CRU scientists and on the part of the UEA”.
I’m not sure that Boulton would be so ready to see the Nature editorial (or the RS document that preceded it) as a radical change of tack so much as a more robust assertion of an earlier statement.
His personal position is neither here nor there. Rather than making unverifiable inferences about personal psychology to suggest that an editorial statement in Nature was conceived as an exercise in ****-covering, it seems more advisable to read it as a statement made with far wider professional endorsement about the public status of science; one which has been synchronised with the RS report in order to produce a practical effect.
For my part I would have thought Boulton’s likely to be more concerned about his personal credibility in the RS and among his professional peers than among folk at CA. But I also think it’s reasonable to infer that he’s genuinely concerned about the way the world he represents is seen in places like this.
I thought I made clear that, in my view, especially after hearing Boulton speak, pretty well, at the initial Royal Society consultation on open science at the Southbank Centre in June last year, the Edinburgh man has been concerned by a loss of credibility with his students. This could be a much better influence than seeking ‘credibility in the RS and among his professional peers’ – depending which peers. This is only a hunch. I wouldn’t lose sleep over it if it causes you difficulty.
Bolton performance on the Muir Russell “inquiry” was beyond poor , basic failings which would unacceptable in student essay , are far less acceptable from a person of his position and experience . Unless of course the idea always was to come out with total nonsense while ignoring ‘the nose on their faces’ in which case ‘blinder well played “
I don’t know the man, and it may indeed be true that Mr. Boulton has had “an epiphany,” as Mr. McIntyre (perhaps with tongue in cheek) intimates. But my experience is that most of us human beings–and I include myself, although I’m getting better–are far too slow to learn from experience how great the chasm can be between public figures’ high-minded words and their sordid actions.
So I’m going to guess that this apparently Damascene conversion is largely illusory.
Re: Joe Born (Jun 28 13:18),
Boulton is not going to make a lot of climatological friends by his explicit and disobliging reference to Climategate.
He could have just waffled in general about openness rather than focussing on it – and only it – as an example of the bad things that happen. I think that took a bit of backbone and will have some personal consequences for him, so I’ll cut the guy some slack.
But as Steve says, now the RS has to walk the walk not just talk the talk. At least they are talking in the right direction.
The issue of publication bias has to many medical scientists surprise been somewhat improved during the last years when regulatory bodies have made it mandatory to register your clinical trial before starting it. Many ethical approvals also make it mandatory to publish your results no matter the outcome. Some of the more influential journals (like NEJM) have also increased their publication of negative results further improving things. The pharmaceutical companies have naturally not been drivers of this development. So there is a little bit of hope for the improvement of transparency and ethics within medical science. Climate science? At the moment it seems far fetched especially since most of the regulatory elements and funding agencies are populated with people with a heavy bias towards AGW believers.
There must be a word for the sort of man who wordily upholds high standards in the abstract, in generalisations, but behaves as a heel in particular instances. In addition to the words that refer to private parts, that is
The composition of the panel that wrote this report shows that the major concern of this report are “evil” pharmaceutical companies that suppress “negative” results. Your readers may appreciate some background information. Although authorities like the FDA receive the results of every human clinical trial and every animal study performed with a new drug seeking marketing approval, scientists working at or collaborating with drug companies don’t necessarily publish the results of all such studies. During Phase II of drug development (ie after a safe dosing levels have been established in healthy volunteers), small clinical trials are run to determine what illnesses respond to what size dose of an experimental drug. These small trials may “fail” because the drug actually didn’t work for one particular problem OR because: a) the dose was too small, b) only a subset of the patients responded, the trial didn’t include enough patients to establish efficacy with p<0.05, or c) because one or two patients (who may have multiple illnesses) encountered severe problems which may or may not have been due to the drug. If/when the drug company believes a Phase II study have identified a group of patients who will benefit from a particular dose of a drug, two large Phase III studies are performed to provide conclusive evidence of efficacy (normally two independent trials with p<0.05) and to adequately characterize the statistical risk of serious effects. The FDA audits these Phase III clinical trials and makes only information relevant to the approved indication (including hearing transcripts) available at its website. Although the situation is changing, drug companies are reluctant to publish the results of undisclosed Phase II clinical trials that failed to fully meet their objectives – this could hurt sales if the drug is ever approved and gives competitors (propriety) information that was expensive to acquire. Therefore results available to the public are often cherry-picked for success by the companies that paid for the research.
Unfortunately, once a drug is approved for treating one particular illness/indication, doctors (in the US, at least) can use that drug to treat any illness ("off-label" use). Drug companies collaborate with outside researchers who want to determine an approved drug will help a new group of patients. Although drug companies are allowed to only promote/advertise the use of their drugs for approved indications, companies are continuously getting into trouble for disseminating information about off-label uses. Meanwhile, highly relevant data from unpublished failed or ambiguously clinical trials relevant to off-label use may be hiding in company files. Regulatory agencies are aware of all of this (propriety) information, but the approval process makes public only the information relevant to indications for which drug companies are seeking marketing approval at the time approval is requested. (At the FDA's website, one can see the results presented to the FDA, the FDA's own audit of those results, and transcripts of the approval meeting.
Your comment about pharmaceuticals seems apt to me. The climate “community” tends not to think of themselves in the same breath as Big Pharma, but if the shoe fits.
In Big Pharma news today;
‘Cherry-Picked … Data’
From pharmalot.com
” Evidence Grows That Celebrex Cherry Picked Celebrex Data
More than a decade ago, Pfizer officials crowed about the response to study data for its Celebrex painkiller in a way that may come back to haunt the drugmaker. “They swallowed our story, hook, line and sinker,” senior research director Samuel Zwillich wrote in an e-mail to a colleague. However, the results showed only the first six months of a year-long study and Celebrex was really no better at protecting the stomach from gastrointestinal problems than other drugs.
The e-mail, which was sent in 2000, was among thousands of pages of internal documents and depositions unsealed recently by a federal judge in a long-standing securities fraud lawsuit against Pfizer, The New York Times reports. The overall revelations are not new, but some documents do illustrate the approach too pitch the medicine that was taken by Pfizer and its colleagues at Pharmacia, which sold Celebrex and was purchased by the bigger drugmaker.
In one example: in February 2000, Pharmacia employees came up with a game plan on how they might present the findings once they were available. “Worse case: we have to attack the trial design if we do not see the results we want,” a memo read. “…If other endpoints do not deliver, we will also need to strategize on how we provide the data.” Another document proposed explaining poor results as “statistical glitches” (you can read the e-mails here; see pages 1, 5 and 9).
Meanwhile, some employees were quietly questioning whether the study had any value. In September 2000, Emilio Arbe, a Pharmacia associate medical director, wrote that “I wouldn’t feel too comfortable presenting a fudged version of the facts” and described the decision to use the limited results as “data massage.” And in May 2001, Mona Wahba, who worked on Celebrex, sent an e-mail to colleagues in which she wrote that a new analysis was “cherry-picking” six-month results.
”
and also from pharmalot.com;
” In a damning disclosure, an FDA official charges that Amylin Pharmaceuticals concealed a study that raised heart safety concerns about its Byetta diabetes drug and then hindered agency access to the data when it was discovered, TheStreet reports, citing newly released FDA documents. And later, Amylin execs purportedly lied to investors by failing to disclose that the Byetta study played a key role in an agency decision to reject the Bydureon follow-up treatment.
The Bydureon review and approval was a “long and complicated process, in part due to Amylin’s withholding of information on Byetta that FDA deemed to be important to its evaluation of the safety and effectiveness of Bydureon,” according to a January 2012 memo written by Mary Parks, division director of the FDA office that is responsible for diabetes drugs, TheStreet writes in its expose.
”
AC
It is easy to show that Celebrex is much safer than conventional NSAIDs for the stomach in animals, but relatively hard to prove in people: Doctors aren’t allowed to give enough of conventional NSAIDs to cause a high incidence of ulcers and must stop treatment at the first sign of trouble. Read the actual documents, not the excerpts provided by attorneys involved in the lawsuit.
Steve: Some of the problems I described above have been addressed proactively by some companies (Lilly, for example) and recently by governments. A databases for clinical trial data now exist, but I don’t know how well they work. The Royal Society could be harping about a problem which has already been addressed for recent and current clinical trials, but which still exists for drugs approved long ago.
http://www.clinicaltrials.gov/ct2/info/results
http://www.lillytrials.com/results/results.html
If you want to run a privately- or publicly-funded clinical trial in the US today that might be used to gain marketing approval for a drug, your clinical trial must be publicly registered before you start (indication, number of patients, data to be obtained from each patient, etc.) and results must be posted within 12 months of trial completion – even if you haven’t completed scientific publications. If climate science had similar requirements, when CRU received a public or private grant to study tree rings in Siberia and that study might be cited by the IPCC, CRU’s research plan would be posted at the tree ring database along with the date raw and processed data are required to be deposited.
The FDA does a reasonable job of making drug far more credible than climate science. Some knowledge of their standards might be useful to you. For starters, the FDA reviews plans for clinical trials (including data to be collected and statistical analysis methods) before trials are performed and their statisticians independently reanalyze all of the data. The FDA doesn’t approve a drug when a subset of patients respond to a drug; they demand efficacy (p<0.05) in expensive, independent clinical trials involving only patients expected to benefit from their drug and then limit marketing approval to that patient population. The official written information provided to doctors about a drug is negotiated with the FDA.
For Big Pharma, there are things worse than failing to get a new drug approved: 1) The lawsuits (and legal discovery) that follow when problems are discovered after a drug has obtained marketing approval. 2) Loss of credibility with agencies like the FDA that control the fate of other drugs a company has in clinical trials. Start-up companies (Little Pharma) that will go out of business if they can't get their first drug approved (or bought by Big Pharma) have more incentive to cheat or distort. Although Big Pharma bends marketing rules and certainly wants to present their drugs in the best possible light, I personally would not draw a parallel between Big Pharma clinical research and the climate science shenanigans I read about at Climate Audit.
Steve, I doubt there exists anyone with more respect for you than me. However, your comment “if the shoe fits” about big pharma is insulting. I worked in the Rx industry for 35 years and have been fortunate enough to either directly or indirectly been responsible for 12 new chemical entities that are being used to treat millions worldwide. Every pivotal trial is preceded by years of testing and planning, especially a proactive statistical analysis plan that was agreed by numerous company and academic statisticians, coded independently, and monitored by an independent data safety board. Pleeeeeese don’t lump honorable pharmaceutical scientists with “climate scientists”, at least with a broad brush.
It is far from clear that FDA efficacy testing requirements (as opposed to safety testing) play a constructive role in the first place. Aside from the agency’s obvious institutional bias (dead people from lack of drug approval is invisible while dead people from adverse drug reactions get laid at their door), there isn’t any evidence I’m aware of showing that off-label drug uses lead to worse outcomes than on-label uses. My understanding is that the majority of chemotherapy prescriptions are off-label, for example.
An interesting study on this question is http://mason.gmu.edu/~atabarro/Do%20Off-Label%2024.pdf
The study you cite is interesting, but written by economists. The authors suggest that the FDA should regulate drug safety, but not drug efficacy. An alternative interpretation of their survey data is that physicians who knew the most about drug development were most in favor of keeping the current system. When the FDA is involved in establishing efficacy, experimental and statistical shenanigans like those publicized at Climate Audit are not permitted.
Safety involves determining if the risks of a drug treatment are worth the benefits, and the benefits are only established by efficacy. Although oncology is an exception, drug safety is often determined in large, expensive Phase III clinical trials that are statistically capable of determining the risk of infrequent, but serious, side effects; and it sometimes costs little to collect efficacy data at the same time. Drugs do sometimes fail to show the expected efficacy in Phase III. A new class of drugs that increase HDL and lower LDL achieved this goal in Phase II, but recently failed to reduce the risk of heart attacks in Phase III.
Efficacy data from the published literature may be all that is required for off-label uses by specialists like oncologists and desperate patients with no alternatives. However, the FDA won’t allow drug companies to promote off-label use of approved drugs – such promotions are often targeted at general practitioners and other non-specialists who don’t have time to master the intricacies of hundreds of drugs. Other promotions are directed at patients. So small numbers of [often-desperate] patients being treated by specialists routinely get off-label treatment and large numbers of ordinary patients being treated by non-specialists usually receive drugs with FDA-approved efficacy.
If you read the transcripts from an FDA advisory meeting – where practicing non-FDA specialist physicians advise the FDA whether or not to approve a new drug – you’ll quickly recognize that the needs of patients who might be saved or harmed by new drugs are clearly represented. Compassionate-use programs sometimes allow desperate patients with no other options to receive free drugs awaiting approval.
I wonder if someone could persuade the FDA to license the use of carbon dioxide as an atmospheric treatment for hypothermia in the elderly at doses above 400ppm?
🙂
In the end, follow the money. If the public wants scientists to behave better, the public needs to address its concerns to the those funding the scientists. Not the journals, not the IPCC, not even the EPA or other administrative agencies. In the US, Congress is the place to go. By simply adding appropriate riders to the science funding, Congress can force every study out into the open. To really insure transparency, they can add a whistleblower or private enforcement provision that rewards people for coming forward to show that transparency requirements have been evaded.
In my experience, most scientists do not pay much attention to such editorials–though they should in some cases.
NSF seems to me to only care about a scientist’s work when he applies for renewal or a new grant, to see if he/she was productive on the prior grant, but then they only look at some measure of publications. The idea that they should enforce some sort of standard would never occur to them. They just throw a bag of money over the wall.
The tenor of the Nature editorial is a little clearer and stronger than that of the RS report. Could that be because of the Gergis affair?
I have added Dr Phil Jones famous remark about data openness to the Nature editorial comments – let’s see how long it stays there…. 🙂
I note that in his “good” editorial he repeats the Excel smear.
I am reminded of the grand old Yorkshire saying – “fine words butter no parsnips.”
The point that emerges from Steve’s post is that there is no shortage of high-minded policy and ethical verbiage around. It is when the rubber hits the road (or the parsnips hit the table) that the promulgators of this stuff are nowhere to be seen.
On the most recent occasion when the author of these fine words had the opportunity to butter the parsnips, he not only withheld the butter but pretended not to know that it existed. He now seems to have had a Damascene Dairy moment, but we will not know how genuine it is until he is faced with another bowl of parsnips. Root vegetable enthusiasts all over the world will be watching closely.
Didn’t Geoffrey Boulton offer to recuse himself from the Muir Painting and Decorating team? Or is my memory continuing to fail me?
Compare and contrast…
From Nature’s editorial guidelines:
… a condition of publication in a Nature journal is that authors are required to make materials, data and associated protocols promptly available to readers without preconditions.
A ‘Nature’ editorial regarding attempts by ‘denialists’ to gain access to IPCC’s process information included:
If there are benefits to the e-mail theft, one is to highlight yet again the harassment that denialists inflict on some climate-change researchers, often in the form of endless, time-consuming [FOIA] demands for information…
It seems ‘denialists’, as well as being email thieves, are personae non gratae in Nature’s editorial office.
I imagine that Geoffrey Boulton has now been crossed off Acton’s Christmas Card list. Probably Jone’s too.
The crux of the problem does rather seem to be the funding agencies. They effectively control the whole profession.
“Gavin Schmidt and Real Climate sneered at the idea that CRU should have any obligation to do anything other than what Boulton describes as “partial reporting”.”
Perhaps Mr Schmidt et al should be reminded that the word “partial” has two meanings.
“Without rigorous use and manipulation of data, science merely creates myths.”
Sounds like this guy is a guardian of science. Kudos to him.
In the current Archives of Ophthalmology
(Which deals with eye ills and biology)
Work from eight years ago
Was just marked “no go”
This was testing some new drug dosology
From the link’s described new revelation
It was pulled, based on mere allegation
That the team had not scored
An OK from from the Board
Of Ethics in this situation
It’s important, no doubt, to not fudge it
If the Ethics Board’s slow, wait or nudge it
But the impact felt here
Is smaller, it’s clear
Than climate-work’s trillion-buck budget!
===|==============/ Keith DeHavelle
2 Trackbacks
[…] ett intressant inlägg på sin blogg skriver han lite uppgivet om ännu en proklamation från ”officiellt” håll om vikten av […]
[…] https://climateaudit.org/2012/06/28/boultons-nature-editorial/ […]