Yesterday, in a passing comment, I mentioned an article by Osborn et al, Annually resolved patterns of summer temperature over the Northern Hemisphere since AD 1400 from a tree-ring-density network, as an example of abuse of terms like “rigorous” or “conservative” to arm-wave through proper methodological description. Here’s an example of their use of “rigorous”:
Such infilling is only accepted if it passes a rigorous verification procedure.
But what is the “rigorous verification procedure”? People in the field – be they editors or reviewers – don’t even notice such sentences. That’s the point of the Riemann Hypothesis article. Imagine the consternation if Li had merely said on page 29 that he used a “rigorous” infilling procedure to go from ideles to adels. He would have been laughed in scorn out of the mathematical community. In the Osborn, Briffa et al case, the exact form of the infilling procedure probably doesn’t even “matter”, but the sloppiness of thought and expression grates on me.
It even happens in Nature. MBH98 itself says, without any editor, reviewer or reader apparently objecting:
[RE} is a quite rigorous measure of the similarity between two variables,
This adjective is applied over and over to the RE statistic [in this case, “over and over” is not merely “countable” but “finite”], but what exactly is a “rigorous” statistic and how do we know this about the RE statistic. If one follows the lengthy discussion of the RE statistic where we’ve been active participants, it’s hard to emerge with a good impression of defenders of the position that an RE statistic above 0.0 demonstrates model validity or of the editorial process acquiescing in things like Wahl and Ammann 2007.
I’m not sure why I picked Osborn et al 2008 as an example for inclusion in my rigorous-robust list. I think that I just remembered that it was a particularly boring article and thus a pretty good candidate for use of robust, conservative, rigorous. (And no, I’m not advocating a general hypothesis of a genetic association between the use of these terms and articles being boring.)
However, after I mentioned it in a comment, I had some second thoughts because I only had a preprint and, thinking about it, I couldn’t remember when it had actually appeared. So I searched for the publication particulars and, at Tim Osborn’s list of publications, located the following under the Submitted category:
Osborn TJ, Briffa KR, Schweingruber FH and Jones PD (2008) Annually resolved patterns of summer temperature over the Northern Hemisphere since AD 1400 from a tree-ring-density network. Submitted to Global and Planetary Change.
For your edification, here is a copy of the title page of the version of this article that I have, dated June 2004. Not that I’m a particular support of Osborn, Briffa et al, but, in fairness, I think that after over 4 years, they are entitled to a reject-or-accept decision from Global and Planetary Change.
Climate Audit 
52 Comments
Next questions, then:
Has “Osborn et al 08” been used as supporting documentation in any other paper, or referenced in any IPCC presentations?
What other “Hockey Team” authors and papers were used to support “Osborn et al 08”? Any of the BCP papers used?
Does this mean that “Global and Planetary Change” is now an approved peer-reviewed climate change publication?
What are the archiving and reviewing standards for “Global and Planetary Change”?
Steve: If you’re trying to be ironic, you need a lighter touch.
Wow, and here I am getting upset when a journal takes 6 weeks to review one of our papers. 4 years, ouch.
I agree that most climate scientist don’t view validation or rigor in the same way as those that have real (as opposed to virtual) lives at stake. I have a very different view as a nuclear engineer:
Nuclear Engineer’s View of Validation
BTW, Steve, I love your work. Thanks!
I gave my car a rigorous inspection yesterday.
Four years and still “submitted” almost certainly means that the referees demanded at least one round of re-writing. Four years from submission to publication is, unfortunately, not that unusual but the paper is normally “accepted for publication” or “in press” for a large proportion of that time.
This post is lacking robustness.
Rigorously speaking, that is.
When I have had something take that long to get into print it was NEVER in the first journal I submitted it to (longest was 18 months), and probably had a different title. OR I gave up on it. It is not guaranteed that a scientist’s bio page is accurately updated.
After four years one may want to consider rigor mortis.
Steve:
You naughty boy! My sides are still aching from you Riemann post.
Now I have to chuckle again at the IPCC using non-published papers to refute “peer reviewed” studies.
…and they do it with a straight face…
Perhaps rigorous climate science is like rigorous sex… a lot of huffing and puffing leading to a premature conclusion?
It is in my conservative opinion that this article is rigorous in it’s robustness.
Osborn et al (submitted) need a rigourous-robust rejection from Global and Planetary Change.
I think they’ve been “ro-busted”:
http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6VF0-49G5SBP-1&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=c18b02ff3e6581bdf05fd533c6321580
(Article taken from “Global and Planetary Change” Volume 40, Issues 1-2, January 2004, Pages 11-26 via Science Direct.)
I suppose it could be called “robust”, they managed to drop a co-author and keep the same results.
Either that, or “Global and Planetary Change” needed to have data archived, and one co-author (Jones) couldn’t accept that…
Don’t know if I needed to set the “irony” flag for this one…
11: Okay, I laughed.
Article is paywalled, but a thumbnail to a chart shows some of the history:
Nothing to see here, move along…
Two self-citations and one external. Each has been cited further.
K. R. Briffa, T. J. Osborn and F. H. Schweingruber (2004): Large-scale temperature inferences from tree rings: a review. Global and Planetary Change Volume 40, Issues 1-2, January 2004, Pages 11-26
Matthews, J.A. and Briffa, K.R., (2005): The ‘Little Ice Age’: reevaluation of an evolving concept. Geogr. Ann., 87 A (1): 17–36.
R. D’Arrigo, R. Wilson, B. Liepert, P. Cherubini (2008): On the “Divergence Problem” in Northern forests: a review of the tree-ring evidence and possible causes. Global And Planetary Change, 60 (2008) 289 – 305.
Papers can be take a long time to be published. Ammann and Wahl 2007: The importance of the geophysical context in statistical evaluations of climate reconstruction procedures, was apparently first submitted to Climatic Change in 2000. But nobody seems to have cited it before 2007.
#16 #17 henry and Henry are different
Why can’t these guys just say it. It passes or the AQL is like 99.9% or something like that. If I went to my customers and said the units passed a “rigorous” testing they will say “what is the AQL?” I’m sure most here would agree, since I come from a manufacturing background, that if your AQL is more than a few percentage points or more its a failure, PERIOD. If your are on a aircraft with a acceptance quality level of 95% would you fly? I think not. I have been reading Steve’s post here and while statistically I’m am mediocre, I still get the jest of the posts. Sorry Steve if this seems like a rant.
#19. I’m not saying that their procedure is a bad procedure, I can hardly say that since I don’t know what it is other – other than it being “rigorous”.
Thank god, I thought that maybe my evil side was coming out.
So do we stick with big h, small h; try henry1 and henry2; or go to different handles altogether? I’m using henry (small h) on quite a few blogs. As a matter of fact, using henry (small h) on Open Mind is an automatic delete. He just doesn’t like to answer questions about his “master”…
Re #19:
To be fair, you don’t want to make science the same as engineering. Scientists are trying to understand mysteries, and they NEVER get the answer completely right (or even partly right many times). What you can expect is transparency, good peer review, and a clear acknowledgment of issues that cannot currently be nailed down. And most importantly, a willingness to discuss critiques and reject conclusions based on new data or analysis. This is how most scientific disciplines work.
#22
I agree that science is more apt not to be as precise as engineering. They should however be held to the same quality standards like ISO. A set of predetermined and documented procedures for the discipline. You don’t have to be exactly precise but you have to explain you findings and not say our SKILL says so and a bunch of like minded fellows say so too which makes it “Peer Reviewed”. All things must have documented processes and not colorful language to cover the papers short comings. Like you say some transparency would be nice along with the data on how they got there.
What do you think engineers do? Do you think that building a rocket to go to Mars is not about “understanding mysteries”
Engineers use art to produce goods and services. They use science and technology to accomplish that. For example, the nuclear fellow above describes a model that can predict an outcome to within 20% of the actuality. This is perfectly acceptable for engineers (I are one) as I would now make my gizmo only three times stronger than it needs to be, rather than ten times stronger.
Ancient Roman engineers used to build a full size model. If it fell down, they built a bigger one. Bloody minded buggers. The science has advanced and now we don’t have to do that anymore. The more accurate and repeatable the model’s prediction the narrower the design factor.
Engineers are also held accountable and responsible for the results. Some get rich, some kill themselves, and most get on with life and earn a living. Scientists in general, and it seems climate scientists in particular, are not held accountable.
“Rigorous” means “we wrote it down in pen instead of pencil.”
As a usual lurker, there are some relevant differences between pure scientists and engineers that I think may clarify the ongoing culture conflict at this blog (I apologize for the length):
1) The modal pure scientist is in a race to get credit for making discoveries before other people do. Whether he or she has in fact done so is generally decided by a peer group. Replication is one important way in which results are validated but it is often neither necessary or sufficient to convince the community (at least provisionally). Many experiments are very hard to replicate without lots of tacit knowledge, and a failure to replicate may just be a failure of technique. Questions about technique are therefore often answered by authority–whoever is believed to be most skilled in the appropriate technique for a given experiment can expect to have his or her assertions believed about why someone else failed to replicate a result. This dispensation does not apply to mathematical derivations, however, because those can be directly checked and replicated or not. Statistical analyses seem to fall in an intermediate gray area. Classically, the main consumer of pure scientific output was other pure scientists who used this output in their own investigations, hence having the strongest possible motives to use sincere and careful judgment about which results to believe. Those judgments directly affected the scientists’ chances for making discoveries in the future.
Under these circumstances, strict procedures for data archiving and methods disclosure are not that important. Scientists will supply the level of disclosure that optimally balances credibility with convenience. Those who under-disclose will find their results ignored more often; those who over-disclose will incur excessive costs in time and money and will find themselves falling behind in the race to discovery. An experiment need only work enough of the time to generate results that constitute the discovery, provided that it can be documented when it is working properly and when it is not. Excessive fussiness about making apparatus reliable and phenomena reproducible on demand only hinders the race to discovery. The same may be true for the degree of detail in a theoretical derivation–something with lacuna that still points to the right answer may be all that is necessary to claim credit for a discovery (“proofs” are important but not critical outside of mathematics proper).
2) Engineers are responsible for implementing technology to solve specific physical problems. Their solutions are expected to work reliably under a range of conditions, and failures tend to have large and expensive consequences for third parties. Hence they (and their various clients) are reluctant to take things on faith, even when that is faith in the expertise of other people. Many engineering solutions will be operated and maintained by persons other than the original developers, so there has to be a great deal of visibility about exactly what was done and why. Public credit for engineering achievements is not the overriding incentive for most engineers most of the time, so data hoarding for non-proprietary reasons is counterproductive. Elaborate testing and wide access to all data and models used is a form of protection for everyone, including the engineers primarily responsible for design and fabrication of artifacts.
I submit that what has happened with the climate debate is that the norms and practices of pure science–which Steve and others with an engineering perspective deride as sloppy and careless–are being misapplied to what is closer to an engineering situation. Before any of these questions were policy-relevant, the norms of pure science that evolved to maximize the rate of discovery were appropriate in climate studies. But once the findings became hugely important for the lives and well-being and freedoms of everyone in the world, a shift toward the engineering norms was called for. It didn’t happen, even though some lip service was paid to practices such as data archiving and code sharing. I hope that CA and other voices can accomplish this painful transformation in working practices so that we don’t end up building gigantic new policies on a foundation only strong enough to support the satisfaction of individual curiosity.
I think srp’s comments above should be framed. This hits the nail on the head. As someone who spans purely theoretical to very applied problems myself, I think he has exactly defined the problem.
#27 srp
I echo Craig’s comments about your insights into the importance of seeking changes in the practices of climate science. In particular, I believe that you have also pointed out the exact reason for complete disclosure of data and methodology along with identifying the role that CA has tried to play in this process. (Bold mine)
I third srp’s comments, and second Craig’s. Srp nailed what I was trying to say beautifully.
Climate science is fine as science. Sucks as engineering so far.
We are still at the front of this process. First step in engineering a solution to a problem is to understand the problem. To define it and get agreement from the stakeholders that it is the actual problem. I have solved the wrong damn problem with grace and style in the past. Each time was a learning opportunity that cost me.
Srp, that comment of yours help me define why I actually get so suspicious around this climate science. I’ll be using it to explain myself going forward.
I will argue that portions of climate science are not fine as science.
As soon as an element is treated as “accepted”, it has moved from science to engineering.
The real argument being made by IPCC, AlGore, IPPR and more is “the science is settled”. As soon as that is actually true in any practical way, we’ve moved from science to engineering.
Policy involves engineering (producing real-world results) rather than science (investigating What Is).
What we have here is an agenda-driven world trying to shove very unsure science into engineered solutions.
Suppose the Army Corps of Engineers had to implement Cold Fusion today? [Someone pick a better analogy please!]
This is more expensive, longer term, but also uncertain with respect to known outcomes.
MrPete: your comments are of course what happens when we treat rush-to-be-first science as if it were engineering. Let’s try another analogy: a lab is rushing to test some new therapy for cancer in the petri dish. It works (this one time, in their lab). The science is now settled and humans should be given this therapy immediately…what? You want tests in mice and then clinical trials? You obstructionist! The gap between the first discovery and the real world is often huge. Many many patents are granted for discoveries that can never be commercialized and many cancer cures are never available for human use because they are toxic or don’t work in a living body or something.
Regardless of the hands on or off aspects of science versus engineering, there is no reason the same principles can’t be applied at least as a goal. Perhaps it’s just a matter of how transparent the processes are. When the light is turned on, the roaches run away, right?
I actually think Steve’s particular bias comes from his mining background. For decades, there was a real problem in the mining industry that severely limited the credibility of anybody trying to attract investors. So many people got burned by fradulent mining assay documents and falsified lab data, that the governance over the industry and the industry itself have become hyper sensitive to how data is collected, how it is analyzed, and what claims can be made based on the data. It is very similar now in some respects to how medical trials are regulated by the FDA.
In my eyes, the standards Steve is trying to apply to climate science are the same as those demanded in the mining industry. When asking people to make multi-million dollar decisions based on scientific data, the mining industry now demands that the data be collected, analyzed, and presented in ways that make fraud and falsification of findings very difficult. Since climate science has passed beyond the realm of a “pure science” (where assumably nothing really earth shattering is riding on the conclusions other than bragging rights at the post-conference trip to the hotel bar), and into the realm where governments are being asked to make multi-TRILLION dollar decisions based on it, I don’t think it is too much to ask for climate scientists to start being a little more diligent and lot more open about how their data is collected and evaluated.
Anyone know Pielke, Jr personally? srp’s comments look like something that ought to be featured in one of his articles.
Great job, srp, in succinctly summarizing what may be the REAL issue behind all the hoopla.
I also laud srp’s comments. But when a scientist’s conclusions are shown clearly to be based on flawed analyses and methods, then it’s time for him to admit it and get back to the lab. The stonewalling and obfuscation demonstrated by some climate scientists is not a part of proper science. He/she also has the responsibility to provide data and code.
Math & Science:
Li submits a paper. Costs time, but not his job.
He’s shown to be wrong, in one small but significant area
He withdraws the paper (NO error allowed in a math paper)
Engineering:
I’m told to solve problem X
I study, test, propose a solution
Reviewer (my boss) says: “Will you bet your job and the house you can’t yet afford on the answer?”
I go back to the lab and come back two hours later: “Yes”
Climate Science (some aspects):
Neither of these fit
“Climatic Research Unit” eh?
I wonder how many people are now employed in such establishments, all no doubt churning out paper after paper (accepted, rejected or otherwise) attempting to justify continuation of that employment.
It never ceases to amaze me that people are so quick to cast sceptics as being “in the pay of the oil companies” without realising just how much larger the pro-AGW gravy train truly is.
The standard for climate scientists, as with all other scientists, should be reproducibility – nothing more.
I would hope that Lonnie Thompson would archive all of his measured data, but it doesn’t fundamentally disturb me if he doesn’t. His experiment is reproducible: 1) I went to this location, 2) I retrieved an ice core, 3) I analyzed it, 4) this is what I found. Anybody who wants to can repeat the experiment [No Starbucks, though…].
The paleo-reconstructions are an entirely different situation, however. A lot of this stuff doesn’t appear to be reproducible. These ‘experiments’ seem to be of the form: 1) I collected no data on my own, 2) I picked a small subset of the available data – but I won’t tell you which, 3) I decided I didn’t like some of the data, so I adjusted it – but I won’t tell you by how much, 4) I used a new type of math in a secret software algorithm to combine the data – trust me, 5) I found this compelling and robust trend – the world is coming to an end.
Asking these paleo guys to provide ID and version of data, as well as what math they used to analyze the data is just forcing them to report a reproducible result. There’s no reason to force them to do more.
The standard for policy-driven / policy-affecting organizations should be higher than that for scientists, however. If an individual doesn’t report a few adverse tree ring data, that’s his moral / ethical problem to live with. Organizations should randomly census many tree locations and report everything, ‘good’ or ‘bad’.
Jim, much of the work done by climate scientists is NOT simply “collect field data and report”.
Their analysis methods are a huge part of the picture. I believe that’s true even for LT.
Thus, without the original data, nor access to their code/selection/etc criteria, it is impossible to replicate.
“Impossible to replicate”…
… particularly when we can go to the exact same tree 20 years later, take two samples a few inches apart, and get wildly differing results.
I *hope* and I *trust* that ice cores are far more replicable than that. I really do.
I think that if anything Steve is too accepting of poor quality work by the IPCC and others.
Here’s my hypothetical:
Would you put a loved one on a commercial jetliner designed or built with this level of rigor? How about even to mining or the FDA’s standards?
It’s only a hypothetical, because in this country you can’t. The level we have found necessary goes beyond even that. We add things like design assurance plans, traceable requirements, document control (all docs under version control), explicit test procedures and test standards. That’s in addition to data archiving and transparency. All of that is pre-certification.
Post-certification adds scheduled inspection, more verification, incident investigation (ala Obama’s jet) and if there’s a defect found by inspection or investigation, there’s an agency with the authority to ground every single aircraft that might be affected (like the MD-80s earlier this year). The equivalent to an FAA Airworthiness Directive would be to invalidate every single paper that cited MBH98 until its defects were corrected.
We’re not talking about that level of catastrophic failure, though. Even if every aircraft in the world of some type were to simultaneously fail catastrophically, fatalities would only reach into the hundreds of thousands.
This is more serious. This is about implementing policies that will affect every single one of us, and possible consequences (from any choice) where we’re talking about things like “percentage excess mortality”. The policies (like burning food) resulting from this may already be killing people (though they’d be very hard to count).
We’re not talking about putting one loved one on a plane. We’re talking about all of them. I really don’t see the IPCC as taking this seriously.
Mr Pete:
If this is so, then that would be a publishable result.
I’m not apologizing for any particular scientists, but I do see a difference between 1) actual data collection + analysis / data manipulation, and 2) pulling some unidentified fraction of historical data from the ether + analysis / data manipulation.
Whether any given type of proxy is good or bad, variable or consistent, if “good enough” spatial sampling conditions are described in a publication, the experiment can be reproduced.
If the experiment’s reproducible, the truth can eventually be determined.
It may be that if two ice cores are taken 30 meters apart, you get drastically different results. If so, a person who attempts to reproduce work will discover this and publish. Then the next question will be are the ice cores really different or are there big differences in the way Drs. X and Y are manipulating the data ? Those are the sort of scientific question that can be resolved in a reasonable amt of time.
If Dr. X chooses not to disclose all of his data and methods, and Dr. Y goes to the same location and reproduces the result, there’s nothing stopping Dr. Y from publishing all of the data retrieved from his ice cores + the math methods he had to use to repoduce X’s result.
There are good reasons to ID a particular tree during sampling. You might want to know, for example, if sampling ancient BCPs could lead to eventual damage to the trees. Where you and Steve re-cored some Colorado BCPs and could show directional problems with those trees – that was fantastic.
But even if the tree IDs weren’t reported, it’s enough to core lots of trees in the general area and show that a prior researcher’s data are full of a suspicious number of outliers. Why is it that a random sampling of trees in the area has a std dev of 4.2 units, but researcher X was able to find a group of trees with std dev = 0.12 units ? The obvious implication will be that X didn’t report all of his data.
Asking for more than reproducibility is insulting, in my opinion, b/c it’s a backhand way of accusing scientists of fabrication.
We save photos of particle collisions b/c it’s insane to force another scientist to build their own particle accelerator. We don’t force chemists to prove through documentation that they ran an experiment, however. If they provide a spectrum and say it’s from analysis of compound Z under a repeatably defined procedure, then we take them at their word until somebody raises questions of reproducibility. I don’t see why climatologists should be treated differently. The solution is more scientists trying to replicate experiments.
Jim, going backwards a bit through your logic:
Fine. Ice cores are expensive to collect. The government has rules requiring disclosure. We don’t enforce the rules.
Ahhh… but what if Dr Y cannot reproduce the result? Without access to X’s data and methods, we must either accept X’s result as conflicting — on faith — or we must toss X’s result. This is a science blog, so faith-topics are banned. I take it that “accept on faith” is probably not an acceptable course of action.
The question has been raised, endlessly. Look at Craig L’s work, for example. It contradicts other proxy studies. Who is correct? How can we know anything at all if those who sit on one side of the table refuse to turn over their cards?
Eventually, all the cards must be turned face up. It’s time.
To NOT do so is to act as if one has something to hide. Perhaps there’s nothing to hide, but the actions of such a one are indistinguishable from one who DOES have something to hide.
This is why sunshine is such good disinfectant.
Yes, we believe the best of everyone. “Trust but verify” is very good medicine for science.
We’re not asking for MORE than reproducibility. We are asking for that much however. It is insane that Steve must go to ridiculous lengths to reverse-engineer others’ methods and data. These are not trade secrets, they are publicly-funded scientific investigations, explicitly designed to inform and influence major investments in our planet.
Jim Edwards wrote:
In the case of the FAA processes I referred to, they’re not there to to guard against fabrication. They’re there to guard against mistake and error. The accusation is that the scientists are human. That’s not an insult. It’s just something we’ve learned, with lessons paid in blood.
The difference is that chemists aren’t saying we need to spend $45 trillion, drastically alter our lives or prosecute skeptics. “Extraordinary claims require extraordinary proof”. Those are some pretty serious claims. I recognize that not all climatologists are making these claims, but the loudest voices are. If you think it’s not fair, you’re free to take it up with Dr. Hansen.
#44, Mr. Pete:
Be careful not to conflate 2 issues [I’m not saying you are…]. If Dr Thompson is self-financing his research [Of course, he relies on grants…], then the expense of replication is irrelevant. He should have no general duty to go beyond publishing a reproducible description. The data would be his to lock in a safe or flush down the toilet. If the government pays to build a particle accelerator or finance an ice-coring expedition to Antarctica, however, then Uncle Sam should be able to enforce whatever disclosure terms were negotiated with the researcher.
This is a separate duty that has absolutely nothing to do with the expense or public policy considerations of Thompson’s work.
I hope everybody can agree that researchers must be held to their agreements to disclose data and methods [IF an agreement exists]. Please don’t hit me with the FOIA argument b/c it begs the question of “What is the gov’t buying with its $, data or services ?” A lot of grants fund exploratory work but no final paper – so I’d guess in many cases there is a bargain for effort, not results. In those cases the gov’t would have no right to data. [I’m NOT saying it’s OK to make such an agreement, just that the parties should be bound to the agreement they actually made.]
Even if there’s no agreement to disclose data, there’s nothing stopping Uncle Sam from using eminent domain to force the sale of the undisclosed data. What’s the fair market value of unused data sitting in a filing cabinet ? $1000 ?
If Dr. Y can’t reproduce the result, he should publicize that fact amongst other researchers.
The controversy would likely draw more investigation into the cores and the analysis techniques, as I stated. In the meantime Dr. X’s result would be suspect; what’s wrong with that ?
I’m glad that we agree. That’s what I said in #39:
Steve: Jim, I’ve reviewed and excerpted US federal law on data archiving. I agree with Pete’s point and do not agree that he is conflating anything. There has been a very clearly stated federal policy requiring data archiving that has been in place since 1991 that applies to recipients of NSF grants (which includes Thompson). This isn’t up to the scientists. NSF isn’t supposed to negotiate with the scientists; NSF is supposed to enforce data archiving policies put in place at a senior level. It doesn’t. I agree with Pete’s point. In this case, I’m afraid that you’re arguing a theoretical point that’s not at issue.
#45, Dishman:
Scientists don’t design airplanes. They do little experiments.
Airliners are designed by teams of engineers in consultation with MBAs, PhD economists, and the intended purchasers of the planes in order to provide a safe and cost-effective transportation platform that the manufacturer can sell.
You’re mixing up two important points, here.
1a) What requirements should law or custom place on scientists to publish the results of their experiments; 1b) should climatologists be singled out for a higher burden ?
2)What requirements should law or custom place on Al Gore, Dr. Hansen, the IPCC, or any other organization or person that attempts to scare society into passing expensive laws that are purported to be necessary to avert a planetary catastrophe ?
I will repeat, the answer to 1a / 1b is published results should be described sufficiently to allow reproduction of the result, period.
The answer to 2 should be more than good graphics, rhetoric, and hot chicks in tie-died T-shirts.
Jim Edwards said:
Anybody who has ever maintained any equipment knows 2 “rules”
Rule 1. Before you start to troubleshoot, be sure that what you have is an actual equipment malfunction. I can’t tell you the number of times I was able to chalk up the problem to “operator error”, or NTF – No Trouble Found.
Rule 2. Accurate measurements must be made, and compared to the nominal values. It doesn’t do you any good to troubleshoot if the test equipment isn’t calibrated. Also, zero must be known and common to all measurements. Only then can you see how far off you are (if any), and repairs or adjustments can be done.
I would not be able to test a radar with missing tech data, a different zero used for each measurement, data that has been “adjusted” after the measurement has been taken, etc.
And my chief of maintenance would have been fired if his comment was “Why do you want to work on the equipment? You just want to find something wrong with it”…
#46, Steve M.
1) I realize that you have studied this area of law
2) I am in complete agreement with you, Mr. Pete, and others as to what SHOULD happen re: disclosure of federally funded data and backdoor 4AR reviewer comments. I’ve stated so in the past.
3) NSF is not supposed to negotiate away the data, but what if they do so ? Saying that NSF is breaking its own rules isn’t likely to get you anywhere in a court of law, vis-a-vis the obstructive researcher.
4) I don’t believe these legal issues are so cut and dried, once the situation gets mucked up. I believe almost all primary grant applicants make disclosure agreements but the same relationship might not exist for all researchers who receive their funds indirectly through another institution. Our discussion was about general rules of disclosure for climatologists, not specific cases.
5) I argued that there is a general professional burden of production for all scientists [regardless of experimental cost], and a separate legal burden for those who accept federal money to do work. I find it hard to understand which of these points is “theoretical”. I guarantee you a judge will see it my way.
6) The professional and legal duties are clearly separable. We should expect published papers to be reproducible, even if they’re privately funded. Privately funded researchers have zero legal duty to disclose source data and code.
I’m shutting up, now.
Steve: Jim, there are layers of issues, but one of the key issues is simply placing sunshine on NSF’s failure to ensure that its grant recipients comply with existing federal policies. As you say, if NSF is negligent, it wouldn’t get anywhere in a court of law with the obstructive researcher, but my hope is that it will get somewhere in the court of public opinion and ultimately force NSF to obey the law and require compliance with US federal policies. Sure, there would still be some non-compliant researchers; it wouldn’t affect the Jan Esper and the Euros, but it would be a start.
Re srp # 27
This post and following endorsements trouble me. I appreciate that they are thoughtful, but they over-generalise. Like Steve, I’m from a mining industry background. We worked under mandatory requirements of materials storage (like drill core) and prescribed reporting standards and intervals. Full disclosure unless excepted for competitive reasons. Jail for fraud.
As a geochemist I’m classed as a scientist, but in reality the work requires interaction with engineers and preparation of work in readiness to hand a payable discovery to the mining engineers. So, the scientist has to walk the walk and talk the talk of the engineer, to a fair degree. No point handing over a deposit with incomprehensible documentation. So the gulf between science and engineering is context-sensitive. I wish people would not play it up. Really, engineers and others make aircraft that use clever materials like polymers that chemists design. I was even seconded to manage a complex engineering pilot plant that used 10 tonnes a day of chlorine gas at 1050 deg C – in a town. There is large overlap between science and engineering.
My problem is with the standards of science conducted by diverse disciplines. Having worked since the mid-60s and seen the emergence of green activism masquerading as science, green science started from a very low base and is still struggling to get recognition. Much of the climate related green science is in this class. A principal failing is lack of accountability. The wildest of claims have been made with no apparent sensitivity to their consequences. Names need not be mentioned.
The more demanding and high quality branches of science seem to attract people who face the challenge that they must succeed to ensure continuing income. Look at the very low disaster rate among the producers of chemicals, like pharmaceuticals. Their inventor scientists know that their employers will go insolvent unless they produce worthwhile goods at frequent intervals. Yet, there is an active conglomerate of green activism “scientists” trying to stop vital actions like vaccination.
But this is a bit of personal philosophy. The real game is to get full accountability, full disclosure, higher standards of proof, etc as have been the core of CA concerns.
And yes, you are right. Scientists do not get the right answer in the way that mathematicians can. Scientists should be humble enough to know that perceived wisdom is often overturned and that they should strive for a ‘best result’ but never an ‘irrefutable truth’. It is pleasant to read a top paper by a top scientist, with clarity of expression, disclosure of doubt or WIP, neat design, logical conclusions from unstressed data, etc. Few climate science papers I have read yet fit these ideals.
Jim Edwards wrote:
Dr. Hansen is an employee of NASA. NASA’s standards are very similar to the FAA’s. I’m currently investigating his compliance.
Mr. Gore is a politician.
The IPCC has its own rules. There has been some investigation posted here regarding how well they follow them. I believe they should be held to standards as I mentioned above, or disregarded. Persons participating in the IPCC are doing so by choice, and should therefore be treated in the same manner. Further, I believe the IPCC should not cite as supporting evidence studies which do not implement Quality Assurance of some kind.
Some researchers offer legal testimony as expert witnesses. I have no mercy for sloppy work there.
For the remainder, I recommend awareness of the situation. Some parties have made the consequences of error exceptionally severe, even beyond any aircraft. If nothing else, I appeal to their professionalism. Use the methods that have been found to be necessary in situations where technical errors can get people dead.
That’s not necessarily true. If I recall correctly, the EPA has been successfully sued many times for breaking its own rules. It might not affect the individual researcher, but it could affect the NSF.
Geoff Sherrington, #50:
I tried to deal with your concerns in my original post #27 by careful wordsmithing. I used the term “pure science” to distinguish the curiosity-driven tradition of Western science that long pre-dated (but has continued alongside) the practical application of research to worldly problems. Subjects such as paleoclimate reconstruction used to be–along with stellar astronomy, paleontology, linguistics, much of mathematics, etc.–matters of scholarly or “academic” interest, whose broader implications, if any, were restricted to aesthetic or philosophical implications. The people pursuing these subjects formed small and fairly isolated communities whose passions were focused on figuring stuff out and getting credit for it from the other members of their community. No third parties were on the hook for their (or the community’s) misjudgments, and the norms of their scientific fields evolved under these influences.
Now things have changed. As Bill F so aptly put it above, “Since climate science has passed beyond the realm of a “pure science” (where assumably nothing really earth shattering is riding on the conclusions other than bragging rights at the post-conference trip to the hotel bar), and into the realm where governments are being asked to make multi-TRILLION dollar decisions based on it, I don’t think it is too much to ask for climate scientists to start being a little more diligent and lot more open about how their data is collected and evaluated.” Applied science as an input to engineering or economic or military or public health problems is a different matter from pure science. Such applied science sounds more like your experience as a geochemist in the mining business–whether you are right or wrong has fairly immediate and presumably somewhat traceable consequences for other people’s money and careers.
BTW, a good if somewhat Panglossian take on the norms of pure science and their self-correcting character can be found in Michael Polanyi’s old essay The Republic of Science. A longer, and much less Panglossian, but very rigorous and eye-opening account is given by David Hull’s Science as a Process, using the field of systematic taxonomy as his example. Hull was pretty close to the events he describes and he knows where most of the bodies are buried in his chosen community. One interesting point in Hull’s description and analysis is that cutthroat prima donna behavior, some more extreme than the stuff that Steve rightly complains of in climate science, was also endemic in systematic taxonomy. Hull argues that on balance this “bad” behavior served an evolutionary purpose in facilitating the spread of new concepts and techniques; the insights of nice guys were often ignored even when relevant to ongoing controversies. But it is doubtful that that behavior would be functional in the context of an applied problem with external clients.