Here’s a discussion of replication policy posted up in the relatively early days of the blog, which I’ve re-posted in light of NASA spokesman Gavin Schmidt’s attempts to justify Hansen’s refusal to provide the source code used in his temperature calculations. It seems that these calculations are important enough to prompt a concern over the “destruction of Creation” but apparently only the elect will be permitted to see these calculations. The discussion of replication is based on experience in economics and social science unrelated to the present controversy but fully applicable to it.
We get considerable criticism from paleoclimate scientists that complying with requests for data and methods sufficient to permit replication is much too onerous and distracts them from "real work". However, the problem is not our request, but that any request should be necessary in the first place. In my opinion, a replication package should have been archived at the time of original publication so that any subsequent researcher can replicate the results without needing to contact the original author. From my personal experience, non-academics typically assume that there are adequate due diligence packages and find it difficult to believe otherwise. It appears that significant academic experience is necessary to instill a belief that a due diligence package is an imposition.
Right now, it is obviously practical and feasible to create replication archives at the time of publication and this is a mandatory requirement in some fields (econometrics). In business, adequate compliance with regulations is often based on available practices. So I’ve tended to think that if this is feasible in empirical econometrics, it is feasible in paleoclimate science (where the structure of the datasets is surprisingly similar to empirical econometrics). During all of this, I have remained firmly convinced that climate scientists will not be able to avoid complying with proper standards for archiving data and methods much longer.
When I first ventured into climate replication, my framework was one of business audits and feasibility and engineering studies, a framework which has been ridiculed by some academics (inappropriately in my opinion.) I’ve often described myself as feeling like an anthropologist in studying the behavior of climate scientists, because their standards of replication and audit (or lack of them) seem as foreign to me as tribal customs must have seemed to early 20th century anthropologists in the South Sea Islands. During this journey, I have encountered some commentary on replication from within the academic community (though not the climate science community), especially from Dewald et al., McCullough and Vinod and Gary King, which captures almost exactly what I had in mind. We referred briefly to this in our E&E article and I’ll expound a little more today on this.
In our E&E article we said:
The ability of later researchers to carry out independent due diligence in paleoclimate is severely limited by the lack of journal policies or traditions requiring contributors to promptly archive data and methods. King [1995] has excellent comments on replication. In this respect, paleoclimate journal editors should consider changes taking place at some prominent economics journals. For example the American Economic Review now requires, as a precondition of publication, archiving data and computational code at the journal. This is a response to the critique of McCullough and Vinod [2003], and earlier work by Dewald et al, [1986]. The files associated with paleoclimate studies are trivial to archive. In our view, if the public archive does not permit the replication of a multiproxy study, then it should be proscribed for use in policy formation [McCullough and Vinod, 2003].
In March 2005, I reported with some satisfaction that we had been quoted in Anderson et al [2005], The Role of Data & Program Code Archives in the Future of Economic Research, available here, which pointed out:
For all but the simplest applications, a published article cannot describe every step by which the data were filtered and all the implementation details of the estimation methods employed. Without knowledge of these details, results frequently cannot be replicated or, at times, even fully understood. Recognizing this fact, it is apparent that much of the discussion on replication has been misguided because it treats the article itself as if it were the sole contribution to scholarship – it is not. We assert that Jon Claerbout’s insight for computer science, slightly modified, also applies to the field of economics: An applied economics article is only the advertising for the data and code that produced the published results.
Our E&E article especially referred to Gary King and McCullough and Vinod. Gary King has a website discussing replication here . His concept of a replication package, included among other things, data as used and source code, so that there was a permanent archive not simply for present reviewers, but future reviewers. King’s concern for future readers is significant – one of the arguments sometimes heard against examining MBH98 is that it ihas been superceded and that it is no longer pertinent to look at it. There are many arguments against the view that it is no longer being used, but here I point out King’s concern that published research be available to readers in the future.
The replication standard holds that sufficient information exists with which to understand, evaluate, and build upon a prior work if a third party can replicate the results without any additional information from the author." This was proposed for political science, along with policy suggestions for teachers, students, dissertation writers, graduate programs, authors, reviewers, funding agencies, and journal and book editors, in Gary King “Replication, Replication,” [pdf, or HTML] PS: Political Science and Politics, with comments from nineteen authors and a response, “A Revised Proposal, Proposal,” Vol. XXVIII, No. 3 (September, 1995): Pp. 443-499. Authors normally follow the standard by making available a "replication data set" to accompany each publication that includes the data, as well as details about the procedures to be followed to trace the chain of evidence from the world to the data and to the tables and figures in the publication. (Putting all this information in the article itself is preferable, but normally infeasible.)
King’s 1995 paper is online here and is worth reading in its entirety. Here is a replication standard proposed in this 1995 paper:
The first step in implementing the replication standard is to create a replication data set. Replication data sets include all information necessary to replicate empirical results. For quantitative researchers, these might include original data, specialized computer programs, sets of computer program recodes, extracts of existing publicly available data (or very clear directions for how to obtain exactly the same ones you used), and an explanatory note (usually in the form of a "read-me" file) that de scribes what is included and explains how to reproduce the numerical results in the article….
Possibly the simplest approach is to require authors to add a footnote to each publication indicating in which public archive they will deposit the information necessary to replicate their numerical results, and the date when it will be available. This policy is very easy to implement, because editors or their staffs would be responsible only for the existence of the footnote, not for confirming that the data set has been submitted nor for checking whether the results actually can be replicated. Any verification or confirmation of replication claims can and should be left to future researchers. For the convenience of editors and editorial boards considering adopting a policy like this, the following is a sample text for such a policy:
Authors of quantitative articles in this journal [or books at this press] must indicate in their first footnote in which public archive they will deposit the information necessary to replicate their numerical results, and the date when it will be submitted. The information deposited should include items such as original data, specialized computer programs, lists of computer program recodes, ex-tracts of existing data files, and an explanatory file that describes what is included and explains how to re- produce the exact numerical results in the published work. Authors may find the "Social Science Research Archive" of the Public Affairs Video Archive (PAVA) at Purdue University or the "Publications-Related Archive" of the Inter-University Consortium for Political and Social Research (ICPSR) at the University of Michigan convenient places to deposit their data. Statements explaining the inappropriateness of sharing data for a specific work (or of indeterminate periods of embargo of the data or portions of it) may fulfill the requirement. Peer reviewers will be asked to assess this statement as part of the general evaluative process, and to advise the editor accordingly. Authors of works relying upon qualitative data are encouraged (but not required) to submit a comparable footnote that would facilitate replication where feasible. As always, authors are advised to remove information from their data sets that must remain confidential, such as the names of survey respondents.
McCullough and Vinod have attempted to replicate empirical econometrics results. Thier experience was pretty reminiscent of my experience with paleocimate scientists, as will be seen below. The Hockey Team purports to excuse their non-production of data and methods to me on the basis that they would readily do so to "peers", but feel no obligation to provide information on data and methods to any Canadian businessman who happens to take an interest in their work. It seems irrefutable that their supposed willingness to produce detailed information on data and methods to "peers" has not been put to the test by any actual requests from "peers" else Crowley could hardly have "misplaced" his data and Mann would have known the URL of his MBH98 data (and the "peers" would presumably found the problems that I encounter.) As I mention repeatedly, I have never asked for anything that I did not believe was part of a due diligence package. The McCullough and Vinod experience shows that the problems are systemic to empirical science. Their analysis of why the problems exist are thoughtful as are their recommendations.
I’ve posted McCullough and Vinod online here. The first part is related to issues of nonlinear problem solvers, which are not necessary for replication issues. If you go to Part IV, you’ll find a terrific discussion of the problems of replication. McC and V:
we found that the lesson of William G. Dewald et al. (1986) has not been well-learned: the results of much research cannot be replicated. Many authors do not even honor this journal’s replication policy, let alone ensure that their work is replicable. Gary King (1995, p. 445) posed the relevant questions: [I]f the empirical basis for an article or book cannot be reproduced, of what use to the discipline are its conclusions? What purpose does an article like this serve?…
Though the policy of the AER requires that “Details of computations sufficient to permit replication must be provided,” we found that fully half of the authors would not honor the replication policy. Perhaps this should not be surprising” Susan Feigenbaum and David Levy (1993) have clearly elucidated the disincentives for researchers to participate in the replication of their work, and our experience buttresses their contentions. Two authors provided neither data nor code: in one case the author said he had already lost all the files; in another case, the author initially said it would be “next semester” before he would have time to honor our request, after which he ceased replying to our phone calls, e-mails, and letters. A third author, after several months and numerous requests, finally supplied us with six diskettes containing over 400 files- and no README file. Reminiscent of the attorney who responds to a subpoena with truckloads of documents, we count this author as completely noncompliant. A fourth author provided us with numerous data. les that would not run with his code. We exchanged several e-mails with the author as we attempted to ascertain how to use the data with the code. Initially, the author replied promptly, but soon the amount of time between our question and his response grew. Finally, the author informed us that we were taking up too much of his time- we had not even managed to organize a useable data set, let alone run his data with his code, let alone determine whether his data and code would replicate his published results.
Replication is the cornerstone of science. Research that cannot be replicated is not science, and cannot be trusted either as part of the profession’s accumulated body of knowledge or as a basis for policy. Authors may think they have written perfect code for their bug-free software package and correctly transcribed each data point, but readers cannot safely assume that these error-prone activities have been executed flawlessly until the authors’ efforts have been independently verified. A researcher who does not openly allow independent verification of his results puts those results in the same class as the results of a researcher who does share his data and code but whose results cannot be replicated: the class of results that cannot be verified, i.e., the class of results that cannot be trusted. A researcher can claim that his results are correct and replicable, but before these claims can be accepted they must be substantiated.
This journal recognized as much when, in response to Dewald et al. (1986), it adopted the aforementioned replication policy. If journal editors want researchers and policy makers to believe that the articles they publish are credible, then those articles should be subject, at least in principle, to the type of verification that a replication policy affords. Therefore, having a replication policy makes sense, because a journal’s primary responsibility is to publish credible research, and the simple fact is that “research” that cannot be replicated lacks credibility…..
We chose recent issues of JIE and IJIO, and made modest attempts to solicit the data and code: given the existence of the World Wide Web, we do not believe that obtaining the data and code should require much more effort than a few mouse clicks. We sent either e-mails or, if an e-mail address could not be obtained, a letter, to the first author of each empirical article, requesting data and code; for IJIO there were three such articles, and for JIE there were four. Only two of the seven authors sent us both data and code…
As solutions to these problems, as part of a symposium on the topic of replication, King (1995) discussed both the replication standard, which requires that a third party could replicate the results without any additional information from the author, and the replication data set, which includes all information necessary to effect such a replication. Naturally, this includes the specific version of the software, as well as the specific version of the operating system.This should also include a copy of the output produced by the author’s combination of data/code/software version/operating system. In the field of political science, many journals have required a replication data set as a condition of publication.
Some economics journals have archives; often they are not mandatory or, as in the case of the Journal of Applied Econometrics, only data is mandatory, while code is optional. A “data-only” requirement is insufficient, though, as Jeff Racine (2001) discovered when conducting a replication study. As shown by Dewald et al. (1986), researchers cannot be trusted to produce replicable research. We have shown that the replication policies designed to correct this problem do not work. The only prospect for ensuring that authors produce credible, replicable research is a mandatory data/code archive, and we can only hope that more journals recognize this fact.
To the best of our knowledge the only economics journals that have such a policy are the Federal Reserve Bank of St. Louis Review, the Journal of Money, Credit, and Banking, and Macroeconomic Dynamics. The cost of maintaining such an archive is low: it is a simple matter to upload code and (copyright permitting) data to a web site. The benefits of an archive are great. First, there would be more replication (Richard G. Anderson and Dewald, 1994). Second, as we recently argued (McCullough and Vinod, 1999, p. 661), more replication would lead to better software, since more bugs would be uncovered. Researchers wishing to avoid software-dependent results will take Stokes’ (2003) advice and use more than one package to solve their problems; this will also lead to more bugs being uncovered. Finally, the quality of research would improve: knowing that eager assistant professors and hungry graduate students will scour their data and code looking for errors, prospective authors would spend more time ensuring the accuracy, reliability, and replicability of their reported results.
As a result of McCullough and Vinod, the American Economic Review has adopted a policy requiring a replication package, including both code and data, to be archived as a condition of publication.
It is the policy of the American Economic Review to publish papers only if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication. Authors of accepted papers that contain empirical work, simulations, or experimental work must provide to the Review, prior to publication, the data, programs, and other details of the computations sufficient to permit replication. These will be posted on the AER Web site.
It’s not just paleoclimate scientists that don’t want people to check their empirical work. The time to deal with replication packages is at the point of publication – not in rear-guard actions after the fact, which require authors to try to remember what they did.
In some cases, journals have adequate policies on data in the narrowest sense (i.e. the numbers and not the code) e.g. our recent discussion of Science. The main problem appears to be administration. As King points out, this need not be onerous. One simple way of ensuring much improved compliance would be to add a form at the time of submission – say like Nature’s declaration of financial interests – in which the authors provide a link to a replication archive (which might be a private archive at the time of submission) with a warranty that they will transfer the replication archive to a permanent archive like WDCP between acceptance and publication. They would have to submit a second online confirmation verifying that the archive had been transferred to a permanent archive.
AGU has perfectly good data citation policies, essentially prohibiting the use of "grey" data. Unfortunately these are not folloed at AGU publications, like GRL or JGR. This is a slightly different issue than the replication archive as a full data citation includes the URL for a digital source version – citation of print publications for digital sources is not adequate under AGU policies for obvious reasons, but is still usual paleoclimate practice.
Climate Audit 
270 Comments
thank you Steve. well written.
I’d like to add a request for clear, objective descriptions of exclusion criteria. Instead of something like, “Obviously erroneous” data was excluded.
The anthropological angle is apt. This is a typical response of an organization not used to doing quality control, when quality control is suggested. “It can’t be done”. Whether it’s furnishing audit data, or whether it’s occupational safety, or whether it’s 6-sigma QC, the response is that it’s unreasonable, and can’t be done.
If Kyoto and credit trading can be done, so can this.
Yeah…so? Understanding a complex natural system is different than business or engineering. Doing science is different than business or engineering. Your whole approach is to apply a framework that won’t work, that doesn’t apply. The beauty is, you can then turn around and point out perceived shortcomings of science as a function of your misapplied approach.
Understanding a complex natural system is different than business or engineering
There may be more to science than math and statistics, but without them there is no science.
Perhaps it should be recast as “The Philosophy Of Global Climate”. That would certainly be more honest.
brian, you claim that:
Some examples of what those differences are, and what difference those differences might make to studying complex natural systems, might make your claim believable. As it stands, it’s just an unsupported opinion, which isn’t worth much on this blog.
w.
I have concluded that the failure of some academics to provide or be willing to provide data is what Ill call the embarrassment factor. They are too embarrassed to show the data in current sloppy and potentially disjointed form and too busy to put it into proper form. In areas, like climate science, where publishing appears to be a more urgent business the more likely that the data will be in less than well ordered form and the more busy the scientists will be moving on to their next publishing project.
To say that understanding a complex natural system is not business completely evades the point: why is it so difficult to provide the means to replicate?
3, so your answer to complexity is for them to just hand-wave? Is that how it’s done by the pros? You’re not making them look good with a comment like that.
Is it just me or has “brian” just dragged a complete “red herring” to try to obscure Hansen’s responsibility? First of all, Hansen is a SCIENTIST. If his work is not replicable then it cannot be considered valid at all. Secondly, James Hansen his feeding at the PUBLIC through. If he wishes to maintain that he is above validation, then who need listen to him on any subject? Even Einstein could not make it up as he went along. Finally, I believe it was engineers who rendered E=MC2 practical.
Actually, everything serious is business. If you don’t treat you activity as such, then it must be a game. Those are the only two choices.
8, “brian” just got jumped up one side and down the other for saying something absurd. A red hearing at least has a ring of validity to it. That was less than a red herring. More like a 7-day old anchovy.
#3 Brian
The systems do work. In many fields they are required, and the time period that they must be replicable is also required and stated. But it is not “understanding a complex natural system” that is involved. Replication is the ability to replicate. Are you telling us that as you understand a complex system, it may not be replicable? In what way is that science or understanding? Replication of results is the cornerstone of science. Further, a common misconception is that engineering is not science. That is most incorrect. Engineering is doing replicable science, usually for profit. Not always though, NASA has used engineers as part of their R&D effort from anything from satelites to space vehicles.
The approach’s “shortcoming” is that it makes someone actually support their claims with real verifiable work. This work does take time and have costs. However, if it is worth publishing, then it is worth archiving. Finally, as someone who must have things saved and replicable, the hardest part is the work, not replication. Once setup and proper methods used, the work to archive is seldom much more than a little of what it takes to do the work. The major obstacle is setting it up the first time and making it a habit. Plus you need to verify that what you set up actually accomplishes the goal. After that, it is like riding a bicycle…it is done more by memory than by thought.
Hansen’s adjusted temperature record would not be admissible and Hanson would not be allowed to testify about the adjusted temperature record in any court. Florida Rule of Evidence 90.705 (actually Section 90.705, Florida Statutes), states:
This provision of the Evidence Code has been consistently construed to require disclosure of all computer code used to process any data which underlies expert opinion. The Federal Rule is similar. In other words, Hansen’s “adjusted” temperature “record”, as the opinion of an expert, would be inadmissible in court anywhere in the United States, as would any evidence or opinion based on his temperature “record”, unless and until the data, and the computer code used to process the data, were fully disclosed.
I guess this shows the difference between business and academe.
Where I spent the last 32 years, if you DIDN’T include your raw (all of it), your paper, report, whatever, ended up in the round file — because it couldn’t be verified.
Decisons involving hundreds of thousands – if not millions – of dollars depended on that kind of rigor.
A better point is that approaching science from a statistics point of view is problematic and “non scientific”. I first saw this when I worked for a major re-insurance company, filled with advanced math wizards doing CFA type work. I was a bit surprised to learn that they approached the economy as a unintelligible black box. They modelled scenarios and based on past economic data, assigned likelihoods to scenarios. Even very high level financial folks seemed oblivious to the laws of supply and demand, or any first principles.
These AGWers do the same thing. Even though it’s a truism that statistics lie, they are continually trying to use statistics to make very specious scientific conclusions. This is not what Steve M wants to hear, since he wants to spend his retirement auditing these clowns.
Sen Inhoffe press release (on AGW):
http://epw.senate.gov/public/index.cfm?FuseAction=Minority.Blogs&ContentRecord_id=84e9e44a-802a-23ad-493a-b35d0842fed8&Issue_id=
statistical analysis of “science” numbers is the same as statistical analysis of “business” numbers.
go make the same mistake on Wall Street and see if those guys think a .15 error is a big deal or not.
Re: 14
Extrapolation using regression models can be dangerous.
>> Extrapolation using regression models can be dangerous.
meaning?
Wow, you all were quick on the pounce there, impressive. By the time I actually submit this, i’m sure there’ll be several more. Anyway…
Firstly, I never said replication wasn’t part of science. I was commenting on the excerpt I quoted. My point is that reproducibility in science is not always accomplished with an auditing or business-style approach.
Science does indeed have uncertainty involved…those with a low tolerance should not venture. But there are degrees of uncertainty, and science is trying to minimize (and measure/quantify) uncertainty as best it can.
The tone here reminds me almost exactly of my experience in petroleum exploration…the engineers simply can’t handle the uncertainty of geology. They get so peeved at the differing interpretations, explanations, concepts, and so on (all of which are driving towards reducing uncertainty in the ultimate goal). The engineers try again and again to ‘replicate’ the process of science and can’t. Lots of money is involved in that too…so, don’t give me that line.
And I am by no means saying science can exist w/out business/engineering…this isn’t a contest. But if business and engineering are “the only serious choices”, then what is science for?
There is a disdain for scientific expertise at this site that is quite scary. Good luck w/ your audits.
(p.s. i’m not literally comparing this to my petroleum story above, so calm down and stop drafting your response about how absurd I am…or whatever other hatred you have brewing)
*delurk*
Re #3: I think this post displays a worldview which is widespread in the scientific community and which is going to have to be addressed one day; that science should be a kind of black box as far as the rest of society is concerned, which only produces outputs and is not scrutinised by anybody else. It comes from an entirely understandable desire for science/academia to be free, and not influenced by e.g. political forces, a rugged independence effectively.
The problem is that the role of science has changed. Where science has no direct bearing on the world, fair enough. And nobody would wish scientists to be subjected to unscientific pressures. But some scientific fields are directly bearing now in very significant ways on society- not just environmental sciences, but medicine and social sciences for instance, and we have an impossible situation where those scientists wish to dictate/heavily influence public policy, but then when questioned shrink back and claim to be just scientists who should be left alone. When science does bear on society, society has a right to ask questions. To use the business example; businesses are audited because their accounts matter to all (due to taxes etc); we can’t just trust them to do an honest job, even though most businessmen are honest. If unscrutinised, the pressure to cheat would be significant.
Likewise, unscrutinised science is under a pressure to “cheat”. If the scientists themselves are heavily politically motivated, as many seem to be, then sorry, I, as an ordinary member of society, want their work scrutinised. That means full disclosure of their methods, data, etc, for independent verification.
In the end, the argument from scientists boils down to “this is the way we’ve always done it”; well that was true of business too before auditing was introduced. Things do need to change. If scientists expect the rest of society to just accept their work, and follow their advice, without scrutiny, they become a kind of priesthood- philosopher kings- and what room does that leave for democracy?
This is the most important science in the world. For it not to be subjected to the most intense, thorough, public scrutiny, is unacceptable. And when scientists become vociferous campaigners, as with Mr Hansen, then they rightly must be exposed to the scrutiny that comes with public political life too, however much they may dislike that idea.
*relurk*
I can understand why everyone is jumping on Brian (#3).
But there’s not much point. He clearly does not know anything about the Scientific method.
Brian, please at the very minimum, go to Wikipedia, for heaven’s sake, and for the sake of your sanity and ours.
The link is here: http://en.wikipedia.org/wiki/Scientific_method
Please read and try to ABSORB the meaning of phrases like:
“Scientific researchers propose hypotheses as explanations of phenomena, and design experimental studies that test these hypotheses for accuracy. These steps must be repeatable in order to predict dependably any future results.”
“Among other facets shared by the various fields of inquiry is the conviction that the process must be objective to reduce a biased interpretation of the results. Another basic expectation is to document, archive and share all data and methodology so it is available for careful scrutiny by other scientists, thereby allowing other researchers the opportunity to verify results by attempting to reproduce them. This practice, called “full disclosure”, also allows statistical measures of the reliability of these data to be established.”
There are many serious investigators who do archive their data and make them available.
Please see for example, descriptions of work done by people like William B. White, Prodessor Emeritus, Geochemistry, Penn State University.
Some of his thinking and that of his colleagues can be gleaned from Journal references as in these links:
Click to access Newsletter%20Fall%2004.pdf
Click to access Newsletter_Spring_05_r1.pdf
and my favourite:
Click to access v65n2-White.pdf
After reading some of the entries above, I don’t see how anyone can question the science inherent in the field of geology, as “Brian” has implicitly done in posting #3.
For those of you who might be interested, please see an interesting 5-minute video at this link from DISCOVERY:
http://link.brightcove.com/services/link/bcpid1137855816/bctid909952904
This video has a correspondent named James Williams interviewing the above-mentioned William B. White about stalagmites (NOT stalactites!). Professor White is doing research on what the stalagmites can tell us about climate changes above ground from where they are found, much in the same way as ice cores are used today. The potential advantage is that caves are found world-wide and they can broaden our understanding (through GEOLOGY!!!) about Climate Change through hundreds of thousands of years without the problems associated with other proxies like tree rings.
Hope you enjoy the video, Brian!
Actually, if you have been paying attention, there is a high regard for scientific expertise here. Very little regard for posers, however.
#3 – Assuming your assertion that science is different from business, propose a way to verify scientific conclusions are correct without replication or verification. If you can’t, then Steve’s proposal will have to suffice until there’s something better.
Business and engineering cannot handle uncertainty. The people who manage the economy and build large scale projects cannot handle uncertainty. Have you ever managed a business or designed a large scale system?
Archiving data and the methods used to replicate whatever’s been done to it has nothing to do with the field. “Understanding a complex natural system” is not a field, business and engineering are. There’s nothing un-complex about building a bridge or an airplane, designing an engine, tracking the stock market or analyzing the GDP of a large country.
Chemistry is mechanical engineering is computer science is mathematics is climate science is stock forcasting when it comes to replication.
If I can’t replicate your methods, what you produce is no better than garbage from a validation standpoint. If I can’t do exactly what you did exactly how you did it, it’s not validating what you did.
#19, Brian:
You said –
I agree, ‘auditing’ is probably not the best approach to reproduce a result. It is,
however, a perfect way to falsify a result.
You can reproduce a result 50 times; that gives you confidence in the correctness of the hypothesis, but it does not prove it true.
You only have to falsify once to demonstrate the hypothesis is wrong.
“This is the most important science in the world.”
It’s trying to become “the most important science in the world” but if “importance = dollars” then I’d say that medical science has the sway. Try and bring a drug to market with the “science” that Hansen, Jones et al is peddling and the gov will get you – or the lawyers will.
Re: #21
I think the point is that many with contrary views and defenders of the consensus come here and attempt to change the subject with tactics like Brian is using. Some are just less obvious. I would really like to hear what those with differing POVs have to contribute — just do not change the subject.
Sorry to butt in again, but reading Brian’s response:
Brian, I’m not sure whether you’re talking about uncertainty because you’ve misunderstood Climate Audit, or to set up a straw man, or because you’re trotting out a standard “science/society interface problem”. The purpose of the auditing is not to find certainty regarding AGW, i.e. to prove a definite yes/no to whether CO2 is destroying creation. It’s to verify whether the science being audited contains errors. As such, it’s a very scientific process.
If an astronomer declares that the stars have disappeared, for instance, it’s prudent for somebody to audit his work by for instance checking whether he opened the observatory dome. To discover it was closed doesn’t prove itself the existence or non-existence of the stars; but it does invalidate the initial claim. If OTOH, other astronomers “audit” by also looking at the firmament and they too find an absence of stars, the original claim is strengthened.
Steve McIntyre isn’t trying to replicate “the process of science” (I take it by your context that you mean “copying the scientific method”) as I see it. He’s scientifically trying to replicate the research. One thing is a basic of science- if person A does experiment X and gets result Y, person B should get result Y from experiment X as well. Reproducibility is the very basis of the scientific method. A scientist who says “I got Y, and I’m not going to tell you how, trust me” isn’t doing science at all.
*lurks again*
Hans…wow, I’ve never seen a more clear example of misreading and lack of comprehension. Please re-read what I said in #19, which is the comment where I mention geology.
I was defending the science of geology, not attacking it. What’s implicit in my statement is that I’m the geologist. I have no idea what your point is. I’m glad you explained the scientific method to everyone…and I surely hope you’re not one of the auditors. But, your misunderstanding my little story is unimportant.
Verify without verification?
Nobody is listening to what i’m saying. Again…I am not against verification or reproducing scientific work to evaluate the conclusions. There are a few responses to my comment implying that, so that you can then tear it down…the ol’ straw man tactic. One more time: I am for reproducibility, scrutiny, anal-retentivity, and so on in the ultimate goal of improving our scientific understanding. I am an academic and spend all day every day trying to pick every little nit out of my colleagues work. What i’m saying is that the business-style auditing may not be the best approach to this problem. I think Ian Bland’s (#20) comment is more along the line of the dialog we should be having regarding this issue. Of course science shouldn’t go unscrutinized…but the analogy that keeps coming up again and again relating this approach to accounting or taxes is, to me, misguided.
I know…you’re gonna call me a poser, or dumb, or absurd….go ahead
OK Brian, I’ll bite – in what way is climate science different?
You see, to me, science is the discovery of understanding and engineering is the application of that understanding (if that makes sense). In essence, engineering relies heavily on science – you cannot build a stable, reliable structure without the knowledge that materials science supplies WRT the properties of the materials you are using, for example. Therefore, to be of practicle use, science needs to be replicable, and there more impact the science has on society, the more stringent this requirement becomes. Quantum physics would seem to most people to be rather esoteric and with little application in their daily lives – until they find out that it is vital to the production of electronics, which is embedded in more and more everyday tools.
The difference that I see in climate science is fairly straightforward – there are no engineers “in the middle”, sorting out the myriad of details that need to be sorted out in any move from theory to practical application. Yes, it’s tedious – even boring at times. No, it’s not always pleasant. But it is vital. And it does involve discovery and learning in and of itself. It seems to me that many scientists (not just climate scientists either, BTW) see such work as unimportant and somehow “beneath” them. This is regrettable, and as someone who has supplied such services to scientists, I can assure you that this is most definately a two-way street – it is at least as frustrating for the engineer as it is for the scientist, and the engineer will feel unappreciated and undervalued as well as frustrated. We (enginners) get over it. You (scientists) should too. It may help you to realise that while you see engineers as specialists supporting your work, so engineers see scientists as specialists supporting their work too. Oh – replacing “engineer” with “auditor” ( or “plumber”, “electrican” etc) and the result is the same.
Ahhhh…
I suspect you have a misguided sense of what constitutes “business-style auditing”. Your background as an acedemic could explain that.
The experiences I referred to in #13 involved process and chemical engineering. Sometimes we were trying to justify new expenditures; other times trying to prove cost savings. In any case, the fundamental arguments were based on process data that had to be verified. I always considered that as “business-style auditing”.
Re #30
Seems Brian’s whole gripe is that he doesn’t like the word “audit”. It feels too corporate for his sensibilities.
Brian, what word can we substitute for “audit” that will be both politically correct and feel comfortable to you?
Ian says (#29):
But the blog description says this:
At least that’s the description that comes up as the description when you google climate audit.
#15
Thanks, Sen Inhoffe press release was very useful. What would have been nice is he had included a plug for better data archiving or more explicit criticism of some in the scientific establishment. Based on the list of citations it looks like Gavin is going to get his wish for more peer reviewed studies – unfortunately the conclusions may not be what he expected.
Update from Norway. Every household, starting from 1.1.2008 will be charged on average about US$466 on their electrical bill because of the kyoto protocol.
What I wonder though what’s it like to play poker with James Hansen. You show him three queens and he shows you a pair of fours and insist that he actually has a full house. Of course he has now intension to show what the remaining cards are. But insist that there are no doubt that he has won the pot. After all he will let his “peers” confirm he actually has won.
Sort of a contradiction in terms. He claims that the world is coming to an end, but refuses to show his evidence! If I had such evidence I’d trow it around all over the place
34, where exactly did that blog description come from?
#37 — google ‘climate audit’ and see the sub-title for the climateaudit.org address (3rd one down when i did it)
if this is wrong…you’d better fix it….because “tries to show how human induced global warming does not add up” sure sounds like an agenda to me
it is worth noting that several areas of science already have disclosure as a prerequisite. For example, in astrophysics, it is virtually without exception that all raw data is publically archived. I believe there have been several cases where re-analysis has picked up errors.
Likewise, in genetics, it is mandatory when you sequence a gene, to put your sequence into a public repository when you publish. When you get a 3d structure of a protein, it is likewise mandatory to put the data into a public repository.
When it comes to science that is required for protecting human health, such as studies to underpin drug safety, that is done to a standard (GLP) which includes built-in business-style auditing.
I must admit that I am bemused by Brian’s claim to be in favour of verification, and scrutiny; but not to believe in “business-style auditing”. Hansen’s computer code is a “black-box” which transforms raw data into end-product, which was the very essence of a paper. If you can’t see the code, or a detailed description, how can you possibly evaluate how good the paper is ?
Whether you call this “verification”, or “business-style auditing”, is not the point. The point is that you need to be able to repeat science, and there is not the information available to do so. And yes, this is an important general principle in all science; because if one person can get away with claiming results, without showing how they did it, why cannot everyone ?
per
But Brian (#30), Steve is a geologist as well.
Perhaps you should address your concerns directly to him.
I clearly am not at your level of understanding or misunderstanding.
Is your intention to just befuddle with your brand of illogic?
Or as Kenneth suggests in #28 are you attempting “to change the subject with [your] tactics…”?
If you really believe that there is some difference in “audit techniques” then say so.
What exactly are the tenets of “business-style auditing” that are inappropriate in your mind?
To me, it seems obvious that even a non-regular visitor to Steve’s weblog would know that he is only saying that the data and methods used in support of the papers done by Climate Scientists need to be posted and need to be accessible. If you agree then no one is disagreeing with you.
I just don’t understand what you are saying.
Steve has never addressed (that I am aware of) any particular AUDIT METHODOLOGY, of a “business-type” or “science-type” or any type. He is asking that Climate Scientists follow the same principles that other scientists (including you presumably) are expected to follow.
What is it PRECISELY that you find inappropriate (audit-wise) in what you refer to as:
“Your whole approach is to apply a framework that wont work, that doesnt apply. The beauty is, you can then turn around and point out perceived shortcomings of science as a function of your misapplied approach”?
Can we even define the difference between a “business style audit” and peer review? is it that the first has a pejorative ring to it? What exactly is the essential difference? Rigor is too much work?
Brian,
Ever had one of your designs “peer-reviewed” by another engineer or construction manager who’s mandate is to either confirm the validity of your design or to discover its flaws? You turn over all your info – calculations, computer inputs, drawings, specifications, etc. And you sweat. You get a chance to defend your design, but the reason for the “peer review” is to ensure that you don’t screw up, and in so doing, screw your employer when a deficient design leads to unnecessary construction costs and, maybe, a lawsuit from your client for incompetence. It’s an internal audit of your decision-making, and it’s gut-wrenching.
I’ve been in the environmental engineering field for 28 years (plus 6 in energy conservation programs), and I still cringe at that audit, though not to the same degree as when I was a newbie (experience helps here). But I would never tell my Inquisitor “How dare you question by design?” That’s professional suicide and intellectual corruption.
But my designs are usually done in a year or less, and any flaws are caught by the audit early before they do any significant damage to anybody (client, employer, my career). What happens if you’ve invested your career since 1988 (almost 20 years) to a thesis of CO2 global warming as the main driver for temperature increases that can be undone by the “deniers'” focus on solar variations, cosmic rays, natural climate changes, recovery from the Little Ice Age, cows farting, compromised data, poor computer coding, etc.? And people start to realize that while the costs for CO2 reduction will be fixed, the benefits part of the Cost/Benefits ratio will possibly shrink to insignificance, making CO@ reduction a destructive, losing proposition.
Could explain the tone of Hansen’s and Schmidt’s latest responses. Falling off the AGW gravy train can hurt.
Is it a standard practices in other scientific disciplines to share data?
Re #43
Dan,
It is indeed standard practice in all science disciplines to provide when asked the data from published reports, given that the data is not specifically embargoed due to specific contractual or policy reasons.
38, Brian: Why don’t you Google “Hockey Stick,” and see where that leads you. LOL.
“we have to get rid of the Medieval Warming Period”
why?
You would think that the journals would required the necessary information to replicate the work for their own purposes of insuring that the data being presented is indeed what the methods purort they say (ie., the Journal should replicate the data before it is ever printed). As a Ph.D. myself, I know that graduate studies require that the student prove that the methods they used actaully produce the results they get. I guess this may not apply for climate science studies.
RE: #2 – I have literally disqualified operations on the spot when they responded in such a manner. One of them tried to bribe me, leading to permanent black listing of that particular enterprise.
RE: #3 – you clearly have no idea just how complex certain engineering / engineered systems now are. A/I is required to design , manufacture and test them. You are wrong.
RE #15: Quick question: is ClimateAudit planning to audit the data/models/equations, etc. used in the Schwartz paper?
RE #43: Oh, and in the work* I’ve published (pharma/biotech) I would not have expected anyone to ask for the raw data – except for gene sequences which were always deposited into GenBank. Basic drug discovery, mechanism of action, or animal efficacy studies are non-GLP and are regarded by the FDA as supporting studies – published papers are OK.
* some of it supported by Federal funds
RE: #21 – Of course, he mentioned geology, where, the “science” spans from dousing and geomorphology (which appears to be the camp Brian is in, I know it well, I was once there myself…. prior to some hard knocks) to truly scientific areas that are completely data driven (seismic reflection, assaying, geochem, etc). Certainly, real scientists do not shy away from data, and replicability.
Brian #34, your argument misses the point.
There is no conflict in defining the purpose of a single action (say, an audit*) to be something different than your broader or ultimate goal. Audits attempt to find errors and inconsistencies. Doing so might be helpful to an organization’s broader goals, but that doesn’t mean the audit and the ultimate goal are the same thing.
For example, in a for-profit corporation, the audit serves the purpose of finding errors and inconsistencies, but the goal of the corporation is maximizing profit. For a scientist, the audit also serves the purpose of finding errors and inconsistencies, but the goal of the scientist is maximizing certain kinds of knowledge, or perhaps proving some hypothesis to be true or false.
(You can argue that it is bad for a scientist to have some desired experimental outcome, even if he manages to avoid biasing his experiments; but I think we see that on both sides of the AGW debate.)
* “The purpose of the auditing is not to find certainty regarding AGW, i.e. to prove a definite yes/no to whether CO2 is destroying creation. It’s to verify whether the science being audited contains errors. As such, it’s a very scientific process.”
If a person was a climatologist and requested all of Hansen’s data, would he give it to them? Or is the refusal to provide source code only to persons such as Steve McIntyre?
Secondly, when a scientific article goes through a peer-review process, what does it really entail? Is it a thorough scrutiny of the data or is it more of a quick once over by other like minded professionals?
Thanks . . .
I have to laugh. Most of us toiling in obscurity would welcome the attention that Mann and Hansen receive. For me, it’s a delight to share data, to receive reprint requests, to see my work undergo scrutiny; I am grateful to those (mostly Asian graduate students) who ask me to explain the details of how I formulated a problem, to justify my use of a particular word, to defend a conclusion. For someone interested in my work (!!), I happily dig through old files for notes or source code.
Do they find errors? Of course! I make mistakes. The point is, they help me; in a few cases, an interesting paper has emerged from such criticism.
This is why the attitude that Mann and Hansen exhibit toward SteveM flabberghasts me. SteveM is a true genius — fantastically gifted at math and statistics — who, free of charge and for no particular reason, has given them exactly the kind of scrutiny that the rest of us crave. Imagine where we’d be if Mann and Hansen had recruited SteveM to help clean up their problems instead of reflexively denying him access to the details of their work.
IMHO, Mann and Hansen have too often behaved like a couple of prima donnas (which, sadly, they’re not); they don’t deserve SteveM’s time.
Now they’re going to pay for it.
I think it`s quite obvious that Hansen et.al have something to VERY BIG to hide.
If they knew that their work was legitimate, then they would rush to give Steve the data/source code, so that when Steve checked it and it was OK, they could say “See, we told you. We are being unfairly persecuted by the oil/mining funded Steve McIntyre which is taking us away from our important work”
They would hold media conference to publicize their vindication and use it as proof of the need for more money and the need to silence the likes of Steve etc etc.
[snip].
Game on….
Attention is what they want all right, but please no focus on the data or methods. Hansen, Mann et. al. have been cooking the books for a long time.
“It is time for us to laugh at the ideologues who try to pretend that any criticism of Global Warming alarmism is idiotic and unscientific. They are the ones who ignore the data; they are the ones who believe on faith alone, without evidence; and, most important, they are the ones who are trying to stifle the opposition without answering it.
The Global Warming alarmists are the anti-science religion that is trying to forcibly indoctrinate and convert everyone while suppressing dissent. And the news media are their patsies, their stooges, their puppets.”
This from a nice summary of the hockey stick debacle. More here: http://www.ornery.org/essays/warwatch/2007-03-04-1.html
#50, Deech56
You noted that in your area of work:
The difference being that in most fields of scientific discovery [like yours], researchers are writing up actual experiments in their journal submissions. A variable is changed and an effect is noted. “I took X and added Y. I heated while stirring slowly at 130 C for 20 min. Z resulted. No effect was seen in the control.”
The description is sufficient for replication. No additional ‘raw data’ are needed. The same can’t generally be said for climatology or econometrics.
Have you ever seen a paleoclimatology paper that begins:
“I took two Earths, and doubled the CO2 concentration on one of them…”
As in econometrics, paleoclimatology papers are generally of the form:
“I used some tiny subset of the vast accumulation of available data. I performed a mathematical transformation and subsequent statistical analysis of my selected data. A trend became apparent. The trend is X.”
How can the result be replicated w/o identification of:
1) the precise identity of the tiny subset of source data, and
2) what math is being done on the source data ?
More importantly, how can a specious claim ever possibly be disproven if the source data and underlying math [including source code, if any] aren’t properly disclosed ?
If were to follow the methods suggested by people like Gavin Schmidt, everybody would just do their own math. If yours disagrees with mine then you must have done it wrong, you haven’t shown that I’m wrong. That’s not science.
This is exactly what is desired. I presume the “…differing interpretations, explanations, concepts…” were all rooted in the same data, no? That’s all we’re asking for, simply the opportunity to “drive towards reducing uncertainty”, yet it’s like pulling teeth.
I never realized my comments would create such a brouhaha…your responses above help me understand your approach. Some are genuine and I will read them…some make no sense at all and do not address the phrase they are quoting of mine in their response. I will try and filter and read the worthwhile opinions.
I guess my main source of skepticism with this whole auditing thing, is that I read on this post and previous posts in the last couple weeks, at least in the comments, a strong call for scientific reproducibility, for climate scientists to follow the tenets of the scientific method, to adhere to the same principles as other sciences, and so on and so forth. On this blog, and on others I’ve read the comments of many (perhaps some of you) claiming that you are only trying to accomplish the due diligence and detailed review of some of the scientific studies. Is this more-or-less what I’m hearing?
But…the description for this blog, per Google is:
The whole purpose is to demonstrate that AGW is false. It says it right there. Right from the get-go there is an agenda. To me this is putting the cart before the horse. A predetermined conclusion is established (i.e., AGW is false) and then you are compiling data to verify that conclusion. This mission statement, or whatever you want to call it, broadcasts to the world that this work has a preconceived interpretation. The scientific method, in addition to the aspects that have been quoted above and in other posts lately, is designed to minimize biased interpretation.
Now…if you claim the climate studies you are referring to did not properly follow protocol, then you have a valid concern, AGREED. I don’t know how many times I have to say that i’m not against scrutiny. But, let’s get this straight. The purpose of the Climate Audit’s audits are to disprove AGW.
I suppose it would’ve been better for me to have known this was the way you were doing things before came into the discussion.
Like I said, I will try and read many of the responses to my comment above to understand more about how this all works. I’ve learned so much already today.
brian, it’s time for you to learn a lesson. Do Not Believe Everything You See In Print. Or on Google. Learn to become a skeptic.
Google the phrase you found… and you will find it was written, not by Steve M, not on this site, but rather here:
http://www.dmoz.org/Society/Issues/Environment/Opposing_Views/Climate_Change_Skeptics/
Surprise… that’s not a particularly good overview of ClimateAudit!
Does making fun of me make you feel better? Picking on a lone dissenting voice in a thread like this is very brave. You don’t know me, I don’t know you…what does this accomplish? It’s extremely difficult to try and read from some the other comments that have constructive things to say while having to deal with that BS.
It’s not really worth it. If your goal is to bully away any ‘skeptic’ of your cause, then I guess you accomplished something.
Brian,
can you explain your interpretation “business audits and feasibility and engineering studies” and why you think applying such a framework is inappropriate for science? As a working scientist, either my claims stand up to scrutiny or they don’t. If they don’t, I have to go back to the drawing board.
— Ralph
(BTW, if you spend a bit more time listening before you write, you will come across one of Steve’s many declarations that he himself is of the opinion that warming is happening, and even that mankind has a role to play.
But like any good auditor, and like any auditor with experience in exposing fraudulent financial schemes, Steve has a nose for discovering errors.
Anyone involved in business (in the West) is used to being audited. Most businesses pay a hefty fee for the privilege of having one or more people go over everything with a fine-toothed comb. They pay to have the auditors write a letter to the management pointing out the troubles seen, pointing out what was not fixed from the prior year’s audit, and so forth.
Moreover, if the auditor cannot certify that everything has been corrected to be up to snuff, then the auditors refuse to write the glowing letter of certification that all is well… and the company’s stock drops through the floor.
That’s the kind of audit Steve is used to doing. That’s the kind of audit any healthy scientific team should be able to go through with aplomb.
That’s the kind of audit that scares GISS to death.
Brian, I am not qualified to address the motives for the desired information, and though motives are important usually, this is not an area where they can be justification for non-compliance of what are, or should be, standard practice. If, upon the release of data and methods, you or others should wish to “counter audit”, and base your reasons for doing so on the ulterior motives of this crowd, you’ll be quite welcomed. The fact is, two plus two does not equal five no matter whether my motives be pure or otherwise.
Hey, what do I care who wrote it…it’s what comes up under climateaudit.org … the normal joe who is searching for CA will see that message affiliated to the site. If it’s wrong, get on Google’s case! Are you really blaming me for this?
If it’s wrong…get it fixed! Audit Google!
Gavin Schmidt has not responded to my request for a recommendation for a clear exposition of 2.5 deg C.
I’m mulling over the following thought: if the science is settled, then determining whether the impact of doubled CO2 is 0.5 deg C or 2.5 deg C or 10 deg C is an engineering problem. Arm-waving expositions – the higher, the colder – don’t answer this.
RE #65:
Brian, I really don’t think anyone is “blaming” you for what Google does. Personally, I appreciate your pointing it out.
Google lists according to hits. What that citation means is simply that more folks are hitting Open Directory than ClimateAudit.
And, the fact that Open Directory is “volunteer” means that there is really no control over what is written there. Kind of like Wikipedia (which to me is one step removed from making it up on your own).
Brian, yes, I’m blaming you for assuming someone else’s opinion of CA came from CA itself. Google is not about truth, it’s about indexing. DMOZ is even worse — whatever the editors want to say, stands.
Google Steven McIntyre and the first entry will be on Wikipedia. I hope you have enough common sense to understand that WP is not about truth either. They allow anything to be written, and after the arguments are over, only “verifiable (non-primary) sources” are allowed… i.e. those who know the truth are not allowed to contribute directly to WP. Only hearsay from others is allowed.
You need to give your skepticism filter a major upgrade.
Steve,
If the science is settled then it definitely is an engineering problem – Fred Hoyle wrote about that years ago. Settled scientific theories are no longer researched to become engineering problems.But when billions of dollars are spent and thousands of scientists are employed to solve some sort of problem, here climate change due to CO2 increasing, and if it has not been settled, (Obvious since research is still going on) then the idea forming the basis of the problem has to be wrong. Ie doubling CO2 causes some sort of specific rise in surface temperature.
Just take Venus, Mars and Earth, plot CO2 vs surface temperature and see how straight the plot line is.
BTW Gavin’s problem is that if CO2 is doubled in the atmosphere, then 50 times that amount has to be added to the world’ oceans (Ocean:atmosphere chemical equilibrium for CO2 is 50:1). Tom Segalstad pointed out there isn’t enough carbon sources in the crust as coal and oil to allow that to happen.
Brian
Are you say that science can’t learn any things from business processes; this seems like a prejudice; may be youre even using the same techniques.
Are you saying that science cannot always give an answer ie your geology Metaphor, if so fine admit the limitations/differing interpretations/explanations/concepts; this one of the problems with climate science. Weve been told that the science is settled, where a simple glance at this site proves otherwise.
Sounds like a mission to audit to me. To be honest I cant see youre problem, as its often better to come at something from an antagonistic point of view collaboration has its limitations.
You think youre getting bullies, then get real you chose to blog so please dont complain when you get passionate responses; people on here take this seriously, and in a scientific manner.
Finaly maybe were getting a bit lost with your arguments; perhaps you could distil it.
brian says: (#65):
“Hey, what do I care who wrote it its what comes up under climateaudit.org”
I think you should care who wrote it. It is a big difference when you read what Steve McIntyre writes about himself or his blog when compared with somebody who doesn’t want to inform people about what Steve is doing.
Brian
My Closing argument is this while Hansan-Schmidt entity and the rest claim that climate science is settle and will tolerate no counter arguments, evidence of facts then ClimateAudit is needed. Whats more, any respectable scientist would understand that and approve of this sites aims.
Brian States in 59…
“But, lets get this straight. The purpose of the Climate Audits
audits are to disprove AGW.”
UM, Brian. This is how an awful lot of science works.
We try to demolish a proposition. When we are unable to
demolish a position, we tend to believe it!
Brians underlining indicates to me that he thinks CA is
somehow illegimate because it is critical of AGW. Now first
of all, CA focusses on a small part of the overall s itself.
Science does not work because we hold “neutral” positions.
Science works because the positions we take can be subjected
to hostile scrutiny and successful propositions surive the
scrutinty. The unique feature about parts of “Climate Science”
is a great reluctance to be subjected to that scrutiny, and
to suggest illegimate motives as driving the desire to
scrutinise the pro-AGW position. This makes it more similar
to sociology than to physics.
This is a question for Brian. Do you believe that SteveM
is strictly truthful when he reports his outcomes? Whether
he attacks a topic with a skeptical of credulous attitude
should not determine whether you believe he will faithfully
report on a matter of fact. You can of course decide, that
because he is skeptical of AGW then he is by definition
untruthful, (I think Steves skepticism may be the basis
of his investigations!) but this does not advance the cause
of an open exchange of ideas.
Basically
Jim (PS, I am a scientist)
Could someome list exactly who in the GISS affair is adjusting what and refusing to reveal what? I am not completely clear:
Who manages the stations
Who makes what adjustments in production of what data series.
Is it Hansen only and GISS only that is adjusting raw data, or is GISS making a second adjustment to data that has already been processed, or is GISS just using data that another organisation has supplied them?
Brian seems to have successfully moved this discussion off the main point, which is how to ensure replicability in a non-experimental science. Steve has pointed out the problems that arose in Economics when both data and code were not available and the response from leading journals in the field, which was to require deposition of data and code at time of publication of the article. The big question is why paleoclimatology should be any different, given that Steve has demonstrated beyond reasonable doubt that there are replicability problems in the area. Instead of making a fuss why don’t the journals just get on and do it and what possible justification is there for not doing it?
RE 73. Hostile scrunity. I love it. You should create a blog.
Re #74,
Peter,
1. Who manages the stations?
It may be different in different parts of the world. In the USA it
may be the National Weather Service for coop stations; it may be the FAA
at airports; it may be a military branch at a military base.
2. Who makes what adjustments in production of what data series.
Anybody who wants to, but anybody else may ignore anyone else’s adjustments.
In the US NCDC (National Climate Data Center) some US stations have been
selected to be included in a group called the USHCN (US Historical
Climate Network), and data from those stations are adjusted by NCDC people.
See: http://www.ncdc.noaa.gov/oa/climate/research/ushcn/ushcn.html
for a discussion of Version 1, and a link to a discussion of Version 2,
of USHCN.
3. Is it Hansen only and GISS only that is adjusting raw data, or is GISS
making a second adjustment to data that has already been processed, or is
GISS just using data that another organisation has supplied them?
Hansen/GISS use some USHCN adjustments, and also use some of their own
adjustments, on data supplied by others via the NCDC. Hansen has not,
AFAIK, published the GISS programs which make their station data adjustments.
Having read the forgoing points, I gather that James Hansen is refusing to provide all the information necessary to replicate his work. He gives the excuse that to do so would be too time-consuming or difficult – is that correct? If so, that is an unacceptable position. For a start I understand he is paid by the taxpayer, and as such he should respond to the wishes of the citizens. As others have said, if his results cannot be reproduced independently, then they are not proper science. In the 21st century such arrogance is completely out of place. Surely someone senior to him should instruct him to make his data and full workings available. Maybe the President should intervene!
brian,
Even assuming the google “description” is accurate. So what if Steve has an agenda.
If the “science” is solid, Steve won’t be able to poke any holes in it. If Steve does suceed in poking holes, than all he has done is show that the “science” behind the AGW story, is not solid and needs more work.
What’s wrong with that?
Many of the scientists pushing the AGW story also have an agenda. Just listen to their public statements. Case in point, Hansen’s “Saving Creation” letter. Do you think that Hansen’s well known agenda of saving the world invalidates his work? Or is it only when the agenda doesn’t match your own, that it becomes disqualifying?
Er, is this the right room for an argument?
The exchanges today would make a good Monty Python sketch!
brian as John Cleese
Re #78,
Derek,
FWIW, I’m not awayre than Hansen has published any particular reason for not
publishing the source code of the programs that GISS uses for adjusting station
temperature data.
So what you are saying Jerry, is that he is simply refusing without giving any reason. If so that is absurd. Can’t some senior politician take this up? After all he is paid out of public money!
>> Update from Norway. Every household, starting from 1.1.2008 will be charged on average about US$466 on their electrical bill because of the kyoto protocol.
As a norwegian, this is a horrifying turn of events. Unfortunately, the norwegians are too honest to cheat, like everyone else. How is this billed, yearly or monthly?
In the UK we have a Freedom of Information Act which compels public bodies to publish information that is “in the public interest” Surely in the US there must be something similar. Perhaps someone should talk to a sympathetic Congressman or Senator?
#79. Personally, I think that it is very unlikely that there are fatal errors in all the underlying studies being relied upon by IPCC. I am far less dogmatic than many readers. I’ve also had experience where shameless promoters have also had big discoveries (Murray Pezim and Robert Friedland are two such examples.) So just because climate scientists put mining promoters to shame in their overselling of their studies doesn’t mean that there isn’t some ore there somewhere. However, a skilled investor still tries to separate the hype from the substance – but that doesn’t mean that everything is rejected.
I also thought that it was very unlikely that there was anything wrong with MBH98 when I started looking at it. I tried to replicate it because no one else had and it seemed like simple prudence that some one should.
In my opinion, there are major and fatal errors of varying type in the other key hockey stick studies and none of these can usefully be relied upon for major policy decisions.
Obviously there are disquieting aspects to the temperature record. I don’t think that any of the studies purporting to show the irrelevance of UHI can be relied on; the bucket adjustments of the SST seem ropy. Notwithstanding that, the canonical temperature histories seem “plausible” and it’s far too early to believe that there is necessarily anything hugely wrong with the final answers. However, Hansen’s reaction, like Mann’s before him, seems over the top and is an incentive to more detailed examination of the record.
Steve McIntyre – RE your request to Gavin of the CO2 doubling backup info…did not you get something on that from Gerry North recently? I didn’t recall much debate/discussion about what he provided…but from your continued pursuit (from Gavin now), is it safe to assume you were not convinced by what North provided? (I’m recalling it was North, I could be wrong on that account though).
Thanks.
#57 >> The same cant generally be said for climatology or econometrics. Have you ever seen a paleoclimatology paper that begins: I took two Earths, and doubled the CO2 concentration on one of them
Jim,
I think I generally agree with you about the need for full disclosure, but you are not correct about your assumption that climatology or econometrics being fundamentally different. It’s not true that one needs an entire earth to perform an experiment. For that to even be a possibility, there would have to be something about the hypothesis that involves the whole earth. Some phenomena that only happens on a global scale, and no where else. What is the hypothesis that implies this. No one knows.
Man has formulated hypotheses about everything from the earth dynamo to stars, to the universe, and at no time, did man need to perform an experiment that involved objects of that size.
There is nothing to stop us from constructing a large enclosure that duplicates our atmosphere, constructing a EM source, and then adjusting the C02 concentration from .027 to .054%, and then measuring the temperature difference.
If this isn’t the correct experiment, don’t blame me, since I don’t have a clearly specified hypothesis to work with.
It seems that the Hockey Stick controversy has not been resolved.
From what I read, Mann, et al and supporters are standing by their revised plots that have the same story (sharp, sudden rise in temps with no end in sight). Do you find this to be the current state of that replication/audit effort?
How large a part of the Hockey Stick story is the effect of CO2 levels on growth ring spacings vs temperature?
#89 >> Hockey Stick controversy has not been resolved
Did you expect that they AGWers would let a simple thing like fact stand in the way of their agenda? There have been far bigger repudiations than the one delivered by M&M + Wegman, and they have not been deterred. This thing has been constantly repudiated since 1896.
In Brian we have a lurker who sent this discussion wildly off topic, and I don’t think it is the first time. Reading many of the posts, we are spending time trying to change his mind, which won’t happen, while he sits back and laughs at the manipulation.
I think it is time to recognize that Hansen and Gavin, (and I sure Brian) made the leap on AGW from science to politics and publicity. The challange for the scientists here is to contunue to push for replication – Steve’s post validating the need. Us skeptics have to:
apply the political pressure to have a goverment employee open his work to scrunity
make journalist back up their stories with facts from experts.
The final piece of the puzzle is Madison Ave. Companies whose advertisements discuss AGW as “settled” turn the average TV viewer into a believer who will demand “something be done”. Then free people everywhere will be in trouble.
John F. Pittman says:
August 21st, 2007 at 3:25 pm
From NASA’s quality of information guidelines:
C.2.b.2. Reproducibility. The measure of reproducibility refers to the extent that the
information is capable of being substantially reproduced, subject to an acceptable degree
of imprecision. In other words, independent analysis of the original or supporting data
using identical methods would generate similar analytic results, subject to an acceptable
margin of error.
Each NASA organization will be responsible for determining which categories of
original and supporting data will be subject to the reproducibility requirement.
NASA will make the information it disseminates and the methods used to produce
this information as transparent as possible so that they can, in principle, be
reproducible by qualified individuals.
When it is not practical to apply the reproducibility standard to data or
information, NASA will ensure greater transparency of the methods used to
produce the data or information.
See the full document.
I tried using the tag listing ie You can use… You can’t. So I’m using Quicktags, which produce standard html. I think the tag list needs an audit.
I agree about engineering. It is being able to replicate your results a million times with no failures.
Good debuggers (mechanical, electrical, software) use the scientific method for debugging.
Since the work is for profit it pays well. However, there is then a premium for being right and a penalty for being wrong.
The reason to include engineers is that they know lots of practical details that are useful in making things work or avoiding large costs. Like getting a component to do double duty in a circuit. Something that someone only familiar with the science might miss.
Replication Required.
The ability to replicate is essential to true experimental “scientific” effort and is in addition to peer review which NEVER involves actual replication of the experiment. We re-learn this lesson every time there is a claim (normally veiled) of a perpetual motion machine; more recently with claims of “cold fusion”; and perhaps very currently with claims of “faster-than-light” motion via tunneling; Further, this lesson was first learned, and requirement first adopted, in the Middle Ages and has been around during the most productive scientific era.
Re: 88
Econometrics is fundamentally different. The standard assumption in statistics is that values of the explanatory variables are exogenously determined by the experimenter. Then, the value(s) of the response variable corresponding to these given values of explanatory variables are observed.
In economic data, more often than not, both the response variable(s) and the explanatory variables are determined endogenously within a larger system.
As an example, consider the returns to college education. Individuals go to college if they expect the returns from their degree to outweigh the cost (both financial and opportunity). Therefore, a simple comparison of earnings of college graduates versus non college graduates is likely to overstate the returns to education.
Failure of classical assumptions is the norm in econometrics. That makes both the theory and practice of econometrics much harder than standard medical experiments where the researcher varies the dosage and observes patients’ responses (and, even that is not that straightforward).
In its current state, climatology is like econometrics before problems like endogeneity, sample selection bias and the like were recognized.
Sinan
Steve I agree with an earlier post. I don’t think Gavin can provide it. I have run through 3 good sources and basically the claim of the effect of doubling CO2 is based on multi dimensional, multivariant computer models. The original models were to explain how the geological cycles of optimum versus ice ages occurred. From simple postulates in the the 1880’s to hundreds of pages of caluculations and approximations in the 1930’s to explain this. CO2 was one of the factors used. The CO2 and weather cycles (atlantic and Pacific) were aproximated using more and more complicated models and assumptions as time went on. The most valid complaint that I have able to determine about the latest models concerns the very relationship that another post of yours asked about. Apparently, there are credible scientists who think the present models in particular treat H2O with almost no credibility, and this causes their estimates of the effect of doubling CO2 to be any where off as much as a factor of 3 or 4. As far as I could tell from my research, it was about 1940 when the last strides were made in relationship equations. It has been computer models since.
How about a preview button to insure that data can be checked in final for before publishing?
That would help reduce errors and some multiple posts with minor corrections.
[snip- this is on replication policy. I don’t want every thread to debate CO2 from first principles.]
Philip from Georgia,
Currently cold fusion is in the replication stage. There is enough known to make replication easy. In the beginning it was not known what features of the experiment were important and so there were deviations.
So far the rate of energy production is small and the number of neutrons detected does not match the energy rate if the process is what it is thought to be. More research is required. In fact it is going on.
This brings up the importance of exact replication in the early days of a new discovery.
Thank you very much.
You are perfectly on point with the need for access to data and codes for replication in Science. The very nature of Science is that any experiment must be replicable to be true. The point of publishing is to inform other scientists of your results, allowing them to discuss, test and confirm what you present, and establishing priority should you be right. If there were a patent and money involved, I am sure that he would have that data package ready for the courts. NASA would insist.
As a graduate student I watched many an honest professor stoop to having to digitize graphs to backwards engineer published results that they didn’t trust, because the authors were unwilling to provide the data.
James Bailey – Physicist
Ph.D. Michigan State University 1994
>> Therefore, a simple comparison of earnings of college graduates versus non college graduates is likely to overstate the returns to education.
What’s your hypothesis? Your mistake is trying to use statistics to make a specious conclusion. Economics is a social science, governed by the laws of supply and demand, and human nature, which is complicated, but knowable.
>> [economics] much harder than standard medical experiments
You underestimate the complexities of the human body.
>> climatology is like econometrics
It’s being approached that way, which is a mistake. climatology is more related to thermodynamics, atmospheric physics, electromagnetics, chemistry, etc. All hard sciences, unlike the social science, economics. As such, a statistical approach is unprecedented.
JFP,
RE: 97
Here’s a couple of papers I found yesterday that seem to look at CO2 from (near) first principles. Both seem to address the problem using one infinite element. 😉
http://icecap.us/images/uploads/Falsification_of_CO2.pdf
http://www.ecd.bnl.gov/steve/pubs/HeatCapacity.pdf
Brian
Sorry to jump on the pile but you are misguided in your thinking. Auditing and replication are FUNDAMENTAL to all business management and scientific enterprise. I could start with Health and safety and go right through software and hardware reliability and on to my final job before retirement, through to major project management. It is impossible to avoid audit and replication in any situation where the requirement demands peer and or public review. There is simply no credibility in producing anything without being able to validate, verify and give ABSOLUTE proof of provenance.
To use climate science language we need a forcing agent to compel Mr Hansen to reveal all his methodology. I suggest that should be via your legislature.
#88 Gunnar,
To roughly approximate the atmosphere in the experiment you would need to do it at very high pressure to insure the traverse of the incident photons through the same number of atoms/molecules as in the atmosphere.
The pressure ratio is (BOE) roughly the size ratio of the height of the box to the height of the atmosphere (say the height at .5 atm.).
That should give the heating potential (assuming you properly account for conduction). Then you have to go after the heat transfer by water vapor or as I prefer The Big Heat Pipe In The Sky.
This would not be a simple experiment (aside from the pressure question). Lots of factors would have to be evaluated i.e. conduction vs. pressure. Temperature gradients in the atmosphere. The temperature to run the experiment. i.e. do you allow water vapor condensation on the cool side? etc.
#102 Gunnar,
You have to count the opportunity costs of the money and time invested in college.
Other rewards (prestige) are some what harder to quantify.
Since NASA is a US government agency, does not the Freedom of Information Act apply? Cannot a FOIA request be submitted for that data and model?
A valid request and release would get to the bottom of it. I don’t believe they would be exempt. A redacted return would make such a bit of a laughing matter.
#83 Derek,
Congress is cognizant of what is going on. At least the Republicans are paying attention. Blogspot is down now or I could provide a url.
I was able to get it from the Google cached version:
http://epw.senate.gov/public/index.cfm?FuseAction=Minority.Blogs&ContentRecord_id=84e9e44a-802a-23ad-493a-b35d0842fed8&Issue_id=“>Congress Looks At Climate Science. It appears that the .15C error in the “splice” has caught Congressional attention. I expect once this is further evaluated laws or changes in laws will be proposed.
Thank you George – is there a lawyer on hand who can answer the question?
Hmmm….
First Brian claims that Steve said that CA is here to disprove AGW, and that makes him a too subjective to audit the science.
Then, MrPete shows him that the description was written by an anonymous editor at DMOZ.
Then, Brian says it doesn’t matter who wrote it.
Rather clumsy attempt to move the goalpost, if you ask me. Brian, you’re not as smart as you think you are.
RE #88,
“There is nothing to stop us from constructing a large enclosure that duplicates our atmosphere, …” 🙂 🙂 🙂
88, there’s plenty to stop us. A number of things don’t scale up, including gravity. If you can’t reproduce the exact temperature and concentration profiles in the scale atmosphere (and you can’t), you can’t build a scale earth. If that were that easy, I’m sure someone would have done it already.
The basic CO2 greenhouse effect can be computer modeled to a degree of accuracy that most people would consider reasonable. The feedback effects, otoh, are a WAG. They aren’t even sure of the sign!
#63 MrPete says:
In engineering it is called a design review. Normal practice in aerospace is to have a presenter(s), his support organization, and specified auditors plus any one else who wants to snipe and has the time. i.e. if you have an interest or expertise come on in. This costs real money. It is done that way because we have a strong aversion to aircraft falling out of the sky. In climate science the current bias seems to be the sky is falling. No matter. An audit (design review) will get to the bottom of the facts.
Here is a link test (I wish there was a preview);
Non-standard methods (re: html) cause problems.
Congress Looks At Climate Science.
What an idiot I am. There is a preview box below the comments.
BTW standard html works. The Link Quicktag does not.
#106 >> you would need to do it at very high pressure to insure the traverse of the incident photons through the same number of atoms/molecules as in the atmosphere
Can you provide a specification of the hypothesis? You could be right, but I’m not aware of a AGW hypothesis that makes it clear that we need to traverse through the same number of molecules. We don’t need to duplicate the complete atmosphere. Only that part which the hypothesis indicates is crucial for this specific effect. We need to isolate that effect.
#106 >> This would not be a simple experiment
I didn’t say it would be simple. However, there are billions of dollars available. The scientific method requires that we clearly spell out a hypothesis, and then construct an experiment.
#106 >> do you allow water vapor condensation on the cool side?
Does the hypothesis indicate that this is required for the alleged affect to take place? I hope you’re getting the point. Based on my knowledge of the hypothesis, which I believe ignores water vapor altogether, we can leave that out completely.
#107 >> You have to count the opportunity costs of the money and time invested in college. Other rewards (prestige) are some what harder to quantify
Exactly, which is why a statistical approach is not correct.
Someone tried a large scale experiment a few years back:
Biosphere
103, that report deals with the first principles of the greenhouse effect in passing, but that’s not what it’s about. That’s actually a clever way to solve a problem that’s been dogging modelers; how to estimate the thickness of the layer in the ocean that interacts with the atmosphere. He inferred the thickness of that layer by regressing temperature lags.
That, in turn, allows them to complete the climate sensitivity model. That’s why that study is groundbreaking; it used a lot of data to estimate that important parameter, which in turn allowed him to get a fairly good handle on the climate sensitivity number (including feedback!). What has the AGW crowd upset is that the sensitivity number is less than the low end of the IPCC range.
RE 115
The Link “button” does work, you just have to play around to understand it peculiarities.
Gavin discusses climate sensitivity here:
http://www.realclimate.org/index.php/archives/2007/08/the-co2-problem-in-6-easy-steps/#more-462
and there is a wealth of info in the recent IPCC report on forcings and climate sensitivity here:
http://ipcc-wg1.ucar.edu/wg1/wg1-report.html
The potential for public humiliation is part of science and engineering. AFAIK engineers/scientist do not take courses in how to deal with that aspect of the job.
I’m quite comfortable with public humiliation (look at the bit on Preview I just did – LOL). You pick your sorry a** up and resolve to do better so the odds of it happening again are reduced. Pay more attention. Be more careful. Study the science more carefully. It is a Zen discipline. Those good at it are Masters. You find a lot of those in engineering.
Steve, my favorite Einstein quote is appropriate:
“The right to search for truth implies also a duty; one must not conceal any part of what one has recognized to be true.”
Would Einstein have refused to reveal his calculations on relativity to non-peers, simply because he believed (but did not know) they couldn’t handle the equations? Not on your life.
He also said about a large group of critics that “it doesn’t take 100 scientists to refute my work, merely a single fact.”
#116 Gunnar,
I was trying to show what the difficulties might be. So I agree with your points on the gas experiment. Unlike aerodynamics the scaling laws for atmosphere experiments are not well worked out.
Re: the economics of college. I think you have to start with a statistical evaluation of the population. Then you have to decide if for the purposes of your study if the median (i.e. individual oriented) or the mean (i.e. collective results) is important. I do not think you can leave statistics out of the problem.
I think you need statistics in climate science to tease out the various effects (second order if not first) because you can’t do an experiment where you only change one variable.
The question is then: is the statistical method chosen likely to answer the question posed? This argues for openness of method as well as data.
#96 & 102
Surely the issue is how well specified the explanatory model is. In the established sciences the models tend to be much better specified than in the social sciences, i.e., all the relevant variables and interactions among variables are defined or can be experimentally controlled for. In emerging research disciplines like climate science these variables and interactions among variables remain poorly or ill-defined or uncontrolled for. The use of statistics as an analytic tool becomes more problematic with less well specified models.
More critical, however, and more on point for this thread, is that replication is essential for moving a discipline forward by helping to better specify the theories and models we are dealing with. IMO Hansen and others who decline to provide access to their data are doing significant damage to their disciplines.
I would like to make one additional aside: The issue of whether any contributor should be listened to here or in any other forum should not be based simply on their credentials. Credentials like advanced degrees in a relevant discipline can only be a moderator variable tin determining the quality of a contribution – they are neither necessary nor sufficient to produce a valuable contribution. The value of the contribution, and I believe Steve and others have made significant contributions, is based on whether or not their findings are replicated and trigger additional research.
Long-time lurker weighing in …
As background, I’m not a scientist; I’m an accountant. My career, like that of many accountants, began in auditing.
In my professional experience, the second any audit topic starts down the Wizard of Oz path (i.e., “pay no attention to the man behind the curtain”), further investigation almost always leads to detection of issues which may range from fraud (at worst) to sloppiness on the accountants’ part that would be reported to senior management.
Furthermore, audits of complex, black box areas require the most effort and rigor. If I had an audit client with complex derivatives, for example, who would not let me see the contracts or their calculations of the balances reported in their financial statements, they could not get a clean audit opinion. I believe this is akin to what is happening in climate science today, where trillions of dollars of economic value are proposed to be committed to a dubious endeavor based on unchecked assumptions.
And before anyone says peer review serves this function, I have one word for them … “Enron.” Enron was “audited” by people who were also cheerleaders for the company (with apologies to the diligent, competent auditors I knew at Arthur Andersen on other engagements). Enron is a vivid example of what happens when a system that relies on independence, objectivity and a healthy dose of skepticism, in this case between audit client and auditor, breaks down.
This is, unfortunately, the state of climate science today, where AGW cheerleaders get research grants while skeptics are equated to holocaust deniers. There were a few “deniers” at Enron and Andersen, too.
Enron, Worldcom and the rest brought my profession Sarbanes-Oxley, which at its core sought to restore the wall of independence between companies and thier auditors. Maybe climate science needs its own version of Sarbanes-Oxley.
Brian:
Welcome. The one thing Steve does here that is different from RealClimate is that editing and essentially banning people are limited to those who are rude, crude or perennially drive a thread off topic. We have seen a few of each – hence a certain level of sensitivity.
So my questions to you are: Should Hansen post his code for adjustments? Should Mann et al post their data?
Should Jones post his data?
#120, and in regard to the link you provided to RC and to Gavin’s post..the best comment there is this one (after many posts already with long and complicated explainations and questions):
“It seems like quite a number of posters here are looking for an easy-to-understand explanation for why increasing CO2 is a significant problem. Perhaps someone would be happy to oblige.”
LOL
#125. Andy, as someone who’s at least somewhat knowledgeable about both company accounting and climate temperature calculations, the procedures involved in calculation of Hansen’s temperature index are really a form of “climate accounting” done by people with no training in accounting systems – thus, the non-existent or execrable audit trails and their failure to even appreciate the importance of audit trails. The vicious tone of Mann’s response to criticism was worthy of Andrew Fastow.
RE: “The whole purpose is to demonstrate that AGW is false. It says it right there. ”
Wrong. The purpose is the accurately characterize AGW, and its actual impacts. Is that too much to ask?
RE: #61 – Wrong. My purpose is to shatter your bogus argument regarding geology being some sort of touchy, feely inexact, non auditable science. You are just plain wrong. Wrong in #61, and wrong all around!
I think these are important points in maintaining credibility for those who would criticize some climate scientists for sloppy work and/or merging political and scientific issues. Overreaction and generalizing are major detractors from the discussion of those issues here at CA and they draw even more distractions from those who come here with consensus POVs when they zero in on the over reactions instead of discussing the specific issues at hand.
I personally have a very dim view of the effectiveness of the likely attempted government mitigations of any potential adverse effects from AGW, but at the same time feel that there remains much uncertainty in determining what those adverse effects might be. I have a very wide minus and plus on the probability range of these effects.
Jim Hansen, does seem to me to be a scientist who has a well known and advertised political plan for AGW and while that by itself says nothing about his scientific conclusions or proficiency, it does give me pause when he uses his scientific position to proclaim confidences of future climates that have much more to do with his political stance than any statistical processing of the current state of climate science can demonstrate. I have the same problems with the IPCC likelihoods and uncertainty proclamations.
If one had Jim Hansens confidences in government mitigations one might find reasons to be more confident in the more adverse climate projections. I find myself at the other end of the political scale and I am sure that it effects what I am willing to except in terms of subjective confidence levels. Whereas I doubt that Hansen would concede much at the less adverse climate effects scale, I as a layperson am not at all certain about where the more adverse climate effects scale lies.
I do think that some of the sloppiness observed in climate scientists work can be attributed to their hurry to get their message out to a scientific and political world.
RE: #97 – You have raised a key point. The initial, underpinning studies regading the “Greenhouse Effect” and the role of CO2 in regulating climate, were done against the background of geological uniformitarianism, by people who did not realize that Antarctica has not been in its place foe eons, and that the closure between North and South America was not long in terms of geological time prior to the onset of the Pleistocene. Had plate tectonics knowledge been part of the underpinning, how might the train of thought have differed from what we actually ended up with?
RE: #125 – Bravo! Well stated!
#130 Steve,
May I add, if geology is so inexact why do mining companies and petroleum companies employ so many of them?
BTW semiconductor manufacture is not an exact science. There are many interactions and despite what some scientist may say, plasma physics is not an exact science except in very limited circumstances. I’m dealing with that very problem in the design of a fusion reactor. Some physicists believe you can derive a general equation for the problems involved. Others believe the science must be applied to the exact conditions involved and that there is no general solution or even worse that only experiments can provide a solutions to a given set of conditions.
It may very well be that climate can not be solved by a general set of equations (in a reasonable time frame). Thus gathering data and trying to figure out what it means may be the only way forward. Thus the importance of quality data and the exact methods used to interpret it.
I’m in two minds as to why the source code for temperature calculations is being held back. On the one hand its owners are correct in that it will be examined in detail for the tiniest flaws, adding little to the debate. On the other, maybe they are just embarrassed about the coding style and the poor documentation (having looked at ModelE code, that wouldn’t surprise me).
But perhaps someone will tell me. Is there enough data around to do a “climateAudit” temperature reconstruction?
After all, the requirement to be able to reproduce a result is not the same as the requirement to copy. Often you learn more by doing the same thing in a different way.
Coming from a modelling background, I know full well that it is completely unrealistic to publish all the data and methods in the way suggested here. To reproduce the result you need either a huuuuge amount of data, or you need the program plus a supercomputer. A line would have to be drawn somewhere between the small temperature and temperature-proxy datasets and bigger model datasets.
Within the modelling community “reproduction” is done by having many models and hoping that they all produce “similar” results given the same inputs. That said, individual models are audited to a degree both internally and externally, eg. being free for use by academics.
135, nonsense. There’s no modeling involved here. This is simply some statistical manipulations on some data. You can do it on a 486.
Even the models that the IPCC does, which involve numerical integration, are doable on a decent desktop pentium. Today’s desktop computers are an order of magnitude faster than the Cray supercomputers of a generation ago.
I came across this while Googling. The Guardian newspaper (UK) did a survey of scientists to ask “What is the one thing everyone should learn about science?”
One wise response was:
“I would teach the world that scientists start by trying very hard to disprove what they hope is true. When they fail, they have a good reason for believing what they hope is true, and can even convince others of its truth. A scientist always acknowledges the possibility of error, and is less likely to be mistaken than one who always claims to be right.” (from Antony Hoare Senior researcher at Microsoft Corporation)
This represents one type of scientist. Another is the scientist who develops a theory that he becomes convinced is true before it is fully tested. He then looks only for evidence to support his theory (and ego) instead of maintaining appropriate openmindedness and self-honesty.
Guardian link:http://www.guardian.co.uk/science/2005/apr/07/science.highereducation
That would be because they are not considered part of the evidence that your drug works, merely the basis for your considering that it might and to begin the investigation process. They really aren’t interested in your inspiration.
That would be because they are not considered part of the evidence that your drug works, merely the basis for your considering that it might and for beginning the investigation process. They really aren’t interested in your inspiration.
Pardons for the double-post. When I tried to submit, it gave me a whole series of errors, and then a resubmit (with grammar correction) resulted in my having the “Madness of King George” thread displayed instead of this one. Odd.
re: 135
The bug that Steve McIntyre discovered was not tiny and it went undetected for 6 years. The code that genereted that bug could, as well, be described as trivial.
Code auditing is normally called V&V. It is standard practice. However at NASA probes are sent to Mars with memory leaks in the operating system, critical components interacting with different systems of units etc. So I would guess that the sort of bug that was detected by Steve McIntyre would not be described as entirely unexpected given their history.
#136 Larry
Clarification – I was saying that you need to draw a line somewhere because carbon-copy replication of some science is too expensive or pointless.
(The models of a generation ago are definitely not very interesting now, and current models won’t run on a PC very quickly at the required resolution. But that’s off-topic).
Brian. If you ever come back, please be less transparent.
Your #3. The discussion isn’t about understanding complex systems. Or what’s “the same” as something else. It’s about archiving data and methods for replicating something, either experiments or procedures or whatever. Whatever field the replication is in and regardless of the specifics. Obviously, code validation is different than manufacturing process, but . Mixing up different subjects in #3 as you did tells us one of three things; either you don’t know what you’re talking about, you’re misunderstanding what’s being said, or you’re here to cause trouble. Maybe there’s more I haven’t thought about, but you hopefully get the point.
In #30 you said “What im saying is that the business-style auditing may not be the best approach to this problem.”
No. What you said in #3 was far more agressive and hostile “Your whole approach is to apply a framework that wont work, that doesnt apply.” “Yeah, so?” And the granddady, the ulterior motive assumption: “The beauty is, you can then turn around and point out perceived shortcomings of science as a function of your misapplied approach.”
You said you are in academics. Never had another member of the faculty check up on you or your classes? Get graded as to performance by the management of the educational facility? Never had your syllabi looked at or your own continuing education plans checked as to if you were doing them or not?
Ever check your car to make sure the mechanic actually replaced your oil filter?
Let me see…. This non-sequiter in #19
I believe you are saying Steve is acting like an engineer trying to replicate the process of science. No, he’s a statistician with experience in mineral exploration and training in politics and economics that’s using mathematical analysis to verify the results of science, when the materials to do so are available. Either because they are archived as they should be, or those involved provide them.
Then there’s the way you change the subject in #34 (from a Goggle hit, give me a break; a 10 second search would have shown where it’s from), mentioned again in #59, is not actually on this site. It’s a blurb in a directory listing at DMOZ as has been pointed out. Because there’s a link on DMOZ, Google brings it up, and brings up the link to here. It’s just the Internet.
Verifying that “what is said” is “what actually is” is the point. I would think that would help show “how human induced global warming does add up and to what extent” rather than “how it does not add up”. Steve says it very very well in #86 what he thinks about it all, forget about some Google hit that doesn’t apply.
So, in the end, I ask, rhetorically and probably for nothing, why isn’t investigating statistical methods in a paper using statistics (an “audit”) valid? Why is replicating station adjustments with the same data and methods as originally used, why is that in no way at all even nearly similar to making sure the accountants did the math correctly and that that the balance sheet is correct the same way the company did it, or even checking to see if they even used GAPP in the first place? Of course the areas not “THE” same thing, and wouldn’t use the same methods, but as to purpose?
To make sure what is said is what is actually the case.
Back to real life.
That’s a very interesting point Ian, in #29. How many visitors that are confrontational to “everyone” actually believe this is about anything other than verifying that “what is said” is “what actually is”? Why does there have to be some sort of strange ulterior motive to validating the implementation of an algorithm itself, that it does what it says it does? Or to see if high-quality stations meet standards? Or to check that robust reconstructions exist in the presence of bristlecones?
This one started back in #50 and has had a bunch of comments, I think “Have you ever seen a paleoclimatology paper that begins: I took two Earths, and doubled the CO2 concentration on one of them ” started it. In my opinion, any large-scale experiment to physically replicate major portions of climate is impossible. There’s too many unknown factors and unknown relationships. How do you create a controlled experiment for something chaotic, not well known and unpredictable? That’s why they use models.
For those of you that asked about what James Hansen said about not giving up the code (#82, #78 etc) , I don’t thihnk he’s said anything specific (anyone got something?)
But Gavin Schmidt has. Here’s some of the stuff he basically said were the reasons for not giving up the code, over at RC.
But we give out a lot of other stuff.
But people always want more.
But it won’t run out of the box.
But we don’t have the time.
But you haven’t proven the need.
But there’s no reason to think it doesn’t do what it says it does.
But the papers describing what we do are enough.
But the dumps are impenetrable and undocumented.
But algorithms not in code or pseudocode but described are more valuable.
But the results prove the methods.
But you’ll learn more doing it yourself.
But it’s already been proven it works.
Which leads me to infer:
But we’re not giving anyone the code.
142 Steve, it’s on the fringe of on-topic, because Steve M. asked the question of NASA, where does the 2.5C number (from CO2 doubling) come from. That does get into modeling issues, but the modeling was done by various parties under the IPCC umbrella. I think we know that NASA derived that number from one or more of the studies that were referenced in the IPCC report, the question is, how exactly did they get from that range to that specific number.
I suppose that by asking the question, there’s an implication that he may want to audit that calculation, and if so, that will lead back to the IPCC, which is unauditable (as a UN agency, they answer to no one), but the individual chapters may be subject to FOIA.
But you have to unravel it a step at a time.
Btw, I wish we had the British law in the US, where all taxpayer-supported academic research is subject to FOIA. That would crack this right open (and the IPCC would probably use a lot less US research as a consequence).
#88, 102, 116 Gunnar:
I differentiated the practice of much of climatological science from other areas of science by noting [#57]:
You said:
then:
and:
It sounds like you’re in agreement with me, then. If climatology were being practiced like other sciences, we would “clearly spell out a hypothesis, and then construct an experiment”, but instead it’s “being approached that way [like econometrics]”, which, you say, “is a mistake.” The “statistical approach” which has been used by Hansen, Mann, and the entire Team of spaghetti-graph makers is “unprecedented”, you tell us.
So how is the practice of much of climatological science not “fundamentally different” from that of the other “hard sciences ?”
re#143
I submitted a comment a couple hours ago now…but has yet to come up, got lost in some cyberspace error, or was blocked. I can’t imagine it was deemed inappropriate, since the myriad responses to me are still coming up.
It was long, and i can’t rewrite it exactly…the essence was that my #3 comment was meant to get a thorough response. And as I’ve said above there are numerous cogent arguments i’m still reading. Stop painting me as the monolithic ‘enemy’ or opposite ‘side’….i’m merely being skeptical of the approach advocated here. And, i’ve learned a lot from some of your comments to that skepticism (and try and filter out the cheapshots and out-of-context retorts/insults). Re the google thing, please see Steve Moore’s response to me (#67).
Also, for people like me who are not engaged in the minutia of the auditing, it would be great to have a list at the top of CA of “papers/studies currently being audited” and/or “persons/institutions currently being audited”. This would help, and you wouldn’t have to waste time responding to me.
144 Sam, I think it goes without saying that Hansen et al don’t think that their work should be scrutinized by anyone outside of their tribe. That was the point of the usufruct/gorilla rant. We don’t have time for jesters, because the sky is falling. No surprises there.
137, Years ago, Dixy Lee Ray said essentially the same thing. Something along the lines of “a scientist develops a theory, and then tries as hard as possible to break it”.
I don’t see that going on with Hansen and co.
#147
We love you, Brian.
146, Maybe I’m stating the obvious, but that’s the nature of the beast. Certain scientific experiments can be done in double-blind fashion in a controlled laboratory experiment, and produce results that someone else on the other side of the world can verify by repeating the experiments. Other fields, such as paleoclimatology (or paleo-anything) are exercises in historical interpretation, and involve trying to match a theory to an inferred proxy data set that implies a variable of interest. You can’t repeat that experimentally, because to do so would require taking new ice cores or bristlecone cores, or what have you, and the data sets will be slightly different.
To get some sort of outside validation in that situation, you can do two things; either repeat the data reduction (which means publishing the raw data and the source code, and see if you get exactly the same result, or collect new data, and see if you get approximately the same result.
Obviously, the latter isn’t a realistic option for an outside auditor without a substantial grant.
#146 >> It sounds like youre in agreement with me,
Jim, you’re right, sorry for the confusion on my part.
>> how is the practice of much of climatological science not fundamentally different from that of the other hard sciences ?
Ahh, I went too fast, missing the word “practice”, and thus the whole point. My bad. Yes, you’re right, the practice is fundamentally different, and it shouldn’t be, since as you say, it’s a hard science.
If a scientist does not release his data, working processes including source code, and full results to be scrutinized by potential critics, what they are doing cannot be called in any way be called science. I suggest from now on you all refer to the activities of Hansen, Schmidt et al by the correct terminology–incantations.
Scientific progress is made because of the process of falsification, when falsification is prevented due to the use of incantations in place of science, you have stagnation.
>> paleo-anything
Larry, you’re right about paleo being more historical than hard science. However, a claim is being made that there is AGW in the here and now. As such, the science to determine that is a hard science.
But it’s not an experimental science! It’s not hard in the physics sense, and not soft in the sociology sense. It’s somewhere in between. Maybe we can call it a firm science.
Re # 106.
Something like this has apparently been done at the University of Chicago MODTRANS facility and the resultant absorption graph can be found here
http://www.webcommentary.com/asp/ShowArticle.asp?id=websterb&date=070601
Makes room for thought, doesn’t it?
Hansen’s “lights out” seems so unprofessional, a bit like the last gasp I see when sometimes people who are being investigated are being held to account, they throw their toys out the pram, how dare we ask questions? We must be complete idiots to questions them!
I’m not a physcologist but when some people commit fraud they play the part , and beleive they are innocent in a sense that whatever is there for the taking is rightfully theirs and if they stretch facts a bit or lie thats OK because its only details.
I’m not a scientist but I did A level physics and maths back in the 70s, and it was always stressed to us how rounding up or down errors could skew results. Also transparency of scientific process was always stressed.
Certainly the process of taking legal cases to court is a gruelling experience, where you have to be open to the weakness in your case, the possibility you may be wrong , and completely transparent about how you gained evidence.
Also deliberately concealing, or witholding inconveinent evidence could end up with severe criminal action being taken against one.
In view of what some of the proponents of AGW propose and the cost to the ordinary man in the street I beleive we should demand a high level of proof.
I applaud climate audit for their quest for truth.
If someone could put decent evidence in front of me that climate change was destroying the world and man made, that I could not challenge easily , I would be camped out at Heathrow trying to persuade people not to fly.
RE 147.
Brian, hang around. one of the BEST threads we ever had was the Parker UHI
thread. One fellow took it upon himself to defend the parker paper in depth.
He ran the gauntlet. In the end we composed a list of questions for the author.
He respnded. I don’t know if anyone’s mind was changed but we all learned something
So, you hang around! make sound arguments ( and you get the occasional
chance to rant) And folks will engage you. ( sorry about the troll stuff
but you survived)
You mentioned migration of species. old hats here can corrrect me, but we have
not had a thread on that metric. That could be fun.
Can we infer GW from species migration? or is it YAP. Yet another proxy.
Link some of your work. If you strike a nerve folks will be all over it.
If its boring the C02 crowd will take over the thread. ( sorry guys)
Its not hard in the physics sense,
and not soft in the sociology sense. Its somewhere in between. Maybe we can call it a firm science.
Brain, re:147
When making a long post, why not do it in a text editor and then edit, save and post. That way you could reproduce your “results”
#135 “But perhaps someone will tell me. Is there enough data around to do a climateAudit temperature reconstruction?
After all, the requirement to be able to reproduce a result is not the same as the requirement to copy. Often you learn more by doing the same thing in a different way.”
Sure there probably is enough data around to do a independent climate model, but without starting with the same data, how will we know why we’re getting different results? Now if we start with the same data and we get different results, we can compare notes and find out why.
re#158
I don’t remember doing that…
This is good to hear…if any of you have gone into a heavily-guarded thread, it’s difficult respond to real discussion while having to deflect insults (or trying to ignore them). The other day I got called a ‘capitalist pig’ in an ultra-partisan enviro thread and a ‘commie’ in an ultra-partisan right wing thread…all in one day.
My own scientific research is irrelevant to this (i.e., it would be off topic)…I am engaged because I care about science and want to hear all angles and arguments.
Brian. Then you simply went about getting a through response in an incorrect manner at first. (Or actually, you got a through one, just not the type you expected perhaps.) Not a problem.
I don’t consider you or anyone here some sort of enemy of any kind, monolithic or otherwise. Just when you say “I am for reproducibility, scrutiny, anal-retentivity, and so on in the ultimate goal of improving our scientific understanding.” some of us are perplexed due to some of the other things you say (like when you said “it doesn’t matter who wrote it” in regards to what they wrote at DMOZ). Or confused as to how seeing if something does a) what it says it does by doing that exact same thing yourself the exact same way and/or b) analyzing the thing that does it in detail with a variety of methods (e.g. statistics, recalculations, general state) isn’t doing that. The stated position of improving our scientific understanding.
I can, and have, “argued” the “other way” both here and elsewhere. (http://www.climateaudit.org/?p=1885#comment-129135 for example on what would lead somebody to correlate human activity with warming) It’s just there’s three groups of people, rather. Those that go too far in one direction or another in their conclusions (or the strength of those conclusions?) and those that are more like me, in the middle.
(For example, if somebody tells me that glaciers are melting, and I go look at discussions and data on mass balance levels and movement categorizations. If I find we have no idea about most of them, and the rest either a) had the number with a positive overall mass balance double from 2001-2005 b) had more than half do nothing at all, that’s what I find. That isn’t saying “it’s not warming” (since I think it is at least somewhat) it’s just saying “I checked the data and some are melting and some aren’t, your unqualified statement is too simple to represent the actual state of ‘the glaciers’. etc.)
I just happen to think that making sure things work as we’ve told they work requires open access to data and methods. There doesn’t have to be more than that to it. And it works, it seems; Steve “audited” some of the information coming out of the effort to verify if the stations are high quality or not by looking to see if they meet siting standards, and found a data error that’s been going on for 6 years. It couldn’t be seen without validating things and comparing inconsistencies.
I think you were just miscategorized things. Also, my point’s been that for most here it’s not if somebody disagrees or not, it’s about how they go about doing it. I realize that #67 (about DMOZ) pointed it out, I hadn’t seen if you’d acknowledged it or not. It’s an easy mistake for those unfamiliar with some of the peculiar behaviors of search engines. But again, it’s perplexing when you state it doesn’t matter who wrote it at first. Perhaps if you do remember it’s how you go about disagreeing, you won’t have to filter out the cheapshots and out-of-context retorts/insults because there won’t be any? Then after you finish reading the numerous cogent arguments, you’ll have a better idea that we are basically what we say we are. Again, I think Steve’s position is pretty well described in his own words here on this topic and on other comments he makes.
#108
NASA in its Procedures has ruled that code and even programs are not subject to FOIA. They can be asked for though from the FOIA office. They will forward it to the head Engineer. There are other Procedures and OMB regulations in response to Federal law that could yield results in the future. These I have posted on other threads.
As it applies to this thread, the DQA (data quality act) and a new OMB regulation concerning data archiving and otother issues are indicating that it may not be long before it is federal law that Hansen, etc provide this information for public and for audits. The recent laws do not necessarily have audit functions, but do contain language that the OMB has to provide a way to ensure that Federal Agencies meet these laws and requirements. SInce OMB is composed of many accountants, I expect an audit requirement once Congress gets wind of more errors. Especially a change of .3 that meant a whole new paper was published, and yet a .15 change has been called meaningless. I believe our congress can do better than simple math and will not be impressed. As that many of them have had to be audited in past and present [professions, I expect they will expect OMB to implement audits.
Gunnar said:
No, actually that’s a description of weather. Climatology is the study of weather averaged over decadal, or even multi-decadal time periods. As a result it’s really an observational science and as such statistics will play an important role in studying it. This is not really that novel, Astronomy and Astrophysics are two other branches of science that are mostly observational and commonly rely on statistical approaches.
Sorry about that Brian, my post in #162 was in response to #147 I guess I’m saying nobody will attack you if you’re reasonable was my point.
Larry, from #148, that’s why I tried to distill Gavin’s reasons he gave at RC for not providing it. Not much of substance to me, I consider them excuses for “we don’t want to so we won’t.” Like you said, no surprises, at least not to me. I’m still disappointed we even have to be discussing it.
As far as “migration of species” I think wildlifer used that in another topic (Does Hansen’s Error “Matter”?) as evidence it was warming that was non-CO2, I forget in what context.
Paul Penrose is right in the context of observation; however climate science has modeled and theoretical components where a statistical approach is useful but limited. Mostly due to the complexity of the global climate.
But thats not the point as AGWers rely on observation to prove their case, which is where Steve has successfully shown flaws. For information on the modeled and theoretical issues I would recommend http://climatesci.colorado.edu/.
RE 161.
sorry my bad. Conflated you with another chap.
Does anyone know of a comprehensive review of the various temperature data sets that are in use by the climate research community? I’m aware of USHCN, USCRN, GHCN, and some proxies. What else is in use? A list, descriptions and links (when available) of all available the data sets in one central place would be very helpful.
Steve: I’ve been collecting URLs – look at Station Data under Pages at left. There’s a need for a clear summary.
re: # 158 Moshman,
You said:
There’s been some discussion of the subject when looking at treelines and the changing northern limits of species of trees. Probably searching on those terms will turn them up.
“You mentioned migration of species. old hats here can corrrect me, but we have not had a thread on that metric. That could be fun.”
Along those lines and in conjunction with Pielke’s work, it would be fun to see Gallo’s data on “first crush” dates and tonnage by date. Grapes are picked according to sugar content, which is sun (and heat) dependent. If it is actually “hotter” in the Central Valley then the harvest should be starting earlier.
In the same vein, I believe that a hypothesis that the “best” temperature readings would come from stations at ag colleges is worth a test. Nobody cares more about the weather than farmers.
Brian,
RE #67: I hope you weren’t including my comment in the “cheap shot” category. It certainly wasn’t intended as such, and I apologize if it was taken that way.
(Actually, it WAS a shot at Wikipedia, but…)
Brian:
I will try again (#126). The key questions associated with this thread are: Should Hansen post his code for adjustments? Should Mann et al post their data? Should Jones post his data?
It would help the discussion a lot if you would give your thoughts on each of these. Note that the
Replication Policy is independent of the question of whether an audit would be fruitful or not. You have been quite clear on the latter.
Jim, #168 – let me clarify something. I may not think that a completely rigorous model can be built, but I completely agree that the billions spent to date could have been a lot better spent, and we should have much higher quality science than what we have to date. Instead of having a plan to investigate the issue, we had a plan to hand out money, and then write reports periodically, with non-scientists writing the summaries first, and then fitting the “science” to the summary later.
So we end up with inconclusive science, and people like Hansen shrieking that we don’t have time to be sure. Don’t blame society; we paid plenty. Blame the insiders like Hansen for mismanaging the enterprise.
Re#177 – I rarely quote wikipedia, and never for scientific/technical issues. That’s something we can all agree on. And no worries confusing me w/ someone else.
I made this comment earlier:
Can anyone point me to a ‘high-level’ document or page within this site for this? An executive summary, for lack of a better term? What are the relevant papers being audited, what are the outstanding issues, etc.? I can certainly get that by doing my own digging, but CA could help itself out in the long run by providing such a 1-2 page report. If it exists and i simply haven’t found it, then please point me in the right direction.
RE 162. Sam I think this is more than once this scientist versus Engineer thing
has come up. The latter being schooled in having their cavities probed by the crooked fingered,
ragged nailed inspector , and the former perhaps being more accustomed to the
gentle persuasions of peer review.
184, but the cavity probe does tend to make you pay more attention to the details the next time.
In fact, for consulting engineers, there’s one thing far, far worse than the design review: the contractor with $change order$ in hand. When you have that, you realize that the reviewer was doing you a big favor by ripping you to shreds.
RE 181.
Brian. Just click on home. then click on the threads.
It’s not hidden. as gavin likes to say ‘RTFR’
Geez sorry again, my #182 was in response to some of the discussion on migration (and general other stuff on A=Z without using the rest of the letters in between! 🙂
#181, I don’t know of any such thing either. I just usually read the articles here and post, and glance at stuff on Pielke Sr, RC and Watts.
#183 Right. Something like teaching how to drill for oil with only academic or indoor (or limited outdoor) experience is far different than drilling for it for a few years yourself at 10 different aspects of it and the pain that goes along with it.
Oh, for my own #182, here’s the short version of the case for assuming that it’s already warming at the get go of the exercise:
These three areas:
a) burning fossil fuels (co2)
b) number of humans and number of animals humans use for various purposes (heat, water vapor)
c) land use changes (cities and roads and farms and forests) (heat, water vapor, wind patterns)
Result in the constant addition of water vapor and carbon dioxide into the atmosphere (and to a lesser extent, methane and ozone) That, combined with the various man-made heat absorbing materials and the effects of both of these upon the climate system in general, is probably responsible for the average global warming trend upwards weve seen since we started tracking things with thermometers.
Whoops, b) should have been (CO2 and methane)
>> Therefore, a simple comparison of earnings of college graduates versus
>> non college graduates is likely to overstate the returns to education.
> Whats your hypothesis?
The hypothesis, which many people, including policy-makers, take for granted is that returns to a college degree are positive.
> Your mistake is trying to use statistics to make a specious conclusion.
Economists use econometrics to try to estimate such effects when the assumptions of classical statistics do not hold.
> Economics is a social science, governed by the laws of supply and demand,
> and human nature, which is complicated, but knowable.
Economics is the study of choice under constraints. Econometrics, which is what I was talking about, is concerned with certain types of empirical questions and developing techniques to deal with the types of observational data that is found in that realm to answer such empirical questions.
>> [economics] much harder than standard medical experiments
You are misquoting me … I said:
>> You underestimate the complexities of the human body.
You are misrepresenting what I wrote.
I am not going to pursue this further as it may be too far off topic for Steve’s taste. However, the lessons learned during the development of econometrics as a separate branch of study over the last 80 or so years do apply to the current practice of climatological research.
>> climatology is like econometrics
> Its being approached that way, which is a mistake.
I made a statement about the state of practice in climatology as it exists. The fact is, many people thought for a long time that an experimental approach to economics was not feasible. Many still do. Yet, those of us who prefer the experimental method are making headway.
In the current state of climatology, I am afraid the “mainstream” has not yet come to terms with inherent difficulties that exist when one tries to make inferences based on historical observations.
As I state every chance I get: The temperature observations we have now are the temperatures measured in the places we have chosen to measure the temperature.
> climatology is more related to thermodynamics, atmospheric physics, electromagnetics,
> chemistry, etc.
You make a statement about what climatology ought to be. I am not interested in that. I am interested in what climatology is.
> All hard sciences, unlike the social science, economics.
You know, after a while, this hard versus soft science business gets tedious.
> As such, a statistical approach is unprecedented.
http://en.wikipedia.org/wiki/Statistical_mechanics
Sinan
Sinan, my comparisons between the task of archiving econometrics code and paleoclimatology code is based on the position that the data sets are both similar in size and the scripts in paleo would by and large be no longer than in econometrics.
RE 182.
Sinar. An anecdote.
finding the warm places to live.
http://www.answers.com/topic/banana-belt
SteveSadlov should be all over this
I should mention that of course we don’t know if an audit would be fruitful or not, or a waste of time or not. Those that think it’s worthless could be correct, it could be a fruitless waste of time. (Although given Steve’s track record so far, there’s no reason to think it would be.) But if it can’t even be done, we won’t know.
Even if they think it’s not needed, meaningless, worthless, pointless waste of time (as Gavin has said it is) doesn’t explain why they’re so adament about keeping it. I don’t ever remember them saying it’s a mess so they didn’t want anyone to see it.
But the question is if they don’t have the time to make it run out of the box (which hardly seems to have any bearing upon it) how do they have the time to go over and over on the subject for 400 posts? I also rather wonder why an organization for space studies, run by what seems to be AGW proponents above all else, is responsible for their part in this. I guess it’s just budget and organizational issues that made it turn out that way. Odd.
Anyway, I’m thinking of other possible reasons they haven’t stated. Some are rather silly. a) They don’t have any code. b) They know something’s wrong with it. c) They think something might be wrong with it and don’t want the attention if so. d) They are refusing to give it up to create publicity. e) They like messing with Steve. f) They are secretly ashamed of the way it looks. g) They care for Steve and don’t want to see him waste his time. h) They think they’re too high and mighty to stoop to the level of doing what a mere mortal would like them to. i) They think they’re above all such investigations and checks. j) They are insane. k) This is all just politics to them and it’s some combination for various nefarious goals.
People wonder about these things when trying to figure out why something so simple is such a big issue…
Re Bernie’s inquiry:
If someone’s holding a gun to my head wanting an immediate answer, then yes, of course.
But, when I asked for a summary of what is being audited, to try and connect some dots…I get this response:
I didn’t say it was hidden. I don’t care what ‘Gavin says’…i’m asking you. Going through every thread and associated comments is not a summary. If a summary like the one i’m envisioning doesn’t exist, fine. I’m sure you don’t want to spend your time creating that. I’m arguing that if you do, this will help CA in the long run. There are many like me who would like to have specific knowledge of what you are doing without combing through every detail. Audits of various types will boil the salient details down to its essence.
Banana Belts ….. in California, Climate Zone 16 from Sunset’s Western Gardening Book is a nice example. We literally grow bananas, papayas, any and all citrus, etc and yet reside in between 35 and 40N. Of note, this past winter reminded us we really are not in the tropics, lots of death and destruction affecting aformentioned flora.
Sam,
I vote for a)1)
a) They dont have any code.
1) that will work without a little manual help.
I answered you Brian.
#181, I dont know of any such thing either. I just usually read the articles here and post, and glance at stuff on Pielke Sr, RC and Watts.
If anyone knows such a thing, they’d give it I assume. If not, not everyone is going to say no over and over. In his way, steven is saying there isn’t one, just read the topics, such as the one on the microsite issues looked at in 1952. Or the recent comments on the right. Or the links on the left to things such as favority posts, FAQ, categories or articles.
I’m sorry I can’t help you any more with that. RTFR is a frequent answer on many blogs on this subject, if you’ve spent any time on Deltoid, Real Climate, Rabbet Run and the like. It means “I don’t know” or “it doesn’t exist” or “the information is there and you have to read a great deal of it to understand, summaries don’t help” or “I can’t help you” or “the question you’re asking won’t give you the answer you’re looking for” etc.
Ok, everyone bare with me a moment.
I got started reading Hansen papers in depth the other day after reading a blurb at TNR by Brad Plumer. Plumer’s contention was that Hansen had disclosed his algorithms (adjustments) and that they were included somewhere in a listing of publications he linked to in para 4. The link is a GISS site of GISS Surface Temp Analysis references. Hansen’s are accessible, but of course contain no relevant math other that his SD averaging in ’98. My assumption then was that Hansen is riding Peterson’s methodology somehow.
Question: can anyone save me a trip to the library with links to Peterson, et al 1998, Homogeneity adjustments…, Int J Clim; Peterson et al, 1998, First difference method…, J Geophy Res: and/or Peterson et al, 1997, The quality control of monthly climate data, Int J Clim? (Just trying to follow up on S Blooms repetative suggestion to read up!)
Plus- any critiques or criticisms of same??
Second point- Steve M’s great tongue in cheek reminder that Hansen had been all for availability of data (per the reference to the 1999 statement by the Panel on Climate Observing Systems Status back on June 5th) as in the following:
‘The free, open, and timely exchange of data should be a fundamental U.S. governmental policy and, to the fullest extent possible, should be enforced throughout every federal agency that holds climate-relevant data. Freedom of access, low cost mechanisms that facilitate use (directories, catalogs, browse capabilities, availability of metadata on station histories, algorithm accessibility and documentation, etc.), and quality control should be an integral part of data management.’ Obviously good advise for the goose, but not the Hansen gander!!
Third point- Having four kids soon to be in the taxable realm of the US Gov, I have a multiplied interest in this subject, ie, any taxation taken from my children won’t be available to take care of poor old dad…that’s a joke son.
To have these decisions made on the basis of what A Watt and the volunteers are finding out about what is ‘probably’ the best surface network in the world is just the start of the troubles I have with the team. They have proven their intellectual as well as physical laziness from the fact that they haven’t even sampled a small percent of sites physically. The ROW, forgetaboutit!!!! I guess its just toooo hard to get their big butts out of those AGW causing AIR CONDITIONED offices and labs. Much less physically demanding to sit in the AC and do adjustments, models, etc… Just my first reading of Peterson et al’s ‘Global Hist Clim Network…’ (1998) with the number of assumed, adjusted, use of ‘personal communications’- Where’s the PEER REVIEW IN THAT!!- sorry, now practicing anger management.
Suffice to say, I may read not another AGW, globull warming item again unless the first few words start “Our best guess at this time is…..”
But I’d still like the help on the above articles…
CWells, Court Jester #51, Dallas Local 245
Brian:
Thank you for your response, but that was not a gun I was holding to your head!
Yours is a fair question, though I am not sure anyone is auditing articles per se as much as datasets and methods that appear to be central to the debate. I am relatively new to this site and others are far more technically adept both science wise and statistically. I would say that in the last few months core topics have included: the statistical methods of Mann et al.; the flurry of papers around linking storm frequency with AGW; methodologies for measuring sea temperatures; the integrity of China station data; access to reviewers comments to the IPCC report; most recently the adherence to NWS standards, and the perennial issue of how Jones et al and others have adjusted the raw temperature data. The later gets addressed in numerous ways – but it is core to much that is written. I think there are a few topic areas that Steve sees as beyond the focus for this forum, especially those dealing with the physics associated with CO2’s role in AGW – though many raise the issue.
Now I have written this – I feel that Steve or some of the old and more knowledgeable hands could give you a more precise and accurate synopsis.
just for brian, this site started out investigating MBH’98. See http://www.climateaudit.org/index.php?p=63 for an old overview. That led to a correction in Nature, and when Mann still refused to release sufficient code, a congressional inquiry (Barton). The NAS then commissioned a report, as did Barton, and both made some fairly damning commentary about MBH98. The NAS report had a sumary conclusion that reconstructions before 400 years ago (i.e. before thermometers) were unreliable.
Some other papers persistently fail to disclose data, and so are in-auditable. There are a variety of issues. See, e.g.
http://www.climateaudit.org/?p=25
http://www.climateaudit.org/?p=667
http://www.climateaudit.org/?p=693
http://www.climateaudit.org/?p=1552
http://www.climateaudit.org/?p=1741
http://www.climateaudit.org/?p=1396#more-1396
i haven’t tried to be comprehensive or up-to-date. Nonetheless, there are a few issues raised here.
per
RE 191. Cwells.
The notion that Hansens shows his algorithms is a diversion.
A simple example. Hansen 2001. You can find one almost every paragraph
“As an alternative approach to identifying stations subject to human influence,
we test in this paper the use of satellite observations of nighttime light emissions.
Specifically, we use observations from a United States
Defense Meteorological Satellite taken with a highly sensitive photomultiplier
tube [Imhoff et al., 1997]. Observations employed are generally those taken under
a new moon to minimize reflected moonlight.”
Generally taken under a new moon. How many? what percentage? did you analyze it?
does it make a difference ?
FURTHER:
” A composite of many images is used to eliminate ephemeral light sources such as
lightning and fires. ”
source data? exposure times? evidence that ephemeral source can be disambiguated.
FURTHER:
“The observations were acquired in 1995, so they do a good job of describing
current urban development. The same data have been used to quantify the effect
of urban development on primary productivity [Imhoff et al., 2000]. The spatial resolution of the
data used here is about 2.7 km.”
They do a “good job” quantify, explain, document!! 1995 to 2006 is 10
years of urban growth…..Spatial resolution is 2.7km?
I assume the defense asset used a CCD. and the pixel is square at 2.7KM per side.
I want to see the MTF of the lens system.
I want to see the backend chip and how it processed the photon bucket. I want to see
how nightlights pixels match up with actual sites. ( there are projection matrix
issues here)
The text doesnt address any of this.
So, Cwells. The answer is this.
If hansen were building an airplane his documentation would not get him to milling
one piece titanium.
end rant
re #192 and 194
thanks
Re: 183
Steve: I agree with your position on the archiving requirements. As I mentioned, I strayed off-topic starting with my first post by trying to point out also the similarities in methodological problems encountered in today’s Climatology and the problems encountered in Econometrics. My posts were in response to 88 and related discussion. A lot of time passed between my first post and the second so the connection was probably not obvious.
Sinan
#159 windandsea,
Cracked me up!
#161 Curtis,
By starting with the same data and the same code to see if you get the exact same results you have then verified that your data set and the calculator you are running is correct. The two together act like a signature. It provides verification that all is well with the data and that the calculator provided actually produces the results indicated. If sufficient computer power is not available I’d be glad to donate some CPU cycles (ala SETI) to help get results.
Then you go from there with your own calculations. Or auditing of the code or whatever else seems important.
The first thing to do to prove such a set up is to run code and data that gives known results. A base line. Without a base line you will get a lot of hand waving arguments.
Paul Graham,
Please, let’s not talk about the GCMs in the context of science.
Brian,
Although Steve uses this site to audit climate-related issues and papers, the specific selection is limited and at his discretion. Remember, this is all done on his dime with a little help from reader’s donations. Any kind of comprehensive audit of the entire corpus of current climate-related papers is way beyond his capability, and is in fact something that the government should be doing considering the policy ramifications of this work. A summary, however of his work to-date would be nice and something I would agree would be helpful to new readers such as yourself.
Steve McIntyre,
I know this will require more effort on your part, but could you find a way to leave deleted post numbers in the thread? Maybe a user name of deleted.
Once you delete a whole post it screws up the numbering and then it is very hard to figure out which post is which when some one refers to them strictly by number.
i.e. Larry’s currently #177 for instance which refers to #184 or Sam Urbinto’s currently #180 which refers to #181 (and other later #s.
If that is too much, perhaps eliminating numbers all together and forcing people to refer to posts by name and/or date-time would solve the problem.
There is no point in using numbers if you are going to change their meaning. LOL
Steve: I’m sorry, but I don’t know to do re-numbering. The things that usually cause me to hit the delete button are incursions of politics or attempts to argue against AGW from first principles in a statistical thread. Or namecalling. Ask the people who do these things not do this as it will just screw up the thread eventually.
numbers, lol m simon heh Oh, and yes, you have to have a baseline. Yep.
Question: anyone know where I can find an online calculator that would tell me something like for the 2 degree by 2 degree sea square at 0/0 to -2/+2, what the area is size-wise? (or land 5×5 from 50/50 to 45/55 etc)
I found something at http://jan.ucc.nau.edu/~cvm/latlongdist.html that I could get the distance for that and then figure out the square itself, but it would be easier if I could just grab both the area and where it is at once.
(I want of course to find out the reported anomaly for that area for a given month/year and for those squares of land and sea the percentages, and try and figure out where the data came from for that area and what it said.) Or is there someplace that has software to do most of that online?
This is very complicated.
If nobody knows, there’s forumlas in a bunch of places, where I can put together something that will semi-automate most of it, but it would be rather tedious and take me a while. You know, something to put in Excel to calculate it out for me. But that still leaves the issue of finding that square and looking only at it.
Thanks.
Steven M:
Chill babe!!!!
I already knew most of that, just not your direct quotes, but the general gist is so obvious once you get into it.
The S*&T Ain’t THERE!
If my company did our job like they do everyone within the sound of this blog would be dying of asbestosis and mesotheleoma….
I was just trying to do my part in disambiguating the subject.
I guess I’m off to the liberary….
CWells, Court Jester #51, Dallas Local 245
RE 171.
I have been a big fan of the Ag related climate data, but It hasn’t gotten
much traction. Within the audit paradigm it has marginal value.
RE 202
I see this same Theme over and over. Engineers who have to put up with somebody
watching their every move have no patience with prima donna attitudes expressed
at NASA. Drop your drawers turn your head and cough. Next.
RE #57: I would hope that my papers would be replicable (is that a word?), but I wrote them such that they would be understandable to someone in the same field. Experimental conditions, but not detailed protocols were included. I would expect that a scientist who knows how to run an ELISA would be able to replicate that part of the study, but that some experience and tweaking would be required. I would also expect that a scientist would be able to make his own spreadsheet to run the Fisher’s Exact Test like I did – but I did not submit my spreadsheet when I submitted my papers (yes, someone would have to do their own math).
My point is that there is an expectation that there is enough information in a published paper so that someone who was trained in that field can replicate the data. That does not mean that the information is provided at a level that a scientist trained in another field can understand.
And RE #139: Didn’t mention that I deal with drugs that would be approved under the Animal Rule, and that under this scheme the FDA does consider evidence of mechanism of action – all from the published literature.
So when is the Schwartz audit going to happen?
RE 175.
Brian, CA doesnt audit institutions, mostly its papers. A good example is
Parker’s UHI study.
1. The paper is referenced as a “cornerstone” of sorts for UHI issues.
2. Folks worked to get the paper posted free and clear .
3. Parker was contacted and he gave crucial data not in the published paper.
4. People argued for a couple weeks.
5. One reader forwarded a compiled list of questions to Parker, he responded.
That’s kinda how it works.
If you have something that bears scrunity serve it up!
Steve re: numbering,
Since I have no control over other humans (heck I can barely control myself) I think your suggestion is impractical. Esp. now that you are an international resource.
So may I suggest for the regulars: Please use a name or date-time when referring to posts and just expect numbers to point you in the general area. The numbers are not guaranteed to refer to the specific post you are commenting on.
BTW Steve, if the numbers can’t be counted on what is the point of having them? They become a hindrance rather than a help.
RE 207.
I think 207 is a brilliant post
I think you just demonstrated the renumbering problem.
I see now. The numbers are the permalinks. Click on one, and you get a URL that points to that comment.
I think it’s reasonable to just mouse over the name and time/date thusly:
steven mosher says:
August 22nd, 2007 at 7:12 pm
Deech56 ,
So you are saying that only members of the club should have access to the exact science? That seems rather narrow for placing trillion dollar bets with other people’s money. Business would never accept such attitudes.
Let me repost a short bit:
dearieme posted Aug 9, 2007 12:15:51 PM:
“Government scientists ..refuse to publicly release their temperature adjustment algorithms or software”: the default interpretation of that is that they are crooks.
from:
http://powerandcontrol.blogspot.com/2007/08/default-interpretation.html
Re: Deech56 August 22nd, 2007 at 6:16 pm
Releasing data and methods is a matter of public confidence.
Nothing to hide is important when spending other people’s money forcibly extracted from them by government rules, regulations, and taxation. Especially when the numbers get big.
Simon
RE 208.
That is why 207 is QED brilliant
Thanks for pointing out your station data page to me, Steve.
Four global historical temperature studies that receive major mention in the latest IPCC report are:
CRU (Brohan et al., 2006). Their data source: HadCRUT3 http://hadobs.metoffice.com/hadcrut3/HadCRUT3_accepted.pdf
NCDC (Smith and Reynolds, 2005). Their data source: GHCN + ICOADS http://www.ncdc.noaa.gov/oa/climate/research/Smith-Reynolds-dataset-2005.pdf
GISS (Hansen et al., 2001). Their data source: GHCN (including special attention to USHCN) http://www.agu.org/journals/jd/jd0120/2001JD000354/pdf/2001JD000354.pdf
Lugina et al. (2005). Their data source: Three reference books + assorted national data. http://cdiac.esd.ornl.gov/trends/temp/lugina/lugina.html http://ams.allenpress.com/archive/1520-0442/3/6/pdf/i1520-0442-3-6-662.pdf
I haven’t noticed Lugina et al mentioned on your website. On their website and the paper linked above, they describe their data sources:
Does anyone know anything more about Lugina et al’s three major reference books? I couldn’t find anything apart from the title and the fact that it is on magnetic tape. Have Lugina et al released any of their data set?
So, scientific work should not be audited by people from outside the specific field? See what two specialists in forecasting have to say about the value of forecasts provided by climate models, chapter 8 of the IPCC AR4 WG1 report. Quite a trashing! BTW the authors are professors at business schools.
Click to access WarmAudit31.pdf
I have been following this site for almost a year. I would guess that about 1/3 of the posters are engineers of one kind or another. Probably another quarter or so are scientists in one field or another. The one consistent thing that I have observed is that many of the strongly pro-CO2 AGW scientists hold engineers in contempt, while that is not true of the skeptical scientists.
Gavin has said the engineers know just enough physics and math to get into trouble. Now brian argues that engineers can’t tolerate uncertainty and can’t “replicate the science”.
Engineers replicate the science every day. By design, test and experiment engineers verify the laws of nature to a degree that most scientists would envy. Planes stay in the air because some engineer understood not just the laws of aerodynamics but the application of those laws in the face of material property constraints. The same is true of bridges, cell-phones, nuclear weapons and just about everything else that is used by mankind. To say that engineers don’t appreciate the uncertainty of science isn’t just stupid, it sets a new standard for idiots to strive for. Engineers have to make things work despite the uncertainty and they do it day in and day out.
And engineers don’t get second chances, ask the guy who designed the old Tacoma Narrows Bridge.
Apparently, demanding facts to back up statements is considered “gauche” by the NASA oracles. Too bad, in the world of engineers it’s put up or shut up.
Gavin has said the engineers know just enough physics and math to get into trouble, does he? Hey Gavin. Let’s see you go over to the less incompetent side of NASA and make something that flies. Then let’s light it off, and see what happens.
Bob Meyer says:
August 22nd, 2007 at 8:41 pm
Now brian argues that engineers cant tolerate uncertainty and cant replicate the science.
Actually we can stand a lot of uncertainty. We just want it bounded. We will adjust the safety factors and/or inspection frequency accordingly. We will also do a lot of testing.
BTW re: uncertainty. Is brian saying climate science is uncertain? Well that is big news around here. LOL
How about it brian? Is that what you meant?
Also let me add that I have probably replicated more science in a few years than most scientists have in a lifetime. 6 sigma defect free. When was the last time a climate scientist has offered results guaranteed to 6 sigma?
Some scientists don’t think that engineers really contribute anything new. Some engineers think they don’t ever need to consult with scientists. They’re both wrong. The best scientists and engineers know their limits and when they need to ask each other for help.
re:214
Those guys make a very good point, that there is a thing called Forecasting Science (ie the principles of forescasting as judged by empirical studies)in contrast to Forecasts by Scientists. They say IPCC is doing the second, not the first. One of the principles is that the more complex the model the worse the forecast. This might seem counterintuitive but it is easy to demonstrate even with relatively simple “models” such as extrapolation using polynomials. While you can sometimes get a reasonably convincing extrapolations with low degree polynomials, the higher the degree of polynomial the worse it gets (that is, in data containing error). The reason is easy to understand: the higher degree polynomials are fitting the error, not the trend. The climate is complex so mechanistic models have to be too, and hopefully they are better than polynomials, but there are fundamental reasons for skepticism even without any detailed knowledge of the models.
Come on “brian”, let’s have something close to consistencey. You start off by basically saying anything but “science” is practiced by mouth breathing Neanaderthals that cannot possibly demand the “science” conform to the “scientific method”. After that fails you dig up Heidegger from the grave to remind all of us mouth breathers about “uncertainty” which in almost all real world endeavors outside of quantum mechanics is not really acceptable or tolerated with a shrug and a “By your leave”. Finally you roll out Google and what it has as “hits” for Climate Audit.
I can see you are a carefully trained scientist by all your analysis. I am just not sure what you think you have proven here, other than you are wrong. Period. Sine Die.
After becoming aware of Nasa’s problems with temperature trends I feel more and more uncertain of what is the current state of the climate. Somehow I think that the old historical temperature data are easier to interprete in comparison with more recent data. The instruments were more simple and used for a much longer time. Most of “hidden” biases are related to variable degree of urban warming.
At least in Scandinavia it looks like there has been a long term warming trend since the 18th century (increasing winter minima) with a peek in the 1930’s (warm summers).
To avoid biases (which are difficult to quantify) caused by relocations of instruments etc. could it be possible to remove such years or months from local data when it seems that the values jump compared with nearby data? We then use only annual or monthly changes for the remaining time units. We will not get information of the local trend but instead we can calculate global or hemispheric mean trend (although the number of localities varies for each time unit). I am thinking of not adjusting for changes in the local conditions, only to use all “neutral” raw readings for the mean change between the time units. This trend will of course include any large scale drift caused by urbanisation.
Slightly OT (sorry). Can anyone furnish me with a list of scientists (reliable) who disagree with the current “consensus” on global warming?
Whenever I quote Lindzden and Singer in discussions I’m told they are crackpot skeptics and that there are 1,000’s of scientists who agree with the consensus and they can’t all be wrong. I think that if they are all using the same data and methods, they would all agree but that doesn’t mean the data and methods are correct. Even so, the coffee-room “experts” on Global Warming (of which there are many at my company) seem to have complete faith in this consensus. So am I right to be skeptical, or am I just being ignorant? Is there any light in these discussions or is it all heat?
I’m a confused pleb (with a degree in Artificial Intelligence – not Climate Science).
Thanks.
#190 CWells writes:
Not until I know you a whole lot better.
Re 222, jameson:
There are a number of reliable and even top scientists who disagree with the so-called “consensus”, see:
http://epw.senate.gov/public/index.cfm?FuseAction=Minority.Blogs&ContentRecord_id=927b9303-802a-23ad-494b-dccb00b51a12
http://www.canada.com/nationalpost/financialpost/story.html?id=3711460e-bd5a-475d-a6be-4db87559d605
http://friendsofscience.org/
However this may not make you any more popular around the coffee machine because as soon as a scientist challenges the “consensus”, he is accused of being on the payroll of big oil, see:
http://www.msnbc.msn.com/id/20122975/site/newsweek/
http://www.commondreams.org/archive/2007/08/07/3038/
#218
DeWitt
And when they both need to consult with a statistician. In Pharma R&D, statisticians are essential to the the design of both experiments and increasingly discovery procedures.
#224
Thanks Charly, very useful.
Jameson 222:
Check Here: http://www.urban-renaissance.org/urbanren/index.cfm?DSP=content&ContentID=16719
Here: http://climatesci.colorado.edu/
Here: http://sciencepolicy.colorado.edu/prometheus/
Prometheus’s host, Roger Peilke, Jr, is not a a scientist, but he links to a lot of contrarian scientists.
Hope this helps.
Re #223, MarkW
Thank you. I needed more coke spluttered over my keyboard.
re 230.
The technical term for this is NOSE ENEMA.
MarkW is a SteveM in disguise. All his joke are belong to me.
( 50 quatloos to anyone who gets the allusion.. and a free board version of Triskelion)
#222
[off topic – at this site, we’re examining detailed issues regardless of “consensus” so please discuss the sociology of consensus elsewhere. I’ve left a few posts up because I don’t have time right now to go back]
#215,
Bob, very well said! I finally have an explanation as to why the scientific method seems to mean more to me, as an engineer. A generator that I (BSEE) helped to design is providing electric power to the C17 aircraft. The power system protection module that my wife (BSEE) designed is making sure that power transfers happen safely on Boeing 777. The systems that both my father (PhD EE) and I designed are carrying freight all around this country. The steel and aluminum mills that my father automated are orders of magnitude more productive than before. We couldn’t afford to get the science wrong. We couldn’t imagine saying “well, I read a peer-reviewed article that says this should work and they use a lot statistics, so it must be right”
I bet engineers have more math and statistics, probably physics too, than do most climate scientists, particularly if the engineers have advanced degrees. Gavin should take a look at the degree requirements for a BSEE before opening his mouth.
Mark
232, I’d also dare say that a BSChE or MSChE probably has a much better grounding in heat transfer, mass transfer, thermo, and fluid mechanics than these guys.
RE: #215 – great post. Got me thinking. In my decades of experience, I’ve come to realize the following. Different communities have different working styles and different organizational cultures. The academic community has a spectrum of cultures that partially overlaps the commercial community, same goes for government. Within the academic community, there are various cultures or subcultures. In the hardest of hard sciences, such as chem, medical, certain bio, certain physics, certain geology, etc, the culture is anal and data driven, not completely unlike the culture of the better engineering groups in the commercial world. I am a scientist, who worked as an engineer, then as a corporate manager. My particular background is in physics, geology and general electronics and electromagnetism. I scratch my head about Mann et al and about Hansen. Does not compute …. LOL! On the other hand, back when I was in school, I fancied myself as somewhat of a Monkey Wrencher (never actually did anything, but thought about it). I hung out with hard core eco radicals. What were they studying? Environmental sciences, ecology, poly sci, sociology, english. Very non data driven (except for real, truly scientific, non agenda driven ecologists, who are really just math geeks, and rarity …. LOL!). There are scientists, especially in these fields, who are not into the hard scientific approach. Most of their “lab work” has consisted of either descriptive sciences or modelling. This is the essence of this great schism.
When I got my BSEE from Georgia Tech, we had to take 2 years of calculus and 1 year of differential equations, we also had to take a year of pure statistics.
Indeed, even ceramic engineers. They require we EEs take thermo, but not nearly to the level that any of the above do, with little emphasis on any of the remaining topics in your list.
Mark
Just one program … NYU Center for Atmosphere Ocean Science
I would add that some physicists, such as Mann and Schmidt, although properly trained, have fallen in the the “eco” crowd, just like I almost did myself.
You’re not even including your systems theory, control theory and likely, at least one course in signal processing theory (UCCS is now implementing “Signal Processing First,” a program that I think originated at Ga Tech, btw). All of which are math/statistics courses in disguise.
Mark
I think the point you’re missing is that engineers take an equally math-intensive, if not moreso, course program, and that Gavin’s comments are way out of line.
Mark
#235
An economics major concentrating in Econometrics would have roughly the same, with probaly one less year of calculus and 1 more year of statistics!!
The degree requirements for different majors is amazing. When I was doing some work for the NSF on the education and utilization of engineers, the big push from the Deans of the Engieering Schools was for engineers to go 5 years and get a Masters and a BA degree, while most liberal arts majors were essentially down to 3.5 years. Just check out which majors have year or semester abroad programs. More personally, my son went from being a B- Engineering student to an A- History major and his social life improved considerably! Dad was disappointed.
I also did not include a course or two in Fourier transforms.
There was also a very intense two part course in Boolean logic. The professor had written the book we were using (that tended to happen a lot at Tech), when we finished the book, he passed out a 50 page handout of what he had been working on since the book was written.
My first school (U of Mo-Rolla) actually considered making their program a 5-year program to begin with. The average was 4.8 years for graduation anyway – few could manage the 17 hours per semester load for 4 years straight and no summer school. The repercussions would have been tremendous since their “5-year program” would always be compared to other schools with “4-year programs,” i.e. it would have been a PR nightmare.
Mark
237, I’m impressed. Golly.
Can someone give me a link to Gavin’s comments about engineers?
That would be the systems theory classes. We had two – the book was the one by Ziemer, Tranter and Fannin – but those also touched on Laplacian analysis, which was expanded upon in control theory.
I’d say one of the biggest differences in engineering-style mathematical work is the focus on complex varieties of everything, btw. I even had to avoid the term “phasor” in my comprehensive presentation since the math majors on my committee wouldn’t have known its definition.
Mark
Took me 7 years, though I did Co-op all the way through.
Worked at the Engineering Experiment Station, on campus. I always forget what the new name is.
Interestingly enough, I worked with Dr. James Mahaffey.
While some schools seem to be adding climate science programs to the curriculum, you basically couldn’t find them 10-15 yrs ago. I think GISS only offered one undergrad course for Columbia students. The vast majority of existing “climate scientists” have their education in identical fields to many engineers and non-climate scientists. The only difference is the career path after the Bachelor’s or Master’s degree. And from my experience as a Ph.D. student, education for Ph.D. students is 99% research – which is often very specific, self-taught, and defined.
As an applied math undergrad, I was required to take 4 semesters of physics and 2 semesters of upper level undergrad/lower level grad courses in probability and stats. I took two additional electives in physics, plus some exposure in other classes. I’m not sure how many climate science programs have that level of depth in those fields. But oops – I ended up as an engineer.
#245
Spencer Weart said it in comment 1 of:
http://www.realclimate.org/index.php/archives/2007/04/learning-from-a-simple-model
Re #245,
Steve,
See the first comment in
http://www.realclimate.org/index.php/archives/2007/04/learning-from-a-simple-model/
It appears that another person’s comment about engineers may have been misquoted,
and misattributed to Gavin.
Steve:
In the cite, Spencer Weart says:
To which Gavin responds:
It’s rather passim.
DeWitt Payne said:
Some scientists dont think that engineers really contribute anything new. Some engineers think they dont ever need to consult with scientists. Theyre both wrong. The best scientists and engineers know their limits and when they need to ask each other for help.
I agree completely. My point is that I have only seen this problem with the CO2 AGW scientists, not the skeptics.
[Strangely, where I work my problem is that there aren’t any scientists to discuss problems with. The “geniuses” in management let them go because they couldn’t show value added for a particular quarter. We have one PhD physicist and we have him doing data analysis. What a waste of talent!]
Weart. That guy. He got tarred and feathered pretty badly by Lubos.
And where doe he get the idea that engineers don’t know anything about convection? That’s just plain weart.
Re 249:
Steve Milesworthy: I apologize, you are correct. I did mis-attribute that statement to Gavin when it was in fact made by Spencer Weart. Gavin merely said:
Your points are very well taken . Thanks. -gavin
In the context of Gavin’s comment he may or may not have endorsed the statement about engineers (since there were many points made in Weart’s post) but he certainly didn’t try to correct Weart on the issue.
Sorry, the statement was so Gavinesque that I got him confused with the actual author.
steven mosher August 23rd, 2007 at 9:02 am,
How far into the future does your crystal ball extend?
Steve McIntyre – perhaps you could prevent people who go off topic from visiting here. That would solve the problem. AS long as you know in advance who they are.
It looks like I owe a second apology. My post about engineers and scientists was solely for the purpose of contesting the idea that just because something isn’t discovered by scientists then it isn’t science. Engineers are not scientist wannabes who couldn’t cut the math. Facts uncovered by bag ladies are still facts.
So if I offered a distraction from the subject of replication then I am sorry and I hope that we return to the original subject.
There’s at least some relevance to discussing the differences between scientists and engineers. It seems that engineers are disparaged for not having the requisite background for proper scientific practices, of which replication is key, yet engineers are subjected to much stricter standards (than academic scientists) for such practices, replication in particular.
I’ve always viewed my engineering career as a practical application of science. Granted, I’m a bit more on the theoretical side of things now (though I haven’t always been), but I’m still regularly required to actually prove theory against reality. We’re in the process of wrapping up the primary program that I’ve been working and one of the requirements is that not only do I describe everything I’ve designed (theoretical and test analysis), but I also deliver algorithm code for both the simulations and “real-world” implementation. At least, if I want to get further funding I will have to do this.
Mark
Agreed, Mark. But in a larger sense, the sneering of people like Weart is symptomatic of a cultural problem in the climatology field which is exactly the point of Climateaudit. They don’t believe in multidisciplinarianism. They want to go off in their corner and do all of the statistics, all of the physics, all of the astrophysics, etc. and not have anyone outside of the tribe look at what they’re doing. This is exactly the crucial issue that makes this site necessary. So I don’t think this subject is ever off topic.
RE: 204
“So when is the Schwartz audit going to happen?”
I am a daily-obsessive lurker who, like many others, went through withdrawal during the recent outage. I can also be described as a climate neanderthal since I point and laugh at people who claim that fossil fuels are going to doom us all.
That said, I would be very interested in an audit of the Schwartz paper.
We are compelled to be grateful to people like Steve McIntyre who, on their own time and dime, shine a light on shortcomings within some contemporary research practices. The whole issue of the ability to replicate reported results is central to what Steve is doing, and what all of us (including Brian) are contributing to here.
Without generally accepted guidelines, principles, and applicable standards and methods, very little assurance can be attached to the current conclusions delivered by some branches of Climate Science. This is most especially true in the branches of Climate Science that deal with proxies for reconstructing past climate, that build models which simulate future climate, and that then derive conclusions (from those reconstructions and models) concerning the future of our climate and concerning the associated risk to planetary well-being.
The importance of Climate Science is clear to all of us. If the science is not done properly it can lead to incorrect conclusions and huge outlays of public resources for possibly misguided remedial actions. Lets be clear such outlays, if unnecessary, will kill as surely as needed outlays which are never made. There is only one answer to a question like Can there be any harm in playing safe and spending some money now to reduce greenhouse gasses? That answer is YES. It will kill.
Material resources are limited by their available quantities, qualities and the time and energy required for extracting and processing them. Human resources are limited. The cadre of Climate Scientists not doing some other science is expanding by the day.
The economic system can make similar mistakes look at all the real estate agents now looking for new work. But in the publicly-and publicity- directed Climate Science policy arena, there is no market feedback that takes expedient corrective action to prevent more good money being thrown in after the bad at least not until after a much longer period of time.
Opportunity costs can be huge. Spending money to bury carbon dioxide unnecessarily is as bad as building tanks or hospitals and then promptly sinking them in the ocean. Such unthinking approaches prevent any number of more utilitarian uses of those (wasted) resources, whereby lives could have been saved.
On the other hand, if Climate Science were to be done properly and if it was subsequently to validly conclude that we are in imminent danger then its warnings could save our civilization.
There is another insidious potential cost if Climate Science does its work sloppily how much respect does Science (and its practitioners scientists, engineers, technicians) in general stand to lose if Climate Science is wrong as currently promulgated by the Team? The Team is out on a limb in more ways than one. Their personal scientific practices are under attack for following non-professional standards. But they potentially bring disrepute to Science as a whole. That damage may be far worse. When a profession loses credibility it is not easily regained. Just ask a Congressman. Or a President. Or an ex-next President of the United States.
That kind of damage can subsequently lead far fewer people to place their trust in those with leadership roles in society such as scientists. If another life-threatening issue arose that required mass participation who could then blame society if it chose to put its head in the sand? The pharmaceutical industry is today in such a position. So is health care, in the opinion of some.
This is not a trivial question to the many contributors to, and readers of, this web log who are scientists or associated interested parties.
Steves efforts have been primarily directed at raising the level of research quality and thus of the reliability of the conclusions reached. His efforts must be commended.
Whatever the reasons for the Teams obfuscation and obstruction, whatever the perceived differences between scientists and engineers, between politicians and laypersons, between a scientist and a businessman, between those with and those without credentials, between those with malice and those with no agenda or preconceptions, the ship we are all on sails only to one destination, to one future.
What the Replication standard ultimately demands is that Climate Science finally accepts its responsibility (as should those other sciences which today do not) to allow transparent access into its data and methods. It cannot retain the opaque practices of the past. Our common destiny demands this.
The role played for all our sakes by people like Steve McIntyre, is the role of closing the feedback loop for information and its quality. We can only reach a level of wisdom if the quality of our information is good. With most endeavors which require knowledge and experience, a group will always know more than any one single individual, and many groups together will always know more than any one single Team. We all stand on the shoulders of giants (I forget who said that) and we all also stand on the shoulders of regular people (like our parents and teachers, although I dont consider them as regular).
We will only gain by opening up our thinking (as in blogs like this one) and our conclusions to scrutiny by others. No single one of us invented mathematics. We practice what we have learned largely from others. When we allow the feedback loop to close and connect, which examines our work and thinking, we open ourselves and our work to correction, to improvement, to personal growth. The excuse for not so doing can only come from ignorance and fear.
RE M. Simon (#210, 211): No, I am not saying that only members of the club get to see the data, what I am saying is that members of the club might be in a position to understand what is written in a typical Materials and Methods (or the equivalent) section.
“…other peoples money forcibly extracted from them by government rules, regulations, and taxation.” Geez, it’s not as if there’s some kind of political agenda here.
Deech56 August 23rd, 2007 at 7:11 pm,
Do you give your money voluntarily to the government? At tax time do you pay the government less than it wants or more or the exact amount? Why? If the government stopped asking you for money would you give it any way? I can’t think of any one I have ever known who volunteers to give their money to the government. If you are a volunteer tax payer it would be a first. Of course it may just be an American attitude. Probably explains why in America people (for the most part) want the AGW science and remediation programs as rock solid as F=ma or E=mc^2.
The materials and methods are not well defined. Steve has pointed out that that terms like reasonably long and consistently measured are used to describe station selection methods. What algorithm would you use to determine reasonably long? What algorithm would you use to determine consistently measured?
http://www.climateaudit.org/?p=1956
Are those terms of art? Are they good enough for replication?
When your Dr. prescribes medicine does he say you should take a reasonable amount for a reasonable length of time? Or does he say two pills four times a day for two weeks? Would you want your Dr. prescribing according to “climate science” methodology?
Hans: We all stand on the shoulders of giants…
Great post. However, that reminds me of James Annan’s:
“If I have seen further than others, it is by treading on the toes of giants.”
RE M. Simon #262: “When your Dr. prescribes medicine does he say you should take a reasonable amount for a reasonable length of time? Or does he say two pills four times a day for two weeks? Would you want your Dr. prescribing according to climate science methodology?” Ooh – not so good analogy. Even a 10-day bid course is an approximation at best. The dose of medicine is not adjusted for body mass and the pharmacokinetics of a drug in me is not the same it is in you. And despite large clinical trials, the safety of any given drug can vary from person to person. I suppose to apply the standards that are demanded for climate science the drug course in any person would be carefully calculated – maybe after a quick PK study.
Maybe it’s just my POV coming from the world of biology which has greater complexity and messy data due to so many biological and biochemical interactions and individual variation. Despite this, we have learned a lot about what happens in biological systems and we base decisions on what we do know.
And I pay my taxes cheerfully, without coercion. After all, the taxpayers paid for my graduate education, which was at a land-grant state university (getting an excellent return on their investment, BTW), and it’s the least I can do.
Deech56 August 24th, 2007 at 4:01 am,
Excellent! you made my point even better than I did.
Pay your taxes without coercion? I’ll believe that when you tell me you sent them 30% more than they asked for.
BTW Deech,
When you are testing a drug in an animal what is the protocol?
x mg/Kg? or a reasonable amount per Kg?
What would the FDA say if you told them that a reasonable amount per Kg had x effect? Approved for human use? Or suppose in an animal test you told the FDA that you injected a reasonable amount of drug y at some unspecified consistent interval and the results wer a not inconsiderable reduction of an induced tumor. You going to pass that to the FDA reviewer?
Those guys want specifications that can easily be replicated. They want numbaz. Lots and lots of numbaz.
I know that if I give a bolt tightening spec of a “reasonable torque” the FAA would laugh me out of the room. My design review peers would have drilled me a new one.
Or suppose I describe a breaker as tripping at an excessive current but holding at reasonable current? You gonna fly on an aircraft built to such specs? Or suppose I said the minimum air speed for take off was reasonably fast. The FAA going to buy that? Or suppose I say that the captain of the aircraft must abort a take off if he hasn’t reached a reasonable speed while there was still considerable runway left for braking? Want to fly on that one?
What makes “climate science” different?
It’s clear to me from the way these guys (Hansen, Gavin, Mann, etc.) are acting that they have invested a huge chunk of their ego into their work. In effect, their work has now become a part of them, and as such they can’t bear the thought of it being inspected. What if something was found to be wrong with the work? To them this would amount to a personal attack on their very identity, and so they can’t allow it.
I’ve seen this in the software industry where people protect “thier” code as if it were an arm or a leg. When these people are laid-off their first concern is about what will happen to “their” code; they want to take it with them and they can’t so it’s very traumatic.
Good scientists and engineers have learned to keep their work seperate from their egos. You can easily tell who these people are: they ask others to inspect their work. Their focus is on getting the work as correct as possible, and generally their work is of a higher quality as a result.
Re # 176: “Sam I think this is more than once this scientist versus Engineer thing has come up. The latter being schooled in having their cavities probed by the crooked fingered, ragged nailed inspector, and the former perhaps being more accustomed to the gentle persuasions of peer review.”
Steven Mosher, it is clear that you never have submitted an article to a serious scientific journal. Editors from the more high quality journals reject sometimes 50% of the articles they receive, or even more. My experience is that in general at least one of the reviewers is (highly) critical and it costs me a significant amount of time to satisfy the reviewer(s).In general the quality of my articles has improved significantly because of the comments of the reviewers. But is also clear that the best reviews are written by good scientist with a lot of knowledge about the topic, so therefore I don’t believe in the value of an audit but just some person.
Many scientist agree that the review process is not ideal. But they also know there is no good alternative. It doesn’t prevent bad science but is prevents many bad publications.
Simon August 24th, 2007 at 8:27 am
If I said that the torque on the bolts holding an anhydrous ammonia line together was “to be reasonably” tightened, and it failed, then I could be sued and jailed for criminal negligence. Not to mention that I would have to specify the gasket, and materials of construction, wetted and non-wetted surfaces.
gb August 25th, 2007 at 5:23 am If these journals were
how could they allow such handwaving; when the product of a scientific enquiry is supposed to be repeatable and verifiable.
Oh and by the way,
. One of the nicest things about an audit is that it is auditable. Audits have been thrown out when the auditors did not do a proper audit trail in their work. So, if you think an auditor may not have value in their audit, you need only audit the audit. If they can’t produce the materials for you to properly audit their audit, you should discount any claims made. You can also provide proof that the auditers were wrong and have them change the product or comments. I have done this several times.
gb August 25th, 2007 at 5:23 am ,
I think the open audit as practiced here is more valuable.
I admit it is messier and more time consuming. However, sometimes value has its price.
Relative to climate science we are now in a period of “buyers remorse” for failure to do our audits earlier. However, since the technology wasn’t available earlier I think the audit community can be forgiven. However, hanging on to old styles is a mark against it.
In addition earlier open review will help good thinking propagate faster. No need to wait for the official review.
Re #213, I have now more or less tracked down Lugina et al’s three main references for station data:
World Weather Records
http://www.ncdc.noaa.gov/oa/oldpubs/#WWR
Monthly Climatic Data for the World
http://www7.ncdc.noaa.gov/IPS/MCDWPubs?action=getpublication
Meteorological Data for Individual Years over the Northern Hemisphere Excluding the USSR
(an approximate title)
http://meteo.ru/english/data_b/sp.php?id_article=170
Unfortunately the data for some of these is not actually online. Of course they have only 384 stations in their pool, so it is not clear which ones were selected.
Deech 56 August 22nd, 2007 at 6:16 pm said:
I think this is a problem. There is an increasing expectation that journals do not provide the necessary information for replication. Controls, the experiments done; mere detail that a scientist in the field could replicate. I remember Nature showing the creation of knock-out mice with a single sentence; this is a procedure that takes ~2-3 years! Wholly inadequate.
However, it is when something goes wrong that a thorough description of methods pay off. You then have a hope of working out what went wrong, with resorting to quasi-litigation just to find the barest details.
It is also a lot easier if the experiments are fairly simple, with a fairly direct route from the data (which may be shown) through to analysis. However, once you start to get complex analysis (e.g. ratios, mathematical transformations) , the number of ways of perverting a data set becomes large. I am familiar with a whole host of abominations by way of data analysis, all under the murky cover of inadequately described data and methods.
per
2 Trackbacks
[…] http://www.climateaudit.org/?p=1952#more-1952 […]
[…] the NASA scientist (Hansen) who crunches the temperature data will not publish his software code. Making it impossible to verify the conclusions and results of temperature data. […]