It will take a while to go through the responses of Mann, Bradley and Hughes. I’ve taken a first look at the new source code and will commenting on it in due course. I have no information on responses from NSF and IPCC, other than what I read in the Nature interview with the Chairman of IPCC.
Update (Wed.) : Just to clarify, the new source code referred to above is archived at ftp://holocene.evsc.virginia.edu/pub/MANNETAL98/METHODS/multiproxy.f. Mann’s arguments about title to source code discussed below are presumably an attempt to justify his withholding this source code up to now, rather than for withholding this program from the House Committee (which he has not done). That is not to say that there are not issues pertaining to exactly what he has now disclosed, but that’s another story. On the issue of source code, Mann has taken a remarkably legalistic point of view that it is his personal property, notwithstanding its financing by NSF. I’ll discuss here an interesting legal issue about whether the source code belongs instead to the University of Virginia or University of Massachusetts. I’ll also discuss Mann’s surprisingly technical argument in the context of senior U.S. federal government policy on archiving. If Mann’s legalistic position is correct under present NSF policy, then I suspect that this will surprise many policy-makers and certainly suggest the need for a reiew of NSF policies and procedures, either in the form of more forceful contract negotiation and administration by NSF or a change in legal tools or contractual language available to NSF. Mann’s Statements Mann stated the following::
It also bears emphasis that my computer program is a private piece of intellectual property, as the National Science Foundation and its lawyers recognized….And whether I make my computer programs publicly available or not is decision that is mine alone to make….under long-standing Foundation policy, the computer codes referred to by The Wall Street Journal are considered the intellectual property of researchers and are not subject to disclosure.
As support, Mann quoted the following letter of Dec. 17, 2003 from David Verardo, Director, Paleoclimate Program, Division of Atmospheric Sciences, National Science Foundation to me regarding a request for residuals series, not source code (in my opinion, quite different issues pertain to research data such as residuals).
Dear Mr. McIntyre, I apologize if my last electronic message [see here ] was not clear but let me clarify the US NSF’s view in this current message. Dr. Mann and his other US colleagues are under no obligation to provide you with any additional data beyond the extensive data sets they have already made available. He is not required to provide you with computer programs, codes, etc. His research is published in the peer-reviewed literature which has passed muster with the editors of those journals and other scientists who have reviewed his manuscripts. You are free to your analysis of climate data and he is free to his. The passing of time and evolving new knowledge about Earth’s climate will eventually tell the full story of changing climate. I would expect that you would respect the views of the US NSF on the issue of data access and intellectual property for US investigators as articulated by me to you in my last message under the advisement of the US NSF’s Office of General Counsel. Respectfully, David J. Verardo Director, Paleoclimate Program
Mann obviously liked this line of rhetoric and, in answers to question 5, re-iterated the points:
I have made available all of the research data that I am required to under United States policy as set by the National Science Foundation. In accordance with the rules promulgated by the Foundation and supported by the Foundation’s General Counsel, I maintain the right to decline to release any computer codes, which are my intellectual property.
(Q5A).. our policies are fully in keeping with those established by the National Science Foundation.
(Q5B)…
A(Q5C): The source of these policies is the National Science Foundation.
A(Q5D): My computer program is a piece of private, intellectual property, as the National Science Foundation and its lawyers recognize. It is a bedrock principle of American law that the government may not take private property “without [a] public use,” and “without just compensation.”
To leave no doubt on his views, Mann went on to quote the following
Even more recently, the National Science Foundation confirmed its view that my computer codes are my private property. A recent issue of the Chronicle of Higher Education states: “According to David Stonner, of the Congressional affairs office at the National Science Foundation, Mr. McIntyre contacted the foundation last year to ask for Mr. Mann’s computer code. Mr. Stonner said the agency had told Mr. McIntyre that the code was the intellectual property of Mr. Mann . . ..” Richard Monastersky, Congressman Demands Complete Records on Climate Research by 3 Scientists Who Support of Global Warming, Chronicle of Higher Education (July 1, 2005), available at: http://chronicle.com/temp/email.php?id=dopjw74bwvqzvd3k9tekp5avlofvb2yu
Stonner is presumably referring to a letter of July 28, 2004 from NSF to me, replying to my letter of June 22, 2004, which was a broader inquiry into data not archived by Thompson, Jacoby, and others ee here ).
On the question of computer source codes, investigators retain principle legal rights to intellectual property developed under NSF award. This policy provide for the development and dissemination of inventions, software and publications that can enhance their usefulness, accessibility and upkeep. Dissemination of such products is at the discretion of the investigator.
I agree that the NSF said that Mann did not have to release his source code. However, this leaves two issues:
1) did the NSF adequately consider all the legal matters involved? I’ll suggest below that they did not.
2) If he NSF has articulated its policies, a) are these policies themselves consistent with more senior guidelines to NSF? b) should these policies be themselves reviewed? If so, who better to consider the matter than a House Committee? c) if the policies have ramifications outside narrow science matters, should the policies be considered in a broader policy conext?
University Policies
Players that are not considered in the NSF legal opinion are the universities — Virginia and Massachusetts. Mann used this source code both at the University of Virginia and the University of Massachusetts. Most universities, including Virginia and Massachusetts, have policies governing the terms of employment, which clearly reserve title to source code to the university. For example, the University of Virginia policies state:
Each investigator should accurately record all research procedures undertaken, observations made and all results, regardless of whether its value or import is apparent. These records should be maintained for at least five years and all data and notebooks resulting from sponsored research are the property of the University of Virginia.
This policy is set out in additional detail as follows:
The retention of accurately recorded and retrievable results is of the utmost importance in the conduct of research, and it is the responsibility of each investigator to maintain such records in a secure location. Data and notebooks resulting from sponsored research are the property of the University of Virginia. It is the responsibility of the principal investigator to retain all raw data in laboratory notebooks (or other appropriate format) for at least five years after completion of the research project (i.e., publication of a paper describing the work, or termination of the supporting research grant, whichever comes first) unless required to be retained longer by contract, law, regulation, or by some reasonable continuing need to refer to them.
The computer source code used by Mann is obviously an included “record” of “research procedure” and, according to the above term of employment, such code would be the property of the University of Virginia, rather than Mann’s personal property. Although the code has been used by Mann at the University of Virginia, it was originally developed at the University of Massashusetts. At this point, we do not know how much the present edition varies from the original code at the University of Massachusetts. The University of Massachusetts has a highly similar policy stating that “tangible research property including the scientific data and other records of research conducted under the auspices of the university” is the property of the university.
Both the University and the Principal Investigator (PI) have responsibilities and rights concerning access to, use of, and maintenance of original research data. (For the purposes of this policy the term PI refers to the researcher with primary responsibility for directing the project whether or not the research is conducted under a research grant or contract.) Except where precluded by the specific terms of sponsorship or other agreements, tangible research property, including the scientific data and other records of research conducted under the auspices of the University, belongs to the University. The PI is responsible for the maintenance and retention of research data in accord with the policy.
The University of Massachusetts also has the right to take possession of the source code.
5.1 Where necessary to assure needed and appropriate access, the University has the option to take custody of the data in a manner specified by the Vice Chancellor for Research
Accordingly, just because the NSF said that the code is Mann’s property is not the end of the story. The policies of the University of Virginia and the University of Massachusetts certainly indicate to me that the code is most likely the property of one of the two universities — most likely the University of Massachusetts. I must say that it is surprising to me that the NSF was seemingly unaware of standard university of terms of employment and failed to advise me that the code was possibly the property of the university and suggest that I inquire of the university, especially given the number of contracts that the NSF must carry out each year with various universities. I noticed these university policies earlier this year. In April 2005, I sent an email to Ariel Gomez, Vice President of the University of Virginia, (cc: John Casteen, President) requesting the source code under the above policies. . I have received no acknowledgement.
Senior U.S. Federal Policy
Even if NSF has accurately reported on its policies, there is surely a serious policy issue here. The senior policies on data archiving are discussed here and here . I commented briefly on these policies a few months ago here. For example, here is a senior policy statement from the US Global Change Research Program:
The government invests millions of dollars each year in research grants, contracts, and cooperative agreements. In many of these assistance and acquisition agreements the recipient develops data and data products that could be of use for other research and commercial or educational purposes. These data and products may come from various researcher activities including primary data collection, the synthesis or analysis of existing data products, and from calculations and model outputs. At the present, the agencies generally make no requirement on recipients that such data products be made openly available. With the very constrained agency budgets, it is becoming increasingly important that such data and products not be lost but, instead, be used to the greatest extent possible. It is in the interest of good science, as well as in the interest of the public, who fund our science activities, that data and information produced at the public expense, be made available for secondary use in the shortest time period possible. The requirement for making data and information documented and openly available must find its way into the language of our assistance agreements and contracts. The need for such an action has been recommended by the National Academy of Sciences’ National Research Council*
I am sure that these comments would represent the views of senior policy-makers to the extent that they have turned their mind to the matter. The tenor of this policy statement is obviously entirely against legalistic interpretations of title to source code. It also surely provides a clear direction to NSF to implement specific contractual language in their contracts which would prevent the precise situation invovled with MBH98 up to now. It also shows that policy-makers wanted to avoid the possibility of data being lost in moves between universities e.g. Crowley’s loss of the data in Crowley and Lowery [2000]). I am quite sure that policy-makers have long assumed that no paleoclimate scientist whose work was being used for climate policy would be taking narrow technical stands on title to source code. I am sure that they, like me, would have assumed that studies on topics of such importance would have the broadest possible disclosure and closest possible examination. It really doesn’t matter whether NSF has the tools and is not using them or whether NSF lacks the tools. In either case, the situation is surely unacceptable from a public point of view (and as a Canadian whose government is relying on these studies, I think that I have standing to comment on an American policy).
A similar situation applies for IPCC. I believe that it is useful to regard IPCC TAR as a scientific “prospectus”, since the standards of due diligence and disclosure involved in a prospectus should surely be exceeded in an IPCC assessment report (or so we are told). In that spirit, for IPCC to rely on a scientific study, it seems to me that IPCC should likewise ensure that it has adequate tools and procedures to ensure that there is adequate disclosure of data and methods in underlying studies relied upon by IPCC and that there is a policy interest in determining whether this is the case. I can certainly contemplate that circumstances could exist in which IPCC should insist on levels of disclosure exceeding ordinary journal standards or even NSF standards. IPCC impacts go well beyond narrow scientific matters and undoubtedly this is one of the reasons that a House committee with very broad interests has taken an interest in how disclosure is handled by IPCC, NSF and IPCC authors.
Climate Audit 
74 Comments
Is it a good idea for the U.S. Government to pay for research which lets an outside organization select what results get published?
“… These climate simulations are being run at the request of the Intergovernmental Panel on Climate Change (IPCC), which will publish selected results of related diagnostic sub-projects in its Fourth Assessment Report in 2006.”
http://www-pcmdi.llnl.gov/new_users.php
The Program for Climate Model Diagnosis and Intercomparison (PCMDI) at the Lawrence Livermore National Laboratory (LLNL).
Judging my your use of credentials I should hire a plumber to perform brain surgery. We are supposed to accept the anyalysis of climate change performed by an amateur statistisian and an economist over the analysis done by climatologists? How can you expect anyone who is educated and intelligent to agree with your pseudoscience? I guess this is what I should expect from Amerika in the Dark Ages, next we will be told(again) that the world is flat.
It is understandable when Microsoft wants to keep it’s source code secret. Billions of dollars are at stake so secrecy is of the utmost importance.
Why is Mann so concerned with keeping his source code secret? Billions of dollars are at stake.
Using your logic, Bob, we should neglect the mathematics developed by Pierre de Fermat on the grounds that he was a lawyer and government official with no mathematical duties…..
Bob,
Einstein was an amateur when he formulated the theory of special relativity. The “consensus” of the physics community initially was that Einstein was both an amateur and a lunatic who shouldn’t be taken seriously.
M&M are no Einstein’s but you don’t have to be a high genius to debunk Mann. Just honest and clever.
Bob: it is the climate scientists who are amateurs in your twisted logic. The issue at hand is statistics. Given a choice of an econometrician or a climatologist as to who is the expert in statistics, an “educated and intelligent” person will choose an econometrician.
Bob, you seem to be labouring under a couple of ponderous misconceptions:
1. What M & M have done is not an “anyalysis [sic] of climate change”, it is a failed attempt to replicate what you claim to be science. They couldn’t replicate Mann’s results, and they discuss why.
Nor are they isolated in this failure, as no one (including yourself, unless you’re hiding your light under a bushel) has been able to replicate Mann’s work. Replication is the keystone of science. Since the Mann et. al. results cannot be replicated, they are not science.
2. Science is utterly unconcerned with who makes claims. It is concerned solely with whether the claims are true. This truth is demonstrated by the results being replicated. Your concern with the background of the researchers is a (sadly) common logical error, known as the “ad hominem” argument. The “ad hominem” argument is usually used as you have used it here, in place of an argument about the truth of the claims. An “ad hominem” argument generally demonstrates that the person arguing does not have a valid argument on the subject matter, and so must resort to trying to blacken the name of the researchers …
Nice try, though.
A while ago I discussed intellectual property of things like computer code in another thread. It was an old thread, so my comment didn’t get any response, so here it is again for good measure:
I’ve kept away from the McIntyre/Mann controversy over supply of data and source code as I don’t know all the facts. But here is what I would do in similar circumstances over the Tuvalu work. If someone came to me asking for data and source code for the work I did at Tuvalu I’d be hesitant to provide it, for a number of reasons:
1. The original data is in the public domain and my results should be approximately reproducible from the information given in my report. I say “approximately” because even simple analyses like linear regression can vary slightly (e.g. do we weight the data points by their expected uncertainty? If so, how do we derive the uncertainty for each data point?). Also, I’m not interested in quibbling about these small details “¢’¬? if different people can obtain answers that are comfortably within the quoted uncertainty of a result, then I’m pretty happy.
2. I’ve moved on and am about to publish newer results for Tuvalu, based on more recently obtained data. Showing there was something wrong in my previous work would serve little useful purpose and could involve significant time on my part in providing data and answering questions.
3. We have to distinguish between raw data, processed data and source code. The raw data is in the public domain and so is not in question. Source code, however, is in a very different category, at least in Australia. The work was funded by the Australian Government and as such they “own” it. Many governments such as ours have developed a far stronger ownership of such things in recent years because they realise the value of the intellectual property that could possibly be sold afterwards, or possibly used to seed Australian private companies. I am talking very generally here “¢’¬? I’m sure my analysis software doesn’t fall into that category, but the fact is that a taxpayer or foreign national certainly does not have automatic access to software developed with government funding. It’s a rather similar case to that of our equipment “¢’¬? you wouldn’t expect to be able to walk into our Centre and use our computers or microscopes, even if you were an Australian taxpayer. Now, processed data is in an intermediate category “¢’¬? I think I would only provide that if I believed that I (and therefore my paymaster, the Australian Government) was going to benefit. If I sensed that providing such data would result in interminable future questioning and wasting of my time, I certainly wouldn’t provide it.
So “¢’¬? I guess the bottom line is, if someone asked me out the blue that they wanted processed data or software so that they could “audit” it, I’d probably say “no”. However if some spirit of collaboration was involved and we could all benefit, then it would be a very different matter.
Adding to Willis’ comments – this whole episode is looking more and more like pathological science – no one but the true believers can replicate the results.
As for the various arguments over source code, this is much like specifying the analytical techniques precisely when metal concentrations are derived from ore. Mann et al used a particular technique to produce a result. It is, to my mind, an identical issue.
If XYZ exploration company used MNO analytical technique to yield economic grades of gold in ore, and by due diligence a third party cannot replicate the results, then knowing how the result was produced becomes crucial.
In mining the reluctance to divulge such information is tantamount to being somewhat economical with the truth – BreX immediately springing to mind. It seems shonky practices are a human afflication, not specifically limited to mining entrepreneurs and second-hand car dealers.
Mr Hunter, as a matter of interest, what would you do if :
1) Someone demonstrated that they had reproduced your work as you had described it in your published work and based on the data in the public domain, and had got results that were significantly different from yours ?
2) A Parliamentary body asked for your processed data and source code ?
I think that John Hunter has provided perfectly good reasons why he should never, ever be trusted to produce a piece of scientific work upon which public policy should be based.
Like Mann, he clearly believes that climate scientists are an elite apart from any other sciences, and that the normal checks and balances which apply to all of the disciplines may not apply to studies whose conclusions guide the spending of vast amounts of money and the economic future of the world’s population. We should trust them as the saintly and incorruptible persons that they are, for they have the welfare of the world to heart as no others.
I applaud his honesty in admitting that scientific results that he has produced should only be made available for any sort of check by people he personally approves of, whether or not he took taxpayers’ money in doing so. Climate science, in Hunter’s estimation, should only be shared by people who share his beliefs about its conclusions, but the rest of us feeble-minded peasants should provide him all of the facilities that he should ever require for his papal pronouncements from that gilded ivory tower, and we should be grateful for the privilege.
It was John Daly, an extremely brave man, who directly challenged Hunter’s work on “The Isle of the Dead” sea level mark, but now John Daly is no more, Hunter can rest in the near-certain knowledge that no-one will ever check where he got sea-level data for Hobart for 14 years before the first tide gauge was established, or that the rest of his documentation when checked back to the sources did not support the claims made upon them. But then Daly was a nobody, and Hunter knows better than to ever condescend to show his working to an inferior.
Re; John Hunter
Let me get this right, about your point 2. Are you stipulating that you don’t care that there could be the most egregious, fundamental errors in your previous work, simply because you are going to be producing new work ?
It is almost impossible to comment on your analogy, since so much of the necessary detail is missing. Presumably, you publish the processed data ? Presumably, if competent scientists were not able to replicate your processed data, based on your publication, you would be concerned ?
I have to say that this reluctance to accept (tedious) audit is quite surprising. The case of the Ricaurte ecstasy paper in Science shows that the offending lab had to go back and audit everything that they did; and that is how they found (allegedly) that they had been supplied the wrong chemical, hence their erroneous results. Replication is the life blood of science. I fear that many researchers confuse the lack of interest in their work with an immunity from scrutiny.
yours
per
Of course, the Mann comment about intellectual property is hogwash. While there might just be something in the code that Mann is responsible for which demands that it be kept in the utmost secrecy, that’s really unlikely. After all, we don’t see claims from Mann that there is significant innovation, statistical or otherwise in his methods. Perhaps the R implementations indicate that a lot of the stuff is already covered by standard implementation techniques.
Anyway, in effect, all code is covered by copyright, and the copyright owners (UMass or U of Virginia or both, it would appear) can choose exactly the terms under which the code can be copied or used. They could choose to rescind their rights and place the code in the public domain. Another choice is to make the code available free for non-commercial use, and there’s a fine tradition of academic researchers doing that with much more substantial bodies of work. They could even forbid redistribution or use by anyone with the surname McIntyre, but it might be difficult to get a Vice Chancellor to agree to that.
The UMass policy seems very clear about the mechanics of ownership: the researcher assigns the rights to anything produced on university time to the university. If I remember correctly, in the UK, this assignation is written into the contract of employment.
Another oddness in Mann’s comments is that he makes the analogy from physical property, as if releasing a copy of the code deprives him of the original. Of course, it doesn’t. The phrases “without [a] public use,” and “without just compensation” have a lawyer-speak ring to them. But this is a pretty weak line: Mann has already been paid by the NSF to produce the work and the “public use” is stated in the Barton letters.
Cheers, — Jo
A PS to my comment above — here’s a “do as I say, not as I do”.
According to the “LICENCE TO PUBLISH” from Nature, authors are allowed (2c) To post a copy of the Contribution as accepted … for publication after peer review … on the Authors [sic] own web site or institutional repository six months after publication of the printed edition of the Journal, provided that they also give a hyperlink from the Contribution to the Journals web site. These terms seem to have come into effect around about 2003, and are retrospective. It’s surprising that Prof. Mann, as a staunch believer in intellectual property, should choose to publish MBH98 without the acknowledgement required by the licence.
Cheers, — Jo
Mann is also conflating the question of who has ownership of the copyright of source code with the question of its disclosure. Disclosure of source code does nothing, in and of itself, to alter the copyright status of such code. The only relevant intellectual property scheme that depends upon the nondisclosure for its protection is trade secret – and the utter incompatibility of trade secret and science should be obvious.
As for Bob above, what MBH is about is statistics, not “climate science” per se. Anyone competent in the relevant statistical methods is qualified to comment and criticize. Your “don’t tug on the robes of the priests” attitude is exactly what is wrong with AGW advocates today. And that kind of attitude is at its heart really more witch doctor superstition than science.
Reply to fFreddy (19 July):
1. I’d check my results.
2. It would depend on the authority of the “Parliamentary body”.
Reply to John (19 July):
Firstly, are you the same as “John A”? If so, it would be nice to know that I am talking to the same person.
I’ll ignore all the ad hominems that Steve McIntyre seems to have missed.
As regards John Daly and the Isle of the Dead work, the response to your claim that “Hunter can rest in the near-certain knowledge that no-one will ever check where he got sea-level data for Hobart for 14 years before the first tide gauge was established, or that the rest of his documentation when checked back to the sources did not support the claims made upon them” is at:
http://www.trump.net.au/~greenhou/reply.html
Reply to per (19 July):
You asked:
> Are you stipulating that you don’t care that there could be the
> most egregious, fundamental errors in your previous work, simply
> because you are going to be producing new work ?
No, that is not what I said. What I said was “showing there was something wrong in my previous work would serve little useful purpose and could involve significant time on my part in providing data and answering questions”. It is all a matter of the best use of my time — is my time best used in producing the best result I can now based on the latest data, or producing a better result than last time based on old data? It is clearly appropriate to point out obvious and important errors in previous work with a “correction” paper, but this is not appropriate for trivial errors.
You say:
> Presumably, you publish the processed data ?
The final part of it, but of course there is much intermediate processed data that it is not appropriate to publish.
> Presumably, if competent scientists were not able to replicate
> your processed data, based on your publication, you would be
> concerned ?
Yes, I’ve already said that, with the caveat “I’m not interested in quibbling about these small details — if different people can obtain answers that are comfortably within the quoted uncertainty of a result, then I’m pretty happy”.
Finally you say:
> I have to say that this reluctance to accept (tedious) audit is
> quite surprising.
Perhaps you would like to tell us what your job is, per, and then I can prepare a hypothetical audit of that and elicit your response.
Steve: I agree with Hunter about some of the ad hominems. I don’t moderate all the comments, so please be civil. John (Hunter), if there are some specific ad hom’s that you want taken down, send me an email and I’ll probably accommodate.
Substantively, in empirical econometrics studies, which are formally very similar to multiproxy paleoclimate studies, authors have to archive source code and data as used as a condition of submission (to American Economic Review for example). I don’t see anything onerous about this. Would that have been a problem for your original Tuvalu study? Also paleoclimate authors virtually never provide data citations to actual datasets. They will cite publications discussing the dataset, but sometimes use grey versions. Editions of data matter. The AGU has a decent policy on data citation but does not enforce it for paleoclimate authors; most journals e.g. Nature, Science, have no policy.
RE: John Hunter
The case you describe is a normal case. Small differences may occur in the results depending on the circumstances. We all might make mistakes, which have little or no effect on the basic results.
Mann-hockeystick-MM case is different. Very small differences in the selection of data (representing only a fraction of all data) may produce big swings in the results. Because we don’t know exactly how Mann has performed his calculations the source code should be published as such. From that it should be possible to see what data actually were used in the final published version. I think that this is the key question in the controversy. Because of the international interest in the results Mann should use some extra time to show in detail how he got the hockey stick.
Ye gods! I have to say that John’s reply of July 19th 4.55pm is one of the most unpleasant, personalised and spiteful posts I’ve read here.
John Hunter (who I, like everyone here, have never met) is, nonetheless a person who ‘should never, ever be trusted to produce a piece of scientific work upon which public policy should be based’ – a personnalised attack, ‘he clearly believes that climate scientists are an elite apart from any other sciences’ another personnal attack, ‘We should trust them as the saintly and incorruptible persons that they are, for they have the welfare of the world to heart as no others.’ likewise, ‘his papal pronouncements from that gilded ivory tower, and we should be grateful for the privilege’, I don’t know whether to laugh or cry to be honest, ‘But then Daly was a nobody, and Hunter knows better than to ever condescend to show his working to an inferior’. CODSWALLOP! – I ‘knew’ Daly, we frequently corresponded, we never shared more than a sharpish word and i respected him for that – would that you ‘John’ could ever reach his level. As to John Hunter being ‘condescening’, oh well, on more personnal attack in a post like yours probably doesn’t matter.
Now, ‘John’, perhaps you might consider talking about the science rather than practicing your so sadly sour invective in public?
Larry Huldén (20 July): I think there are a few too many “should”s in your posting — as usual, this site is high on pontification and low on practicalities. The science budget is finite. I, for one, do science to the best of my ability taking the degree of care that I consider to be appropriate. If you or others think that you have a better way of doing things, then you need to make a case that your way (e.g. increased “auditing”) will actually result in a better product. I have never seen that case presented. Rather, it seems as if the resources need for substantially increased auditing (e.g. the salaries and overheads of the auditors, increased numbers of scientists to make up for the time lost by existing scientists in providing material to the auditors) are going to materialise out of nowhere.
John, maybe there is however a case for data availability and a thorough replicability study where serious issues are raised about serious science. I think few would argue against the retention of data and methods (including computer programs) as being more or less a requirement for serious science. I say more or less as there is room for greater and lesser degrees of rigour depending on circumstances. However particularly for work that has a significant public policy impact (actual or potential), the requirement is surely at the greater rigour end.
I think the initial and very human response to the sort of challenge that the M&M papers offer is understandable. If this were merely of academic interest then this whole matter would be a storm in a rather small (if erudite) teacup. However the MBH98 &99 papers are not just of “academic interest”, despite the sometimes determined efforts of some at realclimate and other places to so assert. MBH is not just symbolic, if it is shown that the temperature reconstruction it represents has no statistical validity for example, then it has to call into similar question many of the other studies cited as showing similar results, because almost all of them use as at least part of their data the same relatively small set of proxy records.
And one extremely useful outcome of Steve’s work as presented on this blog has been to expose the very unclear foundations of those records. Three of the major dendrochronological datasets (and the three touted as showing the most obvious 20th century warming) have been shown to have major flaws, major enough to potentially invalidate them. The Bristlecone Pines, not a temperature proxy at all; Gaspe Cedars, cherry picked by admission and now lost so unverifiable as to the data quality, and the Polar Urals, significant uncertainties as to altitude variations plus a huge “CO2 adjustment” applied from 1750 that turns an apparent downturn from that point into an upturn.
So from a political point of view, there are many reasons to resist the demands for oversight and review. The Emperor may really have no clothes ! Politically, this is a problem best dealt with by political means, with all that entails. And the resistance to date, although humanly understandable, is all about politics, obfusticated with legalisms.
Scientifically however there is now no justification at all for refusing reasonable requests for replication. It is clearly in the public interest to see if the science can stand scrutiny as serious issues are involved.
Finally, why are so many here suspicious ? I say experience. We know this is politics more than science, and we know that politicians only fight tooth and nail to avoid these issues when they stand to lose, and when they think they will. If they think they have a winning hand, they bring it out, flourish around, and trumpet their success. If MBH98 has, for example, real statistical significance, I can’t see why that fact would not have been paraded for all to see. If that (and the related issues raised such as the inclusion of the BCP proxies) can be unequivocally demonstrated, then that’s it, they win hands down. So my experience says they won’t show cause they can’t. No copyright, no trade secrets, no threats, no intimidation. Plain and simple, they ain’t got the goods.
No-one has ever suggested that there should be a special class of “auditors” whose sole task is to check scientists’ work (what a dull job that would be!), nor continually drag scientists back from work in progress to deal with the minutiae of old work.
Both Steve and Ross have clearly advocated that scientific work, especially work that affects the formation of public policy, should be properly archived and available for interested researchers to check, and if necessary replicate, with minimal communication to the original authors. Since the work done should normally be the intellectual property of the academic institution, a simple request for access should be made to that institution.
Replication is part of the work of scientists using the scientific method. It is not an optional extra to be tagged on in order to be used as a political weapon.
I have personally asked many scientists for their research papers and questioned them about their methods and conclusions. Not one has ever felt the need to find out who I am or what motives I might have. I therefore conclude that Hunter, Mann and the rest of the Hockey Team are the exception and not the rule.
The suposition seems to be that publically funded scientists should be abbliged to revel any and all data pertaining to their work at any time to anyone requesting it. I do not believe that this would lead to more disclosure but less. Scientists would keep everything secret for fear that a competing group might get wind of a possible breakthough and demand that you reveal all your work. While colaboration is an important part of science, competition is also increasingly an important part, mostly because of the way that governments have chosen to fund research.
These comments about source code etc make the mining industry look like positive angels!
Ludvig von Mises wrote many years ago that universities had become captives of special interests and the state.
Seems he has been thoroughly vindicated.
The only reason Mann et al might be reluctant to divulge code is that it might have commerical value to a competitor.
As climate science is supposedly pure science, all I can say is that this reluctance to divulge code is pure horesehocks. Fear of the lawyers from hell preclude any more specific opinions.
Oh I can just imagine it.
Mining company A comes up with a fancy new computer program to model the location of gold ore from readily availble geoloical maps. Company A publishes this fact in its annual report, indicating that this will allow it to make loads more money. Company B asks company A for the computer program, Company A replies, of course! we will send someone straight away to show you exactly how we built the program, the theory behind it and we will even install it on your computers for you.
“I’ve moved on and am about to publish newer results for Tuvalu, based on more recently obtained data. Showing there was something wrong in my previous work would serve little useful purpose and could involve significant time on my part in providing data and answering questions.”
You can’t be serious? It would “serve little useful purpose” for someone to find methodological flaws in your previous work so that you wouldn’t carry them over into your “newer results?”
Yes, God forbid someone spend time correcting errors he/she made. It’s better to just ignore them. Records are meant to be broken, and errors are meant to be ignored.
Daly has “moved on” in more ways than one, yet you see no problem with “showing there was something wrong” in his work. Your website http://www.trump.net.au/~greenhou/home.html#pa still criticizes some mistakes Daly made which Daly went on to retract and/or correct (such as that discussed in “Watery North Pole) before he died. Clearly you seem to think there’s more than “little useful purpose.”
Steve (your comment, 19 July):
> John (Hunter), if there are some specific ad hom’s that you want
> taken down, send me an email and I’ll probably accommodate.
No, please leave them in — they reflect on the poster, not on me.
> Substantively, in empirical econometrics studies ….. authors
> have to archive source code and data as used as a condition of
> submission …… I don’t see anything onerous about this.
> Would that have been a problem for your original Tuvalu study?
I try to keep as good a record I can of everything I do. I keep scrupulous notes. I never edit raw data files, but write (and keep) scripts and programs that generate processed data from the raw data, so that “personal replication” is always possible. I back up my computer daily so nothing is lost. However, if I go back to data that I worked on several years ago, it still takes me some time to work out exactly what I did — because data processing is nowadays a complex procedure, especially in scientific (rather than engineering) fields, where much processing is exploratory and many lines of processing may subsequently be discarded. If I gave my “source code and data” to someone else and expected them to make good use of it without my help, I think I would be disappointed. I could take even more care with my archives but that would take more time and resources. So, compromise is necessary. Climate science may not currently be at the optimum of that compromise — I don’t know. But the current principles of (1) keeping original data, (2) describing, in a publication, the processing in enough detail for someone else to make a good approximation of repeating it and (3) keeping processed data that was used to produce the final publication (e.g. for published graphs) seems a reasonable way of working. I’m sure we could all be more scrupulous about archiving — but no one should pretend it is trivial or easy. And remember there is still the issue of intellectual property of things like source code, which is certainly a consideration in Australia.
Incidentally, I’m gradually gaining more of an appreciation (and I mean that in a positive sense) of your background. Mind you, it took Michael Mann’s letter to the House Committe to reveal your biography at Ross McKitrick’s web site (now why couldn’t you have saved me a lot of trouble and indicated this biography earlier?). An impressive biography it is too (I’m being serious). I was again impressed by the scrupulous details of Middle Assyrian history at http://www.geocities.com/farfarer2001/ — but then you blew it all by giving your address as that of CGX Energy Inc, at good old 120 Adelaide Street ….. 🙂
Cheers,
John Hunter
Steve: If it makes you any happier, this is merely a mailing address at present; I don’t want to use my home address. But what am I “blowing”? The people there are my friends and, if I had to go back to working at business, would return to something related to what I was doing before. As to Middle Assyria, I originally wrote to Mann asking for data on almost the same day as the Baghdad Museum appeared to be looted. I was simply dumbfounded by the looting. I wasn’t so much worried about the art treasures – even if they were stolen, they would turn up at some point. I was more more worried about unedited cuneiform from the start of human history being casually turned into rubble. I guess that the final outcome has not been nearly as bad as I initially feared, but I’m not up-to-date on it. But let’s not get into this.
You missed out (4) making processed data and methodology available for academic request. What would be the point of only having retained all of this stuff only for “personal replication”? Are you the final arbiter?
By the way, other than sarcasm (which Tasmanians speak fluently anyway) if you’d like to point out an actual “ad hominem”, then I’ll be sure not to repeat them. Something along the lines of comparing my opinion with the political acts of Senator Joe McCarthy, for example.
I have accused you of that which you have already admitted – that you don’t feel under any obligation to disclose methodology in order to aid replication if you don’t feel the requestor shares your beliefs about its conclusions.
An “ad hominem” would be a statement to the effect that “John Hunter may be ignored because he’s biased, because he’s a Tasmanian, because he doesn’t shower properly, because he supports this or that political cause or believes this or that religious dogma”. I have not made such a statement. However it’s difficult to swallow such an accusation from someone who has created a website which makes ad hominem attacks on someone who has since died.
I simply ask on what gives John Hunter (or Mann or anyone else) the right seek to assert a truth about the world around us which has public policy implications and yet withhold from public scutiny the key methodological steps which underpin the assertion. I and quite a few others would like to know the answer.
You repeatedly stated that there is no replication of Mann et. al.’s Construction of Temperture, but he claims that there are several (Amann et. al amongst them), who replicated them.
Also, Mann stated that there is no issue with the source code, because it is available on the webspace. (I have no fortan-compiler, so I wouldn’t know whether this is correct.)
So, he also has a lot of scientific organisations behind him, who are rather very conservative about their picks. And to any claim of criticism must be a proportional Popper Falsibility test, otherwise this “war” will go on forever. When will you consider that Mann has won and when will he consider that he is wrong?
I know that he just doesn’t accept this fact, because he is insistent about being right, but what if he really is, which point would get you to give in to him. What kind of evidence do you want/need?
Steve: Off the top of my head, if Mann can show that he has: 1) A statistically significant cross-validation R2 statistic for the 15th century proxy network; 2) a demonstration of robustness using exact methodology to not using bristlecone pines and not fudging the Gaspe tree ring series. I’m pretty safe on this since they haven’t contradicted these points, but change the topic or argue something different. The war shouldn’t have gone on anywhere near this long. These points are simple to test and verify. They are clearly set out in our papers. Wahl and Ammann avoid the issues.
“I simply ask on what gives John Hunter (or Mann or anyone else) the right seek to assert a truth about the world around us which has public policy implications and yet withhold from public scutiny the key methodological steps which underpin the assertion. I and quite a few others would like to know the answer.”
As climate change science has now become a political animal it is reasonable that it is using the tactics of politics. Since when have we the public been able to scrutinise the methods by which our political leaders reach there decisions regarding our lives?
Re: Paul Gosling’s comment re mining company.
What is at issue is not some computer code to discover an orebody, (which is of course highly secret) but the code that a discovered orebody was assigned a publicly published grade of 40 oz per tonne gold, and thus highly economic, (everything else being equal) which potential investors, when engaged in due diligence, were unable to replicate.
Reporting of mining assays are prescribed and any serious mining company does so dilgently and honestly. Shonkies of course take liberties with reporting facts, much as second-hand car dealers when they asset that the car of your dreams was driven by an elderly spinster who never exceeded the precincts speed limits – except that the drag slicks on its wheels cause you some genuflection on the salesman’s veracity.
I personally have no qualms in restricting code or analytical techniques that give a competitive edge in mineral exploration and here Paul Gosling errs, (and I can assure readers here that such code or analytic techniques are never allowed into the public domain). However the public reporting of mining assay results to stock exchanges to persuade investors falls into quite a different area to that misunderstood by Paul Gosling who conflates exploration concepts with published assays of ore deposits as being equivalent. They are certainly not. Secret mining business is kept very secret, while published assays are published according to agreed standards.
So when a company asserts publicly that deposit Z, found by whatever secretive technique, assayed x% Au using an unpublished analytical process, (where X% is defined arbitrarily as highly economic) and investors wish to verify this assay, but find they cannot, it is, like it or not, regarded as fraud until proven otherwise. The directors of company XYZ, discoverers of deposit Z now find themselves having published an unreplicable assay, and the objects of not only stockholder anager, but also of government. Seems this behaviour is inapplicable to science.
So Mann et al are in the same invidious position, a position all of their own making I might add. They asserted that they derived particular results from their computer code and published it. Others, using the same data set, and using the descriptions by Mann et al in peer reviewed literature as to how they, Mann et al, produced those results, were unable to replicate those results. Hence the request for the computer code is indentical to asking how the fictitious mining company managed to produce X% Au from ore that independent auditors were unable to replicate. It is a reasonable request.
Re #30
I am not taking the position that “because climate science has become politicized, therefore political scrutiny is required”.
I take the view that “because climate science has very large public policy and economic implications, those scientific results must be open to much greater scrutiny and public disclosure of methodology for the purposes of replication and audit”. What Steve refers to from his mining background as the availability of the “split core”.
Re #32
Split core is a safety fall back in mining to replicate assays. When split core is assayed and assays do not match assays from the other half of the core, (withing statistical limits, and here we wander into geostatistics, a complex discpline in its own right, based on a mass of empiricism called mining and economics) then the ore deposit has to be redrilled. So it was with Karpa Springs in West Australia, and so it was with Bre-X.
Re #32
Split core is a safety fall back in mining to replicate assays. When split core is assayed and assays do not match assays from the other half of the core, (within statistical limits, and here we wander into geostatistics, a complex discpline in its own right, based on a mass of empiricism called mining and economics) then the ore deposit has to be redrilled. So it was with Karpa Springs in West Australia, and so it was with Bre-X.
Re 22:
Paul, If mining company B wants to buy assets from company A, company A has to follow a due dilligence procedure, i.e. present all their cores andcompany B will drill new ones parallel to existing ones.
Company B will make their own reserve estimate, and company A will have big problems when the ore reserves are significantly lower (e.g. Bre-X).
Exploration geophysics is hard work, most exploration geophysicist change companies frequently and new exploration methods are quickly disseminated. Also mineral exploration is a small world just like climate science. 😀
Readers interested in the concept of science auditing might take a look at the Health Effects Institute web site: http://www.healtheffects.org/pubs-special.htm. Scroll down to “Reanalysis of the Harvard Six Cities Study and the American Cancer Society Study of Particulate Air Pollution and Mortality. A Special Report of the Institute’s Particle Epidemiology Reanalysis Project. Final version, July 2000.” This was a situation where 2 papers by a single team using large data bases and somewhat complicated statistical analyses concluded fine particulate matter was causing a significant increase in premature mortality. On the basis of these 2 studies the US EPA announced plans to tighten ambient air quality standards, a potentially costly rule change. The studies thereupon came under considerable scrutiny, but the data were not publicly available. A scientific advisory committee at the EPA asked for the data to be published, several Freedom of Information requests were filed, and (ta-dum!) the House Commerce Committee demanded the data. Finally the authors agreed to turn over all their data to a 3rd party, the HEI, which conducted an exhaustive audit of the data, a full replication study and a sensitivity analysis. (The HEI is jointly funded by the EPA and an industry consortium.) The audit found only minor and inconsequential errors in the data base, and the replication was pretty close to exact. The sensitivity analysis generally supported the results but did find education, multi-pollutant effects and smoking were influential covariates. So the point is that where a major policy was at stake, the “but-it’s-been-peer-reviewed” argument counted for naught. No one was saying that every single study on particulates and morbidity ought to be audited, but this was a pair of papers that the EPA singled out for prominence to justify its policy change. The authors cooperated with and were vindicated by the audit. If this kind of process can happen for air pollution policy why not for climate policy?
Hans
I am sure that most scientists will be happy to sell their intellectual property. That is not what is being debated here. The argument is that if work is publically funded it should be freely available to anyone. So I could spend three years working on a problem, get close to solving it, then someone else who has more resources at their disposal could come along, demand all my data, finish the work before I could, publish, take all the credit and the next set of funding.
Steve: Paul, doesn’t one milestone occur at publication? If you’re developing something proprietary, then don’t publish. If you want to publish, then you should provide the code and data, as you are now required to do in economics. Also if a study is being incorporated into a multiproxy study, then the multiproxy should ensure that he uses properly archived data. PArt of the problem here is the paleoclimate scientists do not adhere to AGU data citation policies, which effectively require this, but which are ignored. Steve
Dear John Hunter
you said “showing there was something wrong in my previous work would serve little useful purpose “, which to me is very clear as a matter of English. If you assume that there are no errors in your work, or that your work serves little useful purpose, that is another matter entirely and that is something you didn’t state. But you actually proposed that there was something wrong with your work, and then say that this is of little account ! Maybe there is some imprecise English here ?
“Perhaps you would like to tell us what your job is, per, and then I can prepare a hypothetical audit of that and elicit your response.”
Hmmm. In the pharmaceutical industry, research is audited, and performed to the standard of GLP. In UK academia, people keep lab notebooks and are obliged by the Research Councils to keep those for 7 years, plus all ancillary documentation/ computer files/ etc. There is also the significant issue that when I do publish, I do my best to ensure that all the necessary details are provided so that others can replicate my work. The nature of my work is such that most of my results are numerically very simple, and there is only a small amount which goes into publically accessible databases- which is a condition of publication. I have also provided several of my reagents to people who request them.
But this is a distraction. There are many examples of people who have either fraudulently, mistakenly or inadvertently, published data which cannot be repeated. It is this process of repetition which is fundamental to the scientific process, and which separates science from religion. Audit may be tedious- but it is part of the obligation of being a scientist. Or would you rather be obliged to believe in Cold Fusion ?
yours
per
Dear Paul Gosling
all of this is in the context of work that has been published, and where authors are undertaking to provide sufficient details to enable replication.
yours
per
Comment #26 Max states “Mann stated that there is no issue with the source code, because it is available on the webspace.”
Mann is ***ing. Mann will *** but he won’t *** . *****
Read what Mann stated in his letter to the House Subcommittee. He says the source code is his private property and he won’t release it.
Steve: I’ve snipped this a bit ***. Read Mann’s letter a little closer. Mann has reserved his rights to the code, but has actually releasd a new tranche of source code, which I’m in the process of examining. I thought that my preamble was clear on this, but I’ll specify it a little closer.
For anyone who thinks all the key findings of the Mann study is adequately replicated or backed up by other research:
***!!!Climate Science Community Fails To Back Key IPCC Finding!!!***
http://www.geocities.com/poncedeleon_1/ScientistsBackDown.htm
Fact based article shows scientists are reluctant to assert that
“It is likely that the rate and duration of the warming of the 20th century is larger than any other time during the last 1,000 years”
How’d that happen?
Could Steve McKintyre have something to do with it?
Mr McKintyre :
You mentioned you could not directly access Mann’s ftp site, but you have a copy of ‘multiproxy.f’.
Just to check, do you also have the files ‘Readme’ and ‘AlgorithmDescription.txt’ from the same location ?
And am I right in thinking this is the first time he has referred to using a scree test ?
Bother, just checked your post on Corrignedum, please ignore re: scree. Sorry.
Reply to John (20 July):
Firstly, please reply to my earlier question: “are you the same as ‘John A’? If so, it would be nice to know that I am talking to the same person”.
Secondly, you said I “missed out (4) making processed data and methodology available for academic request” — as I thought I’d made clear in my posting of 19 July: “the bottom line is, if someone asked me out the blue that they wanted processed data or software so that they could ‘audit’ it, I’d probably say ‘no’. However, if some spirit of collaboration was involved and we could all benefit, then it would be a very different matter.” — and I earlier gave my reasons.
Thirdly, you said “if you’d like to point out an actual ‘ad hominem’, then I’ll be sure not to repeat them”. Now, perhaps you have some internal communication problems, but Steve has already said “I agree with Hunter about some of the ad hominems. I don’t moderate all the comments, so please be civil.” That comment clearly related to your posting, so please ask Steve to point out the ad hominems if you are unable to identify them.
Reply to various other related postings: I’ve described the steps I take to ensure a reasonable degree of robustness and traceability in my work. I’ve described how I would treat requests for my data and source code, which I think is similar to the way many other Australian scientists in similar fields would act. If you disagree, well that’s fine. But I’m not interested in getting into a debate over this. The thing in which I am mainly concerned on this site is Steve McIntyre and his views (and I think I am coming to some understanding in this respect) — the rest is very much of marginal interest.
This is going to be a long commentary reply so please be patient.
Yes. But more importantly I am an ordinary person who dares to ask questions and even more incredibly, expects reasoned and reasonable replies from people who claim expertise.
Then you are either a) being deliberately obtuse or b) unable to read what I actually said. I’m going to have to use emphasis here since its failed on the multiple times I’ve posed the same questions over and over:
“I simply ask on what gives John Hunter (or Mann or anyone else) the right seek to assert a truth about the world around us which has public policy implications and yet withhold from public scrutiny the key methodological steps which underpin the assertion.”
To clarify, I do not believe that you have a veto on the disclosure of scientific work of whatever kind which has implications for public policy, taxation or the democratic process. Mann’s work clearly reaches that position. Your work on “The Isle of the Dead” and the Tuvalu relative sea-level clearly has public policy implications (otherwise why did Greenpeace publish your comments?)
In the first instance, because the work done in an academic institution belongs to that institution and not to the individual or team producing it.
Secondly, because science is always a collaborative enterprise which relies on researchers sharing work and results with others. Obviously, if a discovery is made, then researchers would want to wait until publication before disclosing anything, but after that, in my opinion, the checking of methodology, data (including source code) is a key process by which a potential discovery is verified and validated.
Such validation should, ethically, be done by entirely independent groups. For you to claim that “the bottom line is, if someone asked me out the blue that they wanted processed data or software so that they could “audit’ it, I’d probably say “no’. However, if some spirit of collaboration was involved and we could all benefit, then it would be a very different matter.” is a clear conflict of interest.
You cannot choose who checks your work – you don’t choose who the peer reviewers are in academic periodicals, for example. In any case, the imprimatur of the work you have done belongs to the academic body who employs you.
Bottom line, unless the work is under non-disclosure for commercial or security reasons, I do not believe you have the right to veto who gets to see it.
If you wish to complain about civility, please do so. I used some invective in my previous comment, but no ad hominems (which Peter Hearnden seems to think means “the expression of any opinions contrary to his personal beliefs”)
To repeat: “… it’s difficult to swallow such an accusation from someone who has created a website which makes ad hominem attacks on someone who has since died.”
I don’t doubt that your scientific methodology is anything less than scrupulous and thorough. I do appreciate that the issue of archiving and retention of proper records is an administrative issue for all researchers. I simply question your attitude towards the sharing of those records – that I think is NOT shared by fellow scientists, Australian or otherwise. I have not found your attitude to be shared by other researchers.
If Steve and Ross have been banging on about anything, it’s the proper public archiving of data and the disclosure of methodology, even to hostile audit, should be a prerequisite for the results of such work to be allowed to affect the body politic.
“John A – of marginal interest to some scientists, yet expected to pay for their salaries and facilities without question“
Re: #27 “As climate change science has now become a political animal it is reasonable that it is using the tactics of politics. Since when have we the public been able to scrutinise the methods by which our political leaders reach there decisions regarding our lives?”
I’ve read this over and over and am flabbergasted! The logic is bewildering – are climate scientists like politicians, ie. liars and thieves? Do we elect climate scientists; can we vote them out of office? Dear lord, is this the level of reasoning that we can expect from the pro-AGW side? If so, there is no getting throught to them.
btw: re#38 – I can find no ad-homs in #10 so rather than asking Steve to tell John what he thought John said, why don’t you just tell him? And why do you care if John is John A.?
Doug Z
My comment re 27 was meant half tounge in cheek. However, only half. Much of the discussion which is going on here, on other sites and in the popular press is no longer about the science, but about the policits of the science. The normal scientific process has been bypassed. Given that it is not surprising that the debate looks more like an argument between politicians than scientists.
The anti AGW side claim that there are a considerable number of scientists on their side. If this is so, and I have no real reason to doubt it, why don’t they use the normal scientific processes. Do their own reconstructions, you don’t need Manns code to do it, use their own code, use dofferent proxies whatever it takes. If they consistently fail to get a hockey stick then people like me will be more inclined to accept that Mann et al were wrong. This constant bickering about ownership of code etc. is becoming tedious and tells us absolutley nothing about climate, past or future.
John: I will address your pontifications once more on this issue and then that is it.
> To clarify, I do not believe that you have a veto on the disclosure
> of scientific work of whatever kind which has implications for public
> policy, taxation or the democratic process.
I described in an earlier posting what I would generally disclose and what I would generally not disclose. I said that, as a result of what I would disclose, “my results should be approximately reproducible from the information given in my report” (or paper if you want to generalise my statements). I explained what I meant by “approximately”. This level of disclosure is based on conditions imposed by my employer (e.g. related to the intellectual property of that institute, and my duty to work as efficiently as possible for it). If you seriously think that you can extract ANY information you like from a public body, related to “public policy, taxation or the democratic process”, then I fear you have never made a “freedom of information” application (and I believe you have only (rather belatedly) had FOI legislation in the U.K. since 2000).
> In the first instance, because the work done in an academic institution belongs
> to that institution and not to the individual or team producing it.
I’ve already made that point — the information I am talking about is owned by my institute. However, I cannot see what bearing this has on your argument.
Finally, you end:
> John A – of marginal interest to some scientists, yet expected to pay
> for their salaries and facilities without question
Unless I am mistaken, you are not an Australian and do not pay Australian taxes. However, if you are offering to make a contribution to my own work, or to my institute, you are very welcome to do so — you may then (and only then) be permitted to put some conditions on the way we act, while making use of that funding.
That’s right John, you stay in your Ivory Tower. But don’t expect me to pay for policies based on your conclusions without them being thoroughly checked – otherwise I’ll send you an invoice for the damage caused.
You do not have a right to limit access to research whose conclusions have public policy implications, intellectual property belonging to the academic instition makes no difference to this point. If you think you’ll get away with (as Mann tried to do) with “my results should be approximately reproducible from the information given in my report” then you are sadly mistaken, just as Mann, the rest of the Hockey Team and even the IPCC are mistaken.
Chickens do come home to roost. Mistakes and deliberate misrepresentations created in backrooms do eventually see the light of day. Truth will out.
There is a big difference between science and engineering. Science speculates and attempts to discover things about the world through theory and analysis of data. The conclusions of science have no value beyond mere conjecture until they are validated by actual events and data that can be predicted by the science. That is where it becomes engineering and crosses over into useful things.
Climate science is in the realm of conjecture at this point. Especially paleoclimate science. Until the science can make perdictions based on measured data — like temperature, pressure, etc and produce results that can also be measured it is just a theory.
At this point in the debate there does not exist any comprehensive measured data sets of acient climates. There are only relative temperatures that can be inferred from things like ice and tree rings. If the inferences are correct then you can guess that the temperatures, humidities, etc. might have been more or less within some large error band in the place where you took the sample. You cannot determine what those temperatures were with any precision. That includes how cold the ice was or how warm the air was.
There is also no data set of the current earth average temperature or an accurate energy balance for the earth that accounts for the entire system. You can only make simplifing linearized assumptions and drop out all the variables you can to get to something that lets you predict that the temperature over the next 4 or 5 days may be warmer or cooler than it was today for local areas. It is called weather predicting. Predicting 100 years in the future for the whole earth to within 1/10’s of a degree is not reasonable at all.
If Mr. Manns computer model can account for the weather over the entire earth 100 years from now it will have been one of the greatist acheivements in human history. Crop failures will be eliminated, droughts will be predicted, floods will be planned for, tornados and hurricanes will be predicted years in advance, and on and on. All this from just tree rings and ice cores!
Now, I have a bridge that you might be interested in purchasing . . . . .
Reply to Paul Gosling and his gold ore…(#22)
Mr Mann and his associates have made a number of far reaching claims…unless we spend trillions of dollars now, we face catastrophy…all this hype based upon the hockey stick. Well, if his programs and methods can’t stand up to scrutiny and peer review, then his results should be pulled/retracted from the IPCC report. Simple as that.
This is not a private company here. This is publically funded research at a tax-payer funded university. Mr. Mann owns absolutely nothing. The university owns the work and the code. If he does not want his results audited – then he should work for a private company (which would then own his work ) and not publish it.
I think McIntyre et al are correct – put random data into Mann’s model – and get a hockey stick out…Mann is afraid to have his work audited…’cause he knows…
Cheers.
As a computer systems engineer who has generated hundreds of thousand, if not millions, of lines of computer code over the last couple decades, I find Mann’s claim about intellectual property somewhat irregular. Intellectual property with respect to software refers to algorithms and coding methods employed. If, as Mann claims, his software algorithms are well known and standard coding methods are used, there seems to be no need for privacy.
If, as it also appears, his source code is owned a University, that body could claim that his computer program and/or source code represents a financial opportunity in that it could be licensed to other organizations for a fee. I suspect that no such claims have been made.
Commercial software is subject to verification and validation rules. Software that is sold for critical financial, engineering, and medical applications are audited as a matter of course. If intellectual property or secrecy issues are involved, non-disclosure agreements are used. Also, it is common practice for source code for proprietary, non-generic, programs to be placed in escrow. Depending upon the importance of the software, the company I worked for would require that outside software vendors demonstrate that the source code supplied would produce the current version of their software.
There were occasions where software vendors were reluctant to allow third party auditing or access to their source code. The obvious example is Microsoft. The source to much of their falls into the industrial secrets category in which coding algorithms and techniques represent business advantage over competitors. The important work above is “reluctant”. In critical business situations, Microsoft allows customer engineers to speak directly with their programmers and even examine portions of source code, under non-disclosure agreement of course. I’ve done that myself on a couple occasions.
Typically, though, the reason why vendors were reluctant to allow access to their source code was simply that it was poor quality or disorganized. That is not to say that sloppy code is not uncommon or that sloppy code won’t work correctly. Commercial production deadlines and budgeting constraints often militate against creating clean code. It is, however, difficult to understand and verify. It is also embarrassing for a programmer when another sees his sloppy work.
So, from a software engineering perspective, I see no justification for the “intellectual property” claim.
Steve: Have you had a chance to look at Mann’s Fortran code? I suspect that one of the reasons why he’s withheld the code is no more than vanity and the prospect of embarrassment. I would be surprised if anyone praised it for its elegance.
Paul Re #41,
Several other proxies than MBH were done in the past years. If one looks at the proxy data only (without the merging of temperature station data), they don’t show a hockeystick, but either a waveform and/or a more U-shaped form. The difference with MBH is that all these reconstructions show a much larger variation in the past, especially a lower LIA and mostly an equal high (or higher) MWP as in the past century. This has consequences for climate models: if these want to project future changes based on their ability to reproduce past changes, then larger values must be incorporated for natural (especially for solar) sensitivity, at the cost of GHG sensitivity.
Most proxies end in 1980 (Moberg even in 1950). The merging of surface temperatures in the same graph has a dramatic effect, as an essential part of the increase in surface temperature is after 1980. But what is known of several proxies (like tree rings in North Europe) is that they don’t follow the surface temperature trend in recent decades. Thus it seems that either some proxies (like tree rings) are not such good temperature indicators, or the station data need an audit (!), or both…
Here is a direct quote from John Hunter a Climate Scientist:
“What I said was “showing there was something wrong in my previous work would serve little useful purpose and could involve significant time on my part in providing data and answering questions”. It is all a matter of the best use of my time “¢’¬? is my time best used in producing the best result I can now based on the latest data, or producing a better result than last time based on old data?”
If something like this were stated by say John Stalker a Building Contractor, how long do you suppose he would stay in business?
JeffN
RE: #47
I must admit that I’m taken aback by that quote of John Hunter’s. If there’s little useful purpose in evaluating a previous work, that can only mean that both its results and methods don’t really matter anymore. I’d sure be disappointed if all my work didn’t really matter anymore.
Jeff Norman (#47):
You say: “If something like this were stated by say John Stalker a Building Contractor, how long do you suppose he would stay in business?”
Thanks — I think this illustrates a very useful point. I am not a “Building Contractor” and Michael Mann is not submitting a mining prospectus. These are DIFFERENT pursuits completely. If you or Steve thinks that scientists should behave like building contractors, then I think this is a very good reason why there is such a huge communication gulf between many on this site and practicing climate scientists.
Re #49: For once I agree with John Hunter – “If you or Steve thinks that scientists should behave like building contractors, then I think this is a very good reason why there is such a huge communication gulf between many on this site and practicing climate scientists.”
Clearly, if the science John practices is of little consequence then I can understand “”showing there was something wrong in my previous work would serve little useful purpose…” and I can understand his attitude even more if it is the case that any errors in his science have no repercussions in the real world.
The gulf between Steve M and John H is simply that Steve chooses to rely on his acumen to make a living – there is a direct relationship between “getting it right” and success – much like a building contractor. John H however, like Mann, can potter away under the protective umbrella of a government or academic organisation where mistakes or sloppiness can be covered up with science-speak, bluster and obfuscation.
If there is one clear message from the “hockey stick” mess, it is that any serious science which advises policy or investment should meet the highest standards of accountablilty and auditability.
John Hunter the climate scientist said:
“I think this illustrates a very useful point. I am not a “Building Contractor” and Michael Mann is not submitting a mining prospectus. These are DIFFERENT pursuits completely. If you or Steve thinks that scientists should behave like building contractors, then I think this is a very good reason why there is such a huge communication gulf between many on this site and practicing climate scientists.”
He seems to have missed the point entirely.
If a Building Contractor does a bad job and does not make good his work, word gets out. Pretty soon no one will contract him because he has a reputation for doing bad work.
It is the reputation for doing bad work that John Hunter the climate scientist has missed. This is not surprising.
If a Climate Scientist does bad work and does not make good his work, word will eventually get out. It may take something as simple as an ex-seaman in Tasmania actually looking at the data, but word does get out.
A reputation for bad work will carry forward regardless of how brilliant any future work may be. Such a waste.
JeffN
The building contractor example is interesting as quite a few legal cases involving misrepresentations and warranties involve building contractors and real estate transactions. Omissions of meterial information have been the topic of much litigation in contracts and are usually held to constitute a misrepresentation.
In my opinion, one of the reasons why MBH98 was so widely accepted was its warranties of statistical skill, robustness, careful proxy selection and relatively even geographical distribution. I do take these warranties seriously. I do not view them as idle “puffs”.
Hunter’s image is curious and condescending. It’s as though he regards “building contractors” are mere tradesmen and climate scientists as a sort of superior caste, who should not have to comply with petty norms applicable to tradesmen.
To: John Hunter
You say:
Funny, I always thought the essence of science is replicability, not getting “a good approximation” of replication.
“A good approximation” is certainly adequate if we were talking about a physical experiment, because two people will never get exactly the same result from a physical experiment. But we are not talking about a physical experiment, we are talking about the results of mathematical operations on a dataset.
Perhaps you could comment on why you personally don’t think replicability is important in mathematical operations, as long as you get a “good approximation”. Would you, for example, believe a scientific study promoting a new arithmetical method that says 500 + 500 = 1001?
I mean, I can approximately replicate their result by saying 500 + 500 = 1000, and my “approximate replication” only differs from theirs by a tenth of a percent, and in your world this sounds like a confirmatory result …
Which is why the keystone of science is replicability, not your “good approximation” of replicability.
If you truly believe your quoted words above, I would be forced to say that you are only a “good approximation” of a scientist … and having read a reasonable quantity of your work, I don’t believe that is the case.
Which makes your comment quite curious, and leaves me to ponder why, like M, B, and H, you are doing all of this ducking and weaving about what is at base a very simple concept, the idea that science should be sufficiently transparent that the results of a mathematical operation on a dataset can be replicated, not approximately, but exactly.
w.
PS — If someone were proposing to spend a billion dollars based on my work, I would want it to be replicated first by someone else. I trust my own work … but a billion dollars is a lot of money. And if I, like the nations “guided” by the IPCC, were proposing spending billions of dollars based on someone else’s work, I would want it to be replicated first.
Willi Eschenbach (#53): You could have saved this tediously long posting if you had simply read my original postings (#8, #24 and #42) which explained why I used the terms “approximation” and “approximately”.
Steve (and all the others who think climate scientists should model themselves on building contractors):
Steve (#52) says:
> as though he regards “building contractors” are mere tradesmen …..
Sorry, but you do not have the benefit of knowing why I see the “building contractor” analogy as so farcically inappropriate. A couple of years ago I employed a building contractor to work on my home, which he promptly contaminated with asbestos dust. I had to take him to court to force him pay for the cleanup. He is still in business.
So please lets forget the “building contracator” analogy ….
white or blue asbestos?
Re #55 But just a minute – ‘A few years ago the IPCC employed a climate scientist to work on their third assessment report, which he promptly contaminated with the “hockey stick”. No one has yet taken him to court to force him pay for the cleanup. He is still in business.’ The analogy gets better.
John Hunter wrote: “If I sensed that providing such data would result in interminable future questioning and wasting of my time, I certainly wouldn’t provide it.” and “Showing there was something wrong in my previous work would serve little useful purpose and could involve significant time on my part in providing data and answering questions.”
I came into this late, so bear with me…
Thera appear to be two sorts of issues are involved here, legal (intellectual property, etc.) and ethical. As I am frequently baffled by the former, I shall reserve my opinions for the latter issues, although I may still be baffled there…
Since I haven’t done (or seen) a relevant poll of scientists, I can’t guess what proportion would agree with John Hunter. I DO know that I do NOT agree with him.
Taking the second quote first: finding an error in my previous work might not be useful but it would depend upon (a) the potential significance of the error and (b) its context. If I were made aware of an error, or potential error, that I did not know of, I would want to identify and confirm the nature of the error and assess these issues (significance, context). It might be that the error was of little significance, or merely represented a difference of opinion on methodology (although that itself could be significant), but I don’t think that I would just assume that (as John Hunter appears to do).
The first quote is easier to address: If challenged to demonstrate how a result was acheived, then I think a scientist (especially if publicly funded) has a responsibility to do so (limited only by legal constraints). This responsibility cannot be negated by the scientist “sensing”, somehow, that doing so would lead to interminable questioning and time wasting. It is, however, NOT the scientist’s responsibility to provide either the expertise or resources required to duplicate the original result. In other words, if challenged I believe professional ethics would obligate that I provide sufficient information for someone to duplicate my results but I would NOT feel obligated to assist them to do so (whether I would or not would depend on the circumstances).
(NB: I don’t normally add qualifications but it may be appropriate in this case: I am an ecologist, PhD, working at Charles Darwin University, with nearly 30 publications in refereed journals.)
Hello Keith McGuiness (#58): When I said “showing there was something wrong in my previous work would serve little useful purpose …” you should view it in the context of the previous sentence, which was “I’ve moved on and am about to publish newer results for Tuvalu, based on more recently obtained data”. I also said “little useful purpose” and not “no useful purpose”. If I have a new and better data set, I believe it is better to spend my time analysing that than looking back at old work searching for errors that may or may not be there. Of course, if I believed that there was a significant probability that an error existed AND that it could be propagated on to my new work, then I would of course take it seriously. Also, as I’ve stated elsewhere, any gross error should be notifed by the journal in the form of an Errata.
Regarding my statement that “if I sensed that providing such data would result in interminable future questioning and wasting of my time, I certainly wouldn’t provide it”: if you do not understand what I mean by “sensed”, then you and your work clearly hasn’t been besieged by the contrarians! I can also assure you that I go (probably too far) out of my way to assist serious researchers to understand my previous work (however old).
So, Professor McGuinness, say someone publishes an article claiming that wasps of a certain family follow phermones of some families of ants to their nests and lay eggs in the larva of these ants, and that their study shows this has resulted in the ants of some of these families releasing multiple different phermones to fool the wasps. You have some doubts, for whatever reasons, that this has actually happened and ask the principal author for the exact genus and species of the wasps and ants studied, the exact location of the study and copies of the IR spectras which were used to determine the identities of the phermones. Would you think this putative author would be correct in claiming you’re asking for too much and that she didn’t have to provide this data? That is in essence what has happened to Steve in attempting to audit the Hockey Stick papers.
Dardinger: “You have some doubts…ask the principal author for…”
If the source was a peer-reviewed journal, then the details you refer to would be in the paper; there would be no need to request them.
I have been informed that John Hunter tried to post a reply to my message (#58) but was blocked from doing so. I am interested to know why.
[PS I neglected to state in my last message (#60) that I am not a professor (nor would I wish to be referred to as "Professor McGuinness" even if I were.)
Steve: I did not manually intercede to block Hunter. We have installed Spam Karma, which has some sort of automatic method for classifying spam. For some reason, it classified Hunter’s IP address as a spam originator – no intervention on my part.
John Hunter (#54), I had most certainly read your posts (#8, #24, and #42) before posting. I still do not understand your use of “approximately”, and I suggest you are using it to try to imply that others have replicated Mann’s work.
If we are shown Mann’s computer programs and data, we should be able to exactly replicate that work. Not approximately replicate it, exactly replicate it.
You keep saying others have replicated Manns work … approximately.
Many people are not satisfied with that, especially given Mann’s history. We’d like to see it replicated EXACTLY, without explanations and excuses saying that close enough is good enough.
While close enough may be good enough for you, it isn’t good enough for scienbce. Science does not require approximate replication, no matter how much you’d like it to.
It simply requires replication.
w.
re: #59
Sorry, when you mentioned a university and a PHD I assumed you were a professor.
Anyway, the point I was making was that … Hmmm. It looks like that silly spam blocker deleted my message too. I wonder why? There wasn’t anything offensive about it in the least.
Willis (#61):
> I still do not understand your use
> of “approximately”
I thought I had explained and given examples. Firstly, the context was: “the original data is in the public domain and my results should be approximately reproducible from the information given in my report.” The reason I used the word “approximately” was “because even simple analyses like linear regression can vary slightly (e.g. do we weight the data points by their expected uncertainty? If so, how do we derive the uncertainty for each data point?”. That seems pretty clear to me.
> and I suggest you are using it to try
> to imply that others have replicated
> Mann’s work
Nope — you made that one up — I was simply explaining how I would act “if someone came to me asking for data and source code for the work I did at Tuvalu”.
> You keep saying others have replicated
> Manns work … approximately.
Now, Willis, tell me where I ever said that.
1. Computer programs are covered by copyright (in some cases, patents). (So is most writing of course). There is a property issue associated with code.
2. Looking at code (or reading a copyrighted document) is not the same thing as taking possion of it. SEC investigators do not need imminent domain to look at documents. Nor is their reading for the purpose of investigation considered a seizure (vice a search). Now if they pulbished material that might be a different story.
3. Grants typically produce property (data, samples, equipment) that ends up being owned and used by the PI or the UNI after the initial grant. Some grants deal with this issue. Sometimes, it’s just understood. Sometimes they ask for a uni kick-in under this understanding. In practice, PIs end up “taking possesion” of much of this stuff since they are the ones who know how to use it. But a uni could intervene (and might with really expensive analytical equipment) or might not. Expecting computer programs (with really little real value to the uni) to be treated more strictly than x-ray diffractometers is a little unworldly.
4. Regardless of the ownership claims, it seems to me that code does need to be shown to others since written descriptions are so poor at describing actual algorithms and will not show bugs. If someone wants to keep their property right, to the extreme extent of not allowing inspection or use by other researchers checking papers, fine. But journals should hesitate from publishing those people who hesitate to show Oz behind the curtain…
5. In reality, I doubt that there is much commercial value in Mann’s code. And I think it is extremely ungracious and suspicious (in practice) of him not to show it. Either he’s hiding something or he is just trying to throw logtistical monkey-wrenches at investigators/critics. Note the difference in this behavior from what Richard Feynman advocates of a scientist. (and the normal practice is to share such code…even to just share useful stuff for getting work done.)
Thanks for the trackback, Steve. This is an old, but important thread. We must again address the scientific ethics to determine whether public policies such as cap and trade based on ‘secret’ science can go forward in good conscience. Along with my current article I’m preparing a follow up that will hopefully be published on http://www.climategate.com addressing readers’ points.
I am hoping to draw as wide a commentary as possible. I would be delighted to see contributions from all parties, especially since the general public have begun to take real interest in this controversy.
2 Trackbacks
[…] reviewed Mann’s arguments to justify withholding his source code from the House Committee back in 2005. McIntyre exposes Mann […]
[…] title under NSF policy, (as well as under university policy as discussed here) , would appear to reside with the institutions rather than the individuals. One also cannot help […]