After nearly 2 months and several inquiries, the editor of the International Journal of Climatology has finally said that they do not require authors to provide supporting data.
Given that funding agencies rely on academic journals to ensure that authors archive data (improperly abdicating their own responsibilities), the moral of this should be that the National Science Foundation and similar agencies should no longer consider the International Journal of Climatology as a qualified publication for the purposes of completing publication obligations under the terms of NSF grants.
The present inquiry arose out of Santer’s refusal to provide data.
On Nov 10, I wrote to the editor of the International Journal of Climatology:
Dear Dr McGregor,
I am writing to you in your capacity as editor of International Journal of Climatology.
Recently Santer et al published a statistical analysis of monthly time series that they had collated from 47 climate nodels. I recently requested a copy of this data from Dr Santer and received the attached discourteous refusal.
I was unable to locate any information of data policies of your journal and would appreciate a copy of such policies.
The form of my request was well within the scope of the data policies of most academic journals and I presume that this is also the case in respect to the policies of your journal. If this is the case, I would also appreciate it if you required the authors to provide the requested collation in the form used for their statistical analysis. While the authors argue that the monthly series could be collated from PCMDI data, my interest lies with the statistical properties of the time series, rather than with the collation of the data.
Regards, Steve McIntyre
On the same day, I received a cordial “Dear Steve” reply, in which McGregor undertook to ask the publishers about data policy, hoping for a response in a few days.
Dear Steve
DELETED BY REQUEST
Glenn McGregor
I wrote back later on Nov 10 as follows:
Thank you for this follow up. For your information, although the Santer SI reported that the inclusion of more recent data does not affect their “H1” results, they either omitted to carry out or neglected to report the results of including more recent data on their “H2” results. I found that inclusion of data up to 2006, 2007 or to the most recent data reverses the conclusions reported in their Table III (using their own methods. I successfully emulated their Table III results for data up to end 1999). It is my intention to submit a comment to your journal reporting these calculations.
I would also like to comment on their H1 data but require the data already refused by Santer in order to carry out the analysis, and, if the data remains unavailable, will, of course, note this refusal in my planned submission.
Regards, Steve McIntyre
On Nov 24, not having any reply, I followed up as follows:
Have you had any response on this yet? For your reference, here are policies at Nature and Phil Trans Roy Soc, both of which require provision of data. In many econometrics journals, authors are required to archive data AND working code at time of submission.
Given the use of climate articles for policy, it is vital that journals have adequate data archiving policies and that they are enforced. In my opinion, you should ensure compliance has been completed (at least in escrow) as a condition of review – this is what econometrics journals now do – as this saves rear guard actions with reluctant authors.
Regards, Steve McIntyre
A couple of days ago, I followed up one more time:
Dear Dr McGregor, you undertook to get back to me in a few days. This has now drifted considerably. Have you determined whether your journal has any policy on data? It seems like a pretty fundamental point and one that your people should be familiar with.
Regards, Steve McIntyre
Today I finally received a reply this time to “Dr McIntyre” stating that it was not the policy of the International Journal of Climatology to require that data sets used in analyses be made available as a condition of publication” and the matter was now “closed”.
Dear Dr McIntyre
DELETED BY REQUEST
Regards
Glenn McGregor
My original request to McGregor was for a copy of the data policies of the journal. I guess that his answer is that there is no policy. In the present situation, I notified McGregor that Santer had already refused to provide the requested data. Now McGregor says that I am “encouraged to communicate directly with the authors”.
McGregor says that the paper was “subject to strict peer review” – as opposed, I suppose, to casual peer review. Yet the “strict peer review” was incapable of noticing that Santer et al had failed to carry out their analysis on the most recent data that they mention and failed to inquire as to the effect of such analysis.
This is very tiresome.
Dec 30 UPDATE: McGregor notified me today that he regarded his correspondence to me (in his capacity as Editor, International Journal of Climatology) as “private correspondence”, chided me for not requesting permission to publish his responses as Editor and requested that his correspondence be removed, which I have done. I retained a quote in which he stated that the journal did not require data “be made available as a condition of publication” as I presume (reasonably, I think) that the policy is not “private”.
Climate Audit 
133 Comments
The circling of the wagons continues.
Is the “International Journal of Climatology” and archival journal?
One would wonder why libraries bother to archive journals. The results of any paper can be requested from the authors and supporting information can be found online. Authors could request peer review and supply these reviews along with the paper for any request. No need for expensive journal subscriptions or libraries to hold copies of journals whose function has now been superseded.
Hopefully this is something the mainstream media might catch up on. I have sent this link to several science editors. Perhaps a public roasting might help open the data safe.
Re: MarcH (#3),
I must confess to on-going confusion with comments such as:
“Hopefully this is something the mainstream media might catch up on”
In my long experience, this type of issue is exactly what mainstream meeja avoid – “too boring; too technical; no public interest; and it is especially in disagreement with our preferred line of propaganda”
Re Tom Gray’s comment.
I suppose that you are pointing out that a journal such as this one is rapidly outliving it’s usefulness.
However, whilst the “results of any paper can be requested from the authors and supporting information can be found online.”… the prospects of actually obtaining such information are diminished where author(s) can simply refuse to supply it.
I had thought that one important purpose of peer reviewed journals was precisely this – that an author cannot hide behind reputation, prejudice or hubris in publishing her/his work, but must make it available for scrutiny by her/his peers. Not merely in for ‘checking up’ (dare I say ‘auditing’), but also to enable replication and advancement of the science expounded therein to the benefit of all.
And lest such author should be reluctant to make her/his work available for all and sundry for their examination, use and perhaps advantage, there are strict rules as to acknowledgement of authorship to counter plagiarism and breach of intellectual property and so on. Such rules ought to assuage an author’s anxieties in that regard.
I suggest that the approach taken by this journal is an abnegation of it’s responsibility to all who might rely upon the provenance of a paper appearing in it. The result of the present enquiry in relation to Santer et al. undermines the reader’s expectations in relation to the authority of the papers published in it.
Perhaps after all I am in some agreement with the referenced comment and the function of such a journal is indeed superseded.
Steve, this and the prior related Santer thread need to be placed more prominently – and permanently – on Climate Audit. Perhaps a new link called “Stonewallers” on the left-hand frame would be in order – something to keep this sort of nonsense front and center.
Re: Sam (#5),
Second to this fine suggestion.
McGregor encouraging you to “communicate directly with the authors” was a nice extra touch.
“Nolo Permitterre Illegitimi Carborundum”
Best for 2009,
Pete Tillman
Re: Peter D. Tillman (#35),
I thought it was Illegitimati Non Carborundum.
Regardless, I second it. It has to be awful frustrating dealing with such a blatant lack of integrity.
Re: jae (#39),
If aging memory serves, this was the wording of a desk-plaque Barry Goldwater kept in his Senate office. Dog Latin, to be sure, in both versions.
Best for 2009,
Pete Tillman
What surprises me is that the real climate scientists, at least so far as I know, never attempt to answer questions relating to data archiving and related matters. Presumably in their world, adherence to accepted professional standards is not an essential part of “good science”.
The problem that I think they face is that, whatever their reasons for this position (which may in fact be valid), the accumulated effect of years of apparent obfuscation, refusal to enable replication and similar practices is that they progressively lose credibility in the eyes of at least some journalists, and the public at large.
This problem is exacerbated by the obvious lack of continued warming over the past few years, while CO2 levels continue to rise. It is further exacerbated by the current economic crisis, which is likely to encourage politicians to find reasons not to spend huge amounts of money on ETS schemes and the like.
It would be interesting indeed to get an informed comment on these matters from a real climate scientist.
This presents a fabulous opportunity for someone to fabricate results from datasets that do not exist – and the International Journal of Climatology will accept it just so long as their peer reviewers rubberstamp it.
Q: If datasets are not required as prerequisites of publication then what do peer reviewers actually do? Do they consult clairvoyants to discern whether the results follow from the data they cannot see?
What a disgrace. This turns papers more into propaganda than science. Reproducibility absolutely requires making data sets available. And without reproducibility, it’s not science.
Perhaps it is time that someone showed one of these journals why data and code need to be archived. Someone could pull a Sokal Style trick.
If some skeptic were to get something published in one of these journals and then come out and say it was all cooked and fairly easy, the press would love it. Perhaps some reader of this blog has the skill and position to pull it off.
But reading this part of the article:
How could one tell the difference?
Gee Whiz – Sad for all of us.
Once upon a time, CNN reported that the north pole would be ice-free this summer. Article here.
I called BS and then followed the arctic melting all summer long. When I was right and the article was wrong, I found out who was responsible for it and emailed them…
I got this as a response.
Clearly, no one needs to be responsible for what they say anywhere in the news regarding climate change or the basis for their statements.
snip – language exceeds blog policy
In other news: International Journal of Climatology to merge with Journal of Irreproducible Results….
Steve,
As for your statement about the NSF etc, surely you don’t actually believe this has any chance of happening. Given that you’ve faced the same barrier over and over is it not obvious by now that science is now a careening train, with nobody at the controls?
Let’s hope the derailing isn’t too gruesome, taking down disciplines that do a good job with those that don’t.
Is there an opportunity to push this further?
The Royal Society and the APS both see themselves as guardians and arbiters of science and the scientific method. They should be invited to comment on this policy of ‘checking science by asking only your friends’. The world would be interested to see their answer….
Is this publication willing to provide a list of the names of these “strict” peer reviewers? (Short of actually purchasing it) Or can we all just assume they are the usual suspects?
I suspect that much of the commentary is well out of order.
Many journals require you to produce original data, such as ice-core measurements, and a description of the methods that you used to generate the final results. That description has to be adequate to enable you to reproduce the original findings.
But I don’t know of any journal which requires you to produce the internal workings of what you did. The only people who can hold you to account at that level are your own University/ Institute, and the funding body; and it is clearly an onerous task to produce the full working for a body of work in many cases.
While I can accept that it would be easy for Santer et al to have archived their working data set and analyses, they did not, and it is not a commonly accepted practice to do so.
Besides which, I do not think it is terribly helpful to have a go at the Editor here. Their position seems reasonable, even if they did not provide a copy of their policy.
The much more constructive issue is to repeat, and point out that you get different results with minimal variation in the data set (and all the other little gems that you pick up are also fair game for comment). Publish 🙂
per
Re: peer review. I am finding it increasingly common in recent years for journals to ask for a list of names of potential reviewers. While clearly it is their choice to use them or not, the difficulty of finding reviewers no doubt often leads to their use of these names. Of course, no one would put forward names that are their friends (wink wink)….
Re: Craig Loehle (#21), even when journals don’t ask for nominations for referees it seems to be fairly common practice for editors to glance through the first dozen references and pick a plausible reviewer from there, so it is pretty easy to steer them in the “right” direction.
Re: Craig Loehle (#21), When I submit a paper, I may request that “XYZ” does NOT review my paper, simply because I do not respect the opinion/work of “XYZ” and colleagues. If you haven’t experienced a paper rejection by an incompetent reviewer (and later saw someone else publish similar work to your rejected work), then you haven’t attended the school of hard knocks.
Re: Dave L (#31), Yes of course you can in some journals request Dr. X not review your work (only some). This does not limit the universe of incompetent reviewers very much, but at least you get to ask that the people who you are explicitly challenging not be reviewers. And yes, I have certainly experienced incompetent reviews, complete with incoherent statements, demands for the impossible, and statements that I did not address concept Y when I have a whole paragraph on Y.
On the one hand, I think Santer is being unreasonable not to provide the intermediate values. On the other hand, I’m not surprised the Journal came back with this response. Given how journals really are run, I would be surprised if they want to take on any major policying activities.
I’m curious about what will ultimately happen with the FOI. Freedom of Information does apply to DOE labs. It would be remarkable if Santer had destroyed all files containing the intermediate results. Most researchers are pack rats when it comes to this sort of thing.
It reminds me of Michael Crichton’s last book, the one entitled ‘Next’.
Re: per (#20)
You say:
This is a really wrong – and I must say – a complete misinterpretation of due diligence and the way science is reviewed properly. Journals are responsible for ensuring what they print is accurate and not false. Moreover, the mater at hand should also be seen in the context that Santer et al. refused a request for data by Mr. McIntyre. A disrespectful and totally out of order act on the authors behalf. Public funded science must be transparent. The best way is for journals to require achiving of:
The demand for full an unconditional transparency is irrevocable. You cannot put any argument forward that would imply that less than full and complete transparency is absolutely necessary. Honest scientist have nothing to hide. Full openness or no respect from your peers. Sorry to disagree so strongly, but I do.
All the best
per, your views are always welcome and I take them seriously.
In most definitions, intermediate products count as “data”. Also Santer et al is a statistical analysis and exact data sets are important for statistical analysis. Otherwise you get into stupid diversionary debates over whether you’ve figured out exactly what the guy did – the sort of dispute that went on endlessly with Mann.
I don’t think that there is much public sympathy for climate scientists on the one hand asking for policy change and on the other hand maintaining property rights in their data/code.
The editor is constrained by the journal policies. So the next step is to ask the journal to adopt proper data policies. I’ve had a couple of small successes in the past: Climatic Change had no data policy prior to my intervention and now they do. Let’s see what this journal does.
Speaking of “strict peer review”, I’ve been unable to find any documents setting out IJC guidelines for peer review. I wonder what they send to reviewers. Or would reviewers simply use their own initiative? Maybe someone else can ask the journal.
I rarely post here, but this gets me. Maybe I should submit a paper minus all it’s citations, after it too has been “rigoursly” peer reviewed (whatever that subjectively may mean). I also wonder who was involved in the “peer” review.
Not that it will do any good, but I have sent an email to your good friend Mr. McGregor about this. As I am just a layman, albeit intensely interested in complete transparency, I’m sure he will give it the same consideration he gave your polite but necessary requests.
Keep up the good fight Mr. McIntyre! There are many who appreciate your efforts despite the wearying struggle!
As long-time readers know, I have no qualifications here beyond what is required to observe a train-wreck-in-progress, but…..
I am wondering if this is a demonstration between what is called “peer review” (where reviewers look for deviations from the Disclosed Truth and punctuation errors) and what ever the process that scientists use is called, where they (try to) reproduce your results using your information.
And I think a Sokal demonstration would be a lot of fun. Maybe you could get Dr. Sokal to put it together, I doubt that there are five more people who would recognize the name.
Is there any mileage in taking the issue up with Wiley, the publisher of IJC? After all, the editors are taking liberties with the reputation of Wiley as a reputable publisher.
For those who are interested, Robert Higgs has a very good article about the “Downfalls” of Peer Reviews!
http://www.independent.org/newsroom/article.asp?id=1963
Following is a letter I submitted to Rep. Mark Souder:
No need. Google “Santer”. The refusals to make the data available are very, very public with even a basic search. It must be particularly galling to have google report your data stonewalling efforts as higher in relevance than any of your papers.
I’d like to state this issue at its most basic level. Any undergraduate science class that assigns the write-up of an experiment teaches you all you need to know on this topic. Let’s all say it together: “Your methods section must include all information required to reproduce your results.” In the case of climate science, where there is no experiment (which traditionally makes it a less rigorous field), any statistical or modelling analysis must logically include sufficient information to re-run the analysis. It is natural that raw data and computations be left out of the publication itself, but it is implied by this most basic requirement that when requested, it should be made available. You don’t need to be a climate scientist, statistician or bottle washer to understand this. If these papers were handed in to a Chem 101 class, they wouldn’t pass muster. Peer review is just the beginning of criticism for scientific papers. The real criticism comes after publication, when knowledgable and disinteristed readers start chewing them up. The failure of the field at large, and all interested scientists in general to speak out against this sort of data hoarding is a disgrace. I just wish I could make my former instructors read this forum and explain why they don’t speak out. They were never shy about tearing papers apart in the classroom and lab. Their silence now is deafening.
Excuse me, I have a paper to write.
“Climate impacts of space aliens”
In my second email to McGregor, I observed that Santer’s H2 results did not hold up when up-to-date data was used.
Now McGregor cheekily sends me a copy of the Santer SI as though this was brand new for me.
If I were in his shoes, I would not be very happy with the idea that Santer’s results did not hold up when data more recent than 1999 was used, particularly when the SI purported to do a sensitivity to 2006. And faced with such a situation, I think that the editor should not be protecting Santer. If they all had clean hands, then they’d be in a better position to stonewall. But the H2 situation isn’t very pretty.
Sune wrote:
yes I can put that argument, and I will 🙂
Firstly is the issue of due process. Journals cannot make such an imposition unless they set it out in advance. I think the problem here is that IJC do not make that requirement. Even where it is made ( and I am thinking of the BMJ) it is put in such a woolly and imprecise fashion as to be purely for show and next to useless.
Second is the issue of time; as Lucia said, journals do not have the resources to act as policemen.
Thirdly is the issue of just exactly what is required to provide the internal workings of a paper. That would mean access (potentially) to laboratory notebooks, computers, medical records, training records, test and equipment validation, etc. Apart from the issues that many institutions do not retain such information (because it is not required), there is a substantial cost in providing the relevant material for someone to look through. There are also numerous problems in terms of confidentiality, reading my handwriting/ obscure notes, etc.
So I fear the standard you set is not in practice approached in many institutions. There are some where this level of diligence is required; in many industrial or Good Laboratory Practice laboratories. But there are different reasons for this.
Well, I advise you to examine how science is actually run in real life. The editors of numerous journals (Science, Nature, et al) are very clear on the limitations of the peer review process, and have said so in print. I do not think I am seriously out of line.
Well, only up to a very limited point; and that point is that they have to trust the word of the author. I don’t know of any journal that does screening of lab notebooks to ensure that the paper is backed up by the interim data. Some journals do screen for obvious falsification; but that is the limit, and journals are absolutely clear of their extremely limited ability to detect fraud/ incompetence.
Couldn’t agree more. As with Mann et al., it was a shock to see a tenured scientist descending into ad hominem abuse, and refusing to provide the data and methods for replication. That anomaly caused me to be interested in this subject, and it has been a fascinating story. So all one can do is to simply note that, and move on. As behaviour goes, Santer’s refusal to provide workings isn’t a crime, and it is the sort of thing that happens frequently.
You just rise above it, and publish your own analysis.
per
Re: per (#42),
Per,
You make a very good case for why governments should never, ever promulgate controversial public policy using science as a basis. Certainly not if science is currently conducted as you describe.
I’m also curious why taxpayers should pay for such “science”. If the public cannot be assured that the results of the “science” they pay for are transparent and reproducible, they should close the pocketbook. If scientists aren’t interested in giving the taxpayers what they paid for, the taxpayers should shut down the funding and the scientists can “just rise above it” (as you say) and keep doing business the way they please.
Steve McIntyre:
I don’t see it as protection here; the editor has done what they ought to have done, and they have said so.
I think the way to deal with this is with a letter (if possible). If you write in and complain, you just add to an Editor’s workload, and there is nothing/little they can do if the reviewers missed it. Editors have enormous workloads, often do it for no financial reward, and have a million sources of hassle; they don’t need more.
If you write a letter/ short article, then you can be extending the science. If it gets in, then there is a little bit of controversy, and the editor will like that- there is a controversy that people are interested in, in his journal !
good luck,
per
No data.
No reproducibillty.
What’s left is just a comic journal.
#43. Ross and I are working on a comment on the H2 results – which merit a comment all by themselves without the H1 data. Santer’s failure to either carry out or report the SENS2 sensitivity study on the H2 hypothesis is quite striking – I noticed it instantly. But not, it seems, the “strict” peer review process.
Re: Steve McIntyre (#45),
It is points as exemplified by this one that are telling in these analyses of papers that are carried out at CA and it is those analyses that are of the most interest to me. While the editor’s reply in this case was probably anticipated, I think Santer’s reply was the important one to note and learn from. Without Steve M making that effort of the request for information and publicizing it here, we would not have another insight into the some of these authors’ reactions and the reactions of others in the science community who come to CA to comment.
It would appear that most of those comments from the science community point to some rather severe limitations on the authority that we should cede to these peer reviewed papers and their conclusions. I judge that it is more important in the case of climate science and its bearing on climate policy to understand these limitations than to be able to change the peer review process in a major way. The next time we hear the IPCC throw out the authoritative reference to the peer reviewed literature and the scientific consensus, we are better able to concede that that is all very nice, but what we really know is that, at least, the papers were not authored by known crackpots.
PS: Glenn McGregor sounds like a great name for a single malt Scotch whiskey.
Re: Kenneth Fritsch (#57),
“PS: Glenn McGregor sounds like a great name for a single malt Scotch whiskey.”
At least in the US, Clan Macgregor is the brand-name for a near-bottom of the line blended Scotch. I bought a bottle once when I was poor. Let’s just say, the brand fits the man….
Cheers — Pete Tillman
Seems to me that if you go very far down per’s “trust me” path, we can exclude the “Methods and Materials” sections in scientific papers altogether and save a lot of space and trees.
Per, 43
You’re assuming that the journal is interested in advancing science. However, just imgaine for a moment the journal prefers instead to advance and protect a certain theory. How would it react when challenged?
per wrote:
I don’t think that they should. But I don’t think anyone proposes that. All they need to do is to check that there is a .zip file that suppose to have all the information necessary to duplicate results in the paper. That would satisfy requirements for archiving. If journal doesn’t want to bother with maintaining that archive, they could require author to maintain it. There are many ways of doing this with minimal effect on journal. For instance, if I wanted to do something like that, I could require a link to archive to be part of article’s abstract.
So you are telling me that they have no backups? They are so disorganized that they can not put things together when they actually write their paper?
I am an engineer. My work affects me and my company. It has no (or very limited) effect on many billions of people. But I keep my raw data and my codes and my documents and everything else in different folders, organized so I can find them. I also back them up to a version control system. I am not following any particular standard and neither I am required to. The only reason I do that is so I can repeat my work myself few months (or many years) later if I need to, and to make sure that single computer crush doesn’t erase my many years of work.
All they need to do is to require proper record-keeping.
After publication, when people actually read the paper and found some problems with it then it absolutely should be a priority for a journal to act as a policeman and find out what is going on. If they don’t do that it will become much bigger problem for science in general. Instead, this particular journal “consider this matter closed”.
I think this is much more of a problem with whole field of climate science, and this journal in particular. “Everyone is doing it” is not a good excuse for something like that.
Mike
#46
bear in mind that, where it matters, higher standards are required. The premier medical journals (BMJ, Lancet, NEJM) have much better rules on disclosure and try harder than IJC to enforce good statistics, etc; that is because false results in these journals can kill people. In the pharmaceutical industry, there is an obligation to work to a strict and rigorous set of standards known as Good Laboratory Practice. Meeting this standard costs a lot of money.
#47
It is a standing complaint that M&M sections are already too brief (to save space). It is not infrequent to see papers with 10 man-years work compressed into two paragraphs, which are so vague as to be useless. While M&M sections are boring to most readers, they are essential if you are trying to find out what happened in a paper; and it really is not unknown for papers to be based on inability of PhD students to work (understand) a complex analytical machine.
#48
when faced with the choice between a conspiracy theory, and cock-up theory, I normally find that cock-up theory fits the facts.
However, in larger terms, it isn’t unknown for there to be fights about scientific ideas. The theory of continental drift was rubbished when it came out, as was the idea that stomach ulcers were caused by a bacterium (helicobacter). Some of the scientific fights become highly personalised, and you can have a “consensus” of people who believe in a wrong idea, and try and rubbish outsiders. Another example is the medical establishment, who bitterly resisted the influx of specialist statisticians; it was all down to the individual’s clinical judgement, and the more eminent the individual, the better their view. The ability of statisticians (epidemiologists) to demonstrate that medics were busy killing their patients by sub-standard treatments was a bitter medicine that was reluctantly swallowed.
per
Steve-
You are conflating two issues with this post.
One is the data archival policies of IJC. If as seems to be the case, the IJC does not have any such policies on the books, then to establish them now would be in the context of this situation. The Santer17 authorship includes some of the most visible, respected, and vocal figures in the climate science community. Why would the IJC editor want to upset or inconvenience these folks? There is absolutely no upside for him to take any action otehr than what he has. Whether it is the right thing to do or not is moot. If you want to improve data archival policies, best to go to journals out of the limelight and before controversy arrives — hard cases make for bad policy, to steal a phrase. If you want Santer17 to release certain data, well, good luck. Maybe the FOIA at DOE will yeild something.
The other issue is that of peer review. Whether or not the editor of IJC cares about certain findings not holding up following publication is also a moot point. Peer review is a cursory examination at the time a paper is submitted. The editor’s job is almost entirely procedural, not substantive. If you want to update/correct the record of knowledge represented in journals, then submit your own paper (as you said you are doing). But the real problem with peer review would not be that the Santer17 was published with some oversights or errors — that sort of thing happens all the time — the real failure is that in the area of climate science, peer review is used more as a gatekeeping function to keep certain views unpublished. I have experienced this first hand (as has Steve). Given the cursoriness of peer review even in the best of cases journals should err on the side of publication of controversial or contested work, rather than squelching. But unfortunately, for many journals this is not the case in the area of climate. So I wish you luck with your own submission to IJC, I suspect it’ll have a rough road.
Pierre Goslin 48:
“imgaine for a moment the journal prefers instead to advance and protect a certain theory”
Well, I wouldn’t think it is so much about the advancement of any particular theory so much as journals of this sort seem most interested in the advancement and protection of themselves and maintaining high esteem among those who “matter” the most. I believe that can sometimes influence judgment on what to and what not to publish and how rigorous the review process. I believe that it is possible that a publication might fear being ostracized or diminished in stature if it published the “wrong” kinds of papers or papers that might prove unpopular among influential members of any particular scientific community. Likewise, they may feel almost obligated to publish papers by more esteemed members of those communities without much regard to any close scrutiny in the review process. In other words, a newcomer or someone publishing something counter to the majority opinion might face a different publication process.
snip
I think this pretty well explains it all. Perhaps informing McGregor of your intent to submit your findings wasn’t the best approach. It may be that he hopes to keep the controversy to a minimum thinking that this might go away. From the noticeably harsh tone of the last letter, I think he had some interesting discussions with some in Santer’s group.
Still, it would be nice if they considered a reasonable policy on data availability. It could only improve their credibility.
To per (various posts)
I was afraid you would make more or less reasonably sounding excuses for having an arbitrarily weak peer-review process. From what I can read elsewhere on this page, many posts also seem to disagree in your defense of smoke-screening results to a degree where it is impossible to challenged results with a reasonable amount of effort. This was disturbing reading indeed.
There is already too much blind trust involved in climate science. We need more checks, not less. After all, we are talking about a science which deals in the atmosphere. Control experiments are hard to come by for obvious reasons. For instance, we sadly cannot go back in time and re-run the world now with no increasing CO2 and then see what really happens. This is a special weakness of climate science.
I think every body knows that not everything down to lab equipment should to be archived – there are after all, limits to our naivity. Read #26 by Mr. McIntyre again – it might be worth while.
Cheers, smileys and other tricks to seem on top of things
Ad jae #47 – very good point. Perhaps one day the Journal of Climatology will no longer require a Methods and Materials section.
Disclaimer: I am not a scientist, nor do I play one on TV.
It is my understanding that Peer Review is not intended to replicate results, but simply to make sure the paper makes sense, is based on valid principles, uses the correct mathematical formulae, etc.
What Steve is trying to do is replicate results. He’s going way beyond peer review. He needs to publish with exactly what he was presented with, and state the results. Then Santer, Mann, whomever, will have to respond accordingly, else their publication becomes a mere footnote.
On matters more important to rational people than the editorial policies of a tinpot journal that is demonstrably of no intellectual consequence, “whiskEy” isz the stuff they drink in Ireland; “whisky” is the nectar of the gods produced in Scotland.
From http://eu.wiley.com/WileyCDA/WileyTitle/productCd-JOC.html
The International Journal of Climatology aims to span the well established but rapidly growing field of climatology, through the publication of research papers, major reviews of progress and reviews of new books and reports in the area of climate science. The Journalb’s main role is to stimulate and report research in climatology, from the expansive fields of the atmospheric, biophysical, engineering and social sciences. Coverage includes:
* Climate system science
* Local to global scale climate observations and modelling
* Seasonal to interannual climate prediction
* Climatic variability and climate change
* Synoptic, dynamic and urban climatology, hydroclimatology, human bioclimatology, ecoclimatology, dendroclimatology, palaeoclimatology
* Application of climatological knowledge to environmental assessment and management and economic production
* Climate and society interactions
http://www3.interscience.wiley.com/journal/4735/home/ForAuthors.html
Announcing an online submission system for the International Journal of Climatology
To facilitate even faster peer-review times IJOC has launched an online submission system that allows authors to upload their files through the IJOC website. In addition the system will allow authors to check the status of their paper throughout the peer review process.
It’s worth reading the rest of advice to authors and then perhaps complain in writing directly to Wiley about Santer.
So, I guess it isn’t really a science journal after all. Is this really the impression they want to leave?
Straw man. Works that consist of statistical analysis or mathematical modelling exist in the form of computer files. No one is asking for rough lab notebooks here. Any data used in a paper exists in easily transferrable form. Steve is not asking for hand written notes, or bar napkins with equations written on them.
Replication of published work is a fundamental of scientific advancement. You read someone else’s paper, replicate their work, and then use it to move on. The failure to replicate resulsts is the classic method for catching scientific failures and fraud. Telling someone to do their own work shows a fundamental misunderstanding of how science advances – at best. At worst, it just serves to shield failures and frauds.
#51. Roger, you may well be right that the submission will face a rough road. However, this is one of those cases where it’s not going to be any fun for the Team, because there isn’t any wiggle room for the Santer 17 – their H2 hypothesis flat out fails using data that they said was available to them. Either way, it will be amusing.
“Our preference is to make stuff up as we go along. This is not the science you are looking for. Move along, move along.”
I have sent the message below to Glenn McGregor. Won’t help, can’t hurt.
“As a lay, regular, reader of ClimateAudit.org, I find that Mr. McIntyre’s most recent post, regarding the failure of Santer and your journal to provide the necessary data for a statistical replication of the paper’s results, brings to the fore, once again, the appalling tendency of many climate scientists and climate science journals to refuse to provide non-author, but otherwise qualified persons, the necessary information to test the paper’s bona fides.
Over the last several years this has been an astonishing eye-opener for me. It would never have occurred to me that “respected” scientific journals, together with their contributors, would conduct their affairs in this manner. Aside from the unprofessional obfuscation, it strikes me as extremely short-sighted. You seem (although I have little doubt that you have read Mr. McIntyre’s post on this latest incident) totally oblivous to the fact that many more individuals across the world, many of whom are highly qualified scientists and statisticians, read ClimateAudit than ever read your journals. In off-chance you didn’t know it, ClimateAudit was voted by the internet as the best science blog in 2007. The days when this kind of thing can be buried in the sand are over.
It has become clear that the so-called “concensus” climate science community is steadily painting itself into a corner from which it can only escape by opening itself up to transparent scrutiny. As the editor of your journal you have the particular scientific and ethical responsibility to see that this happens. Otherwise, you and your compatriots are going to occupy a singularly unhappy place in the history of science.”
Philip F.Howerton, Jr.
snip
Re: crosspatch (#40),
Crichton, RIP, beat you to it, see Aliens Cause Global Warming.
—————————–
Re: Steve McIntyre (#41),
I’ll play along w/ Steve’s ironic faux-naivete’: But that’s why you’d never be in his shoes – you’re a detail-obsessive auditor, and he’s a journal editor who didn’t know the archiving & disclosure policies of his own publication.
Re: PaddikJ (#65)
I just noticed I was partially snipped. Steve – do remember what it was? I honestly don’t – I was just skimming & commenting before I dropped off to sleep (and didn’t bother to save anywhere). Assume it must have been snarky.
thx,
PJ
Re: Ron (#66)
You think RC’s bad, you should have seen the sniping at NRDC’s website for about 18 months after SOF came out (but then his fictional environmental group was a wicked parody of them – NERF (National Environmental Research Fund, or something)).
Re: PaddikJ (#66),
Many thanks for the reference to Crichton—this sure helps to explain the pathalogical hatred for the man expressed by most over at RC. It also gives me some easy to remember non technical talking points when arguing with my “greenie” relatives (most of them will have it in their email bins by now). But there is one thing that leaves me puzzled; why haven’t I seen or heard about this before? I have over four file cabinet drawers full of AGW clippings from the mainstream print media (newspapers as well as half doxen periodicals) from over the past three years —mmmm, maybe there’s a connection here.
Thanks again PaddickJ, and a Happy New Year.
And, if I were Santer, I’d be a little upset that a non-Team member will be writing the next chapters of his paper.
I’d like Santer to answer one question: If a PhD candidate wrote this paper, and wouldn’t supply data, or failed to disclose the adverse results of the H2 test, how would he react?
We’ve already seen one thesis that showed a difference in a previous team paper: Abeneh.
And she was advised by a lawyer not to release the data. That really advances the science.
Does this make “climate science” an oxymoron?
Carbon dioxymoron, at least.
Methinks per is right in this particular issue. Steve McIntyre should commit his time and resources to a better cliamte subject than this. Don Quichotte springs to mind. To be clear, I have the greatest respect for Steve McIntrye and what he has accomplished until now, but sometimes he goes against a concrete wall.
With the new situation in the White House where Obama will make life for the warmists a lot easier, also financial wise, it’s wise to set a firm strategic course. Does CA wants to play in the margin forever, as it is now, or make life for the climate charlatans a lot more difficult by concentrating on a few, but rock hard issues and goes public about it.
Happy New Year to All!
RC is running a post on good reads from 2008. I proposed the International Journal of Climatology as a ‘reliable and trustworthy read’ – but they don’t seem to agree, and haven’t put it up.
There has been “concensus” on the issue of global warming. I read this in papers and all I can do is hope real scientists are cringing at such verbage. My field of study is biochemistry with a background in nuclear power. We have results. You perform the same experiment or statistical analysis with data you get a result. If it can’t be repeated you have learned your experiment has flaws. Maybe a background paper can be written on when consensus became okay in science instead of results. I guess with consensus, data is just not required.
Please note, your comment about Abeneh is inaccurate. She commented that her lawyer advised her not to respond to the inquiries or messages she received from Steve McIntyre because they were “improper,” and the context of the conversation implied she was similarly unwilling to respond to like inquiries from other people. She did not directly comment about releasing or not releasing the data. She did not explain what she believed was supposedly “improper” about such inquiries.
McGregor notified me today that he regarded his correspondence to me (in his capacity as Editor, International Journal of Climatology) as “private correspondence”, chided me for not requesting permission to publish his responses as Editor and requested that his correspondence be removed, which I have done. I retained a quote in which he stated that the journal did not require data “be made available as a condition of publication” as I presume (reasonably, I think) that the policy is not “private”.
Re: Steve McIntyre (#73),
Whatever are the proper duties and responses of Glenn McGregor as editor of IJC, Glenn certainly does not portray himself well personally by 1) not replying in a timely manner to Steve M’s request (why would it take so long to reply when the answer was so apparently obvious as noted by a number of scientists here) and now 2) being apparently not that proud of his reply.
It would appear that the system under which these editors operate is very infrequently challenged and that the editors are, therefore, not forced to articulate the explicit or implicit underlying principles of the journals that they represent. Perhaps they need a CEO who can spare time from day to day operations to contemplate the bigger issues.
Re: KimberleyCornish (#58),
I apologize to those readers from Scotland and Canada who always refer to the nectar of the Gods as Scotch whisky. Although the Webster dictionary prefers whiskey to whisky for the generic drink, I have heard tell that the nectar of the Gods (and priced accordingly here in the US) when referred to as Scotch whiskey limits some people’s images to a yellow colored liquid.
Re: Steve McIntyre (#73),
Jeez, so much for transparency. If the authors of publicly funded research are relying on the Journals to satisfy the requirement to archive data, but the editor of IJC is stating that they don’t archive, then I think the authors and the public need to know that. Given the amount of public funding involved, I should think that McGregor (and his lawyers) would want to issue a public statement making clear that they should not be relied upon to satisfy archiving requirements. But instead, you get a note from McGregor that basically says, “please help me keep this quiet”.
Re: Steve McIntyre (#73), On what basis did he consider the correspondence private? Did he indicate in his correspondence that it was private? Was there an addendum at the end indicating it was such? Or is the standard that anything that proves embarrassing later must be kept private? No doubt Illinois Gov. Rudy Blog considered his conversations about filling Obama’s senate seat private, too.
Re: Steve McIntyre (#73),
McGregor knows that you are a well-known blogger, and that you have already published your letter to him – so if he was concerned about the privacy of his reply why didn’t he request that at the time? He should reasonably have assumed that his reply would be published
Re: #73
AGW is a Country Club and you don’t have a membership, so get the h*ll out. You can’t play golf here. 😉
Andrew ♫
I’m not sure that Steve is wasting his time by drawing a line in the sand. As I understand it, his beef is not with ‘climate science’ per se, but with bad science. He has repeatedly said that he does not reject the concept of global warming, but wants to see it properly proven (or disproven!).
From that point of view, gathering data openly and discussing it openly ARE ‘rock hard issues’. Steve is not fighting AGAINST climate science. He is fighting FOR SCIENCE, in its widest possible sense. Science really needs standard bearers of this kind. Any scientist will do to shoot down a poor hypothesis, but what Steve is doing is keeping the door open for these other scientists to get the data and funding to do that, in spite of attempts by the entire scientific establishment to shut it.
That is why Steve really ought to be a Nobel recipient. Alas, those who bear good advice to a large human organisation which is making a huge mistake are rarely acknowledged, let alone thanked….
Re: Dodgy Geezer (#76), I agree.
Somewhere recently I read that Ben Santer singlehandedly edited the scientists’ report to IPCC so that, instead of it reading as the scientists wrote it, to the effect that they could NOT be certain of human influence in global warming, it read to the effect that they WERE (at least “reasonably”) certain of human influence. Sorry about imprecision of quote but I forget the source at present.
This would mean, regarding #69, that Steve’s issue with Ben Santer is not on the fringe, but dead centre, both regarding players and regarding principles. I think this needs to come to light, become transparent.
snip
Maybe google is the answer to your problems. Ask them to archive all data the way they are archiving books from libraries.
#77. There is no problem with technology. The problem is the refusal to provide data.
This is an excellent example of why the public should expect strict adherance to common scientific standards rather than simply assume them.
#81, 82. Obviously, I do not personally think that a letter from an editor of a journal in his official capacity has any expectation of privacy, any more than a reply from a government agency has any expectation of privacy. The idea is absurd. I asked McGregor today to provide me an on-the-record reply which is unlikely to materialize, leaving a totally insane situation where the journal has in effect refused to reply on the record on whether they even have a data policy.
In passing, there has been a progression of salutations from “Steve” to “Dr McIntyre” to “Mr McIntyre”.
Hehe, they started out informal, then realized that you were addressing them formally and quickly adjusted to your assumed title (must be a Dr. since you do such a good job of auditing them) then realized their faux pas, but wanted to remain “formal” and began again as Mr.
Mark
I am not a scientist although four of my immediate family research and lecture in various branches of science – experimental physics, molecular biology and marine biology – at leading universities. I am, however, an actuary. Neither my scientist relatives nor I can think of any reputable publication to which we have submitted and had papers published that has not required us to make our data available.
In my own discipline the data sources and (if not too extensive)the data itself, the modelling assumptions and the derivation of the formulae used are always made available to the peer reviewers and invariably published as appendices. The only exceptions being where commercial sensitivity is pleaded.
Thank you for publishing your side of the correspondence. We now know that the International Journal of Climatology is not a scientific journal but a propaganda sheet. Climate scientists should now be looking to publish in a more reputable journal if they want their work to be taken seriously.
I am not a scientist I am, however, an actuary. In my own discipline the data sources and (if not too extensive)the data itself, the modelling assumptions and the derivation of the formulae used are always made available to the peer reviewers and invariably published as appendices. The only exceptions being where commercial sensitivity is pleaded.
Thank you for publishing your side of the correspondence. We now know that the International Journal of Climatology is not a scientific journal but a propaganda sheet. Climate scientists should now be looking to publish in a more reputable journal if they want their work to be taken seriously.
Sorry for the repeat – the first did not appear so I altered it sloghtly and tired again. Both posts are accurate.
I sent an email today to Dr. Ben Santer regarding Steve’s data request and attached a copy of The PCMDI mission statement which had already been posted on this blog. In my email, I also attached a copy of Dr. Ben Santer’s Nov 10, 2008 reply to Steve that contained the request “Please do not communicate with me in the future”. Here is my email in it’s entirety and Dr. Santer’s reply:
The Reply
Re: Edward (#89),
Ben Santer says “If he is truly interested in replicating our results, he has all the information necessary to do so”
Is this correct? As Santer has has now stated this as a fact, would be possible to go back to him to explain why it is not correct, and request the absolute minimum that would be required to replicate the results – even after his request to close the correspondance.
I think it would need to be a well thought through letter explaining what you are requesting, why, and what you will do with it (refuting the statement about being unwilling or unable to do the heavy lifting) – not just a short request for data
#89. For the record, I have never portrayed myself as the “victim of evil Government scientists”. What an absurd comment.
It is my opinion that my request for the data set as calculated by Santer was a reasonable request under U.S. federal policies and that Santer should have provided it. It is also my view that this data would be covered under the policies of high-quality science journals (Phil Trans B for example) and it was reasonable to request under the policies and to rebuke the journal for its inadequate policies.
But Santer’s “victim” – LOL.
And Santer asking reader Edward M to let him “do his job” – ROTFLOL.
As a Chemist, this is of interest to me and may be relevant to this blog post:
http://en.wikipedia.org/wiki/Open_Notebook_Science
‘The aim of Open Notebook Science is to make the full record of scientific research available. This enables other scientists to obtain detailed descriptions of procedures, raw and analyzed data to either compare with their own work or to build on. Advocates argue that this can improve the communication of science, increase the rate at which research can progress, and reduce time lost due to the repetition of failed experiments. In particular advocates argue that it enables more effective collaboration[9] and enables new forms of collaboration in which the collaborators are not necessarily known in advance. One of the goals of open notebook science is to “improve scientific communication”.[10]’
Dr McGregor appears to be from across the ditch. A kiwi no less. Perhaps he can finally confirm the whereabouts of the lost city of Wellington. I’m sure the lat/longs would be in the IJC’s extensive archive.
lol what a bunch of clowns.
Next step is to write the NSF and inform them that the Journal does not meet funding requirements.
#96. You’ll have to be careful slagging Mc’s here. John Kenneth Galbraith came from near Guelph, Ontario and went to U of Guelph when it was an agricultural college. He wrote a book about rural Ontario (The Scotch) mentioning McIntyre’s tavern as follows. A city slicker came to the tavern one winter day – perhaps a Starbucks type (like me) – and asked for a martini on the rocks. At McIntyre’s tavern, bottled beer was a little effete for draft drinkers. So the bartender went out back, chipped some “ice” from the outdoor pissoir and placed it in the martini. The city boy complimented the bartender on the “driest” martini he’d ever had.
I know you have explained this before but perhaps it would be helpful to those who may be new here to this issue if you, Steve, could summarize briefly: (1)why Santer’s statement that you “have all the information necessary” to replicate and/or audit his results is not accurate and (2)assuming, for the moment, that you could attempt it with what is available, what procedures you would have to undertake and how much efort would be involved.
Re: PhilH (#98),
Phil: I am sure you know that the real heart of this matter is a failure to follow the basic, time-honored, scientific code of ethics, which should not have to be spelled out in a journal’s publication policies (and it never used to be a problem). You show your work, and you fight with any dissenters. You admit when you are wrong, and you keep fighting. In short, you be a man, not a frigging whimp. As Bender says, science is a blood sport. Why would anyone, let alone someone who calls himself a scientist, adopt such an uncooperative, indeed spiteful, attitude to someone who simply asks him for more information? There are a lot of possible answers, but I cannot think of ONE that would be satisfactory to a thoughtful person. This is disgusting to anyone who seeks the truth. And science is about truth. Period.
#98. This is not as “bad” an example as, say, Lonnie Thompson withholding ice core data on the basis that he’s providing latitude and longitudes of the Dunde Glacier and saying that I’m free to get my own O18 samples.
Santer et al reported on statistical analysis of monthly time series collated from very large data sets at PCMDI – (each time series of say 50K summarizing about 200 MB or so). Santer’s refusal email outlined the collation as follows:
Aside from the snotty tone, this shows that, even stipulating that the “raw information” is located in the terabytes of PCMDI, analysis is required to make the monthly summaries. I’m obviously not averse to collating data and spend quite a bit of time doing this sort of thing. However, in this case, my interest was entirely statistical and I wanted to test statistical results on the data as Santer collated it, without re-doing the clerical work of collating. For a couple of reasons: if I am stipulating the collation, then there’s no purpose in re-doing it; re-doing it introduces the possibility of error and forces extra time to be spent on reconciliation; emulating the methodology is never easy without source code and if there is added potential of collation errors, it simply wastes a lot of time.
Aside from this sort of thing, there are other issues. Is it acceptable for Santer, as a federal civil servant, to discriminate against someone who might be critical? If he’s prepared to share with his pals, isn’t he obligated to share with his critics? Plus there are the terms of his employment. His unit is paid to make this data accessible? “why should he do this sort of work”? Well, one reason is that he’s been paid to do it and the product is not his personal property. Finally, according to my reading of relevant rules, this sort of product falls well within the four corners of “data” under federal policies and the policies of good journals and is requestable. At this point, I’m waiting for answers from DOE both under FOI requests and under their representations to Congress that they make data available. Let’s see what they say. My guess is that someone up the food chain is not going to be amused by this. I cannot imagine a sane manager wanting to get dragged into Santer’s circus.
Re: Steve McIntyre (#100),
Santer is really being petty here, and he’s going to lose.
If your FOI request doesn’t pan out, if you’ll post precise specifications of what you want, I’ll write my congressman and one of my Senators, and I’m quite sure that their staffs will be able to get what you want.
Steve–
Trivia time: Santer is not a federal civil servant. He works for a US DOE lab. The labs are owned by the federal government, but they are operated by by contractors. The employess technically work for the contractor, not the Federal government. It is a very odd arrangement.
Having said that: the projects he works on are almost certainly funded by the US Government, and the the FOI applies to DOE lab. Whatever the case, the intellectual property is not, technically Santer’s. So, other than the picky detail that he is not a federal civil servant, the rest os spot on.
Not quite on thread but:-
Intresting reading in the chapter “We are not amused” on conflicting theories and data availability in Brian Sykes “The 7 Daughters of Eve” Corgi Books
This whole situation with Santer et al reminds me of the circumstances we faced twenty years ago in the civilian nuclear waste repository program when we were trying to adapt the theory and practice of NQA-1 quality assurance programs to what was, in its initial phases, mostly a scientific research project as opposed to a design-build engineering project.
As software developers and information architecture analysts working in an NQA-1 environment, we experienced intense resistance from the scientists when it became necessary to enforce an enhanced level of process discipline on their ways of doing work, especially their highly individualistic ways of documenting and archiving the data and information associated with their data collection and data analysis activities.
This meant imposing uniform standards on how data and information is gathered and maintained so as to make it more easily and reliably retrievable for future use, not only by the scientists and engineers who need the data in carrying out their day-to-day responsibilities, but also to make the information easier to retrieve for those who are in an oversight role, directly or indirectly, officially or unofficially.
A common refrain heard from the scientists was this: No one but other scientists working on our own team will ever be looking at this material in any kind of real depth, so there is no point in taking the time and trouble that is necessary to archive the data using a mapped, end-to-end process-flow approach; i.e. an approach which allows a reviewer to fully trace the pathway from data collection on through information analysis on through to the individual final conclusions as written up in specific reports.
To accomplish this objective for the civilian repository program, it became evident early on that it was going to be an expensive and time consuming effort to implement the kinds of information architectures and the kinds of operational procedures needed to allow full end-to-end traceability of information and conclusions, both final and intermediate. A common question from the scientists was, where is there money and time in our budgets and schedules to do this kind of thing?
The response from DOE and contractor management to the scientist’s complaints was this: for the purpose of licensing a civilian repository with the NRC, most all of the research and analysis material that goes into the license application has the status of being a “quality assurance record” from an NQA-1 perspective. Therefore a highly disciplined and traceable approach must be employed end-to-end in its collection, analysis, and use. Moreover, the information and analysis is very likely to be challenged in specific detail and must therefore be made available in an organized and useful form to those whose role it is to examine and challenge the validity of the license application’s conclusions.
The next thing DOE and contractor management did for the civilian repository program was to supply the necessary budgetary and schedule resources needed to accomplish the needed upgrades to processes, procedures, and information architectures. The scientists had to go along with this program or else go elsewhere to find employment. Some number of them actually did go elsewhere as they found that the QA requirements cramped their personal styles in ways that were personally unacceptable to them.
In addition to the scientists doing a better job of organizing and documenting their work, what we also did was to use Knowledge Management principles to better organize and archive the results of the scientist’s and engineer’s work so that there was an absolutely traceable pathway from final conclusions back through the intermediate data analysis processes back through to the data collection processes. In other words, end-to-end quality assurance that is fully transparent to anyone who wants to challenge the quality and validity of what was done, or the scientific validity of the final conclusions.
Past experience with government science programs demonstrates that taking a fully QA-ed, fully transparent approach to maintaining and archiving climate science data will be a time consuming and expensive proposition, especially for climate research data that is generated primarily using computer simulations.
Freedom of Information Act requests may be of some limited value in retrieving climate science data and analysis documentation, but the commitment of DOE and NASA management to a philosophy of doing quality work in their climate science programs will be demonstrated only by allocating the level of budgetary and schedule resources needed to build a truly useful process and information architecture, one that serves the needs of the scientists and the auditors alike.
It seems to me that Santer’s position is analogous to this: an auditor comes to audit corp X financials and is told that the raw data is all available in the box of receipts and in their checkbook, but won’t release the method they used for actually doing their books (of which there are varied methods in practice). It is all there–go do it yourself, he is told.
I conclude that from about November 10, 2008 to December 30 the Editor of International Journal of Climatology is unable or unwilling to publicly provide its data policies. This publication can hardly remain credible! Perhaps in the next issue the Editor will provide a clear statement of the publication’s data policies. Until such time how can a reader of the publication or a public that hopes to rely upon the assertions made in its allegedly peer reviewed articles not be incredulous that with over a month to respond the editor can not provide any data policies regarding submissions.
This matter is easily resolved in the next issue to be submitted to print.
re: #100
there is the issue of precedent here; if you accept you must provide working for one research project, then you must accept for all research projects. That is a pretty important precedent, and it has been fought over before.
There is also the possibility that, unless Santer was working to a set of data archiving protocols, he may not have the original data set he worked with, nor the interim results.
Re: Scott Brim
I think that is a pretty insightful analysis. Apart from issues of temperament, scientists are often rewarded for their output (papers, grants received), whereas the due process issues (documentation; reproducibility) receive no reward, and given the time required to implement them, are passively discriminated against.
per
Glenn McGregor is the Editor of the Glenn McGregor Inernational Journal of Climatolog (IJC). The Inernational Journal of Climatolog (IJC) is a journal of the Royal Meteorological Society (RMS)published by Wiley Inter-science. The Royal Meteorological Society (RMS) is a principal proponent of “Climate change controversies: a simple guide” published by the Royal Society. HRH Prince Charles and HM Queen Elizabeth II are supporters of these organizations, their publications regarding climate change, and their activities regarding the need to combat climate change: “…my great fear – a long-held one, for which I have been roundly abused and ridiculed – is that by the time these problems are understood and addressed it will be too late…(HRH Prince Charles).
Given the above facts about the nature of the support and supporters of Glen McGregor and IJC, by what reasoning is it to be expected they will permit the principles of the scientific method and data policies of the publication to supercede their assumed urgent need for the immediate application of a precautionary principle before the “problems are understood”? Why would anyone expect them to care about serving the scientific principles and ethics above all else or the policies of the NSF (National Science Foundation) when they have already expressed their belief that the threat is too imminent and the danger is too great to wait for results from the exercise of the scientific principles?
I have to call BS on the reply by Santer, as well as McGregor. “The data is publicly available” Ok, great – in that case:
How does one discern that the data was not modified, as compared to the data Santer used? By modified, I refer to files being added, deleted, or updated since he grabbed his as input? At the very least, you’d need a copy/paste of a directory (directories?) listing to confirm dates and sizes, and none was offered. I’ve seen enough cases where such files are modified without notice to not trust any claim of “it’s out there and publicly available, go get it.”
Best of luck and Happy New Year Steve!
While it is true that most scientists are not rewarded for archiving their data, and some are sloppy, it can hardly be true that within a month of a paper coming out a scientist has lost his data–years later maybe, but not so soon.
Re: Craig Loehle (#111),
Yes, and most scientists I have known keep their data at least until they retire, but most probably until they die (you never know when you may need it…) Scientists tend to be very anal that way, IMHO.
Re: jae (#112), a month ago I had to search in my 25 years old notebook for some detail and was very glad to find it. And no, I will not throw it away when I’ll retire.
Terry–
What Santer means is output from the models is available in gridded form at whatever time intervals at PCMDI. This is true.
However, what’s not available are the monthly temperature values Santer used in his later calculations. Those can be calculated by Steve. My guess is Steve has already done so, and always intended to do so. However, suppose Steve finds a discrepancy at some point? When checking to make sure you fully understand every step, it is often useful to have intermediate values.
It’s not uncommon for people to make the request Steve made. If the intermediate values still exist, these are nearly always granted. So, it’s odd for Santer to refuse.
But he has.
It may turn out that Santer will be forced to turn over the numbers if he still has them in a file, on his computer etc. FOI applies to labs. It’s rarely used to obtain details about research activities of this sort because normally the researchers are happy to answer questions. However, FOI but does apply. So, we’ll see how it is ultimately interpreted.
How many of the bloggers on this site have bothered to read the paper by Santer and his 16 coauthors (who are scattered around the world). The paper discusses four different sets of data on satellite atmospheric monitoring (all producing slightly different end products), two radiosonde data sets (from UK Hadley Centre and University of Vienna, both adjusted for inhomogeneities – and that opens another can of worms), four different surface temperature data sets (based on reconstructed sea surface temperature data sets from Hadley Centre, again, and Climate Research Unit). Are they seriously proposing that IJOC should archive all this ‘raw’ data so that others can try and do the analysis again; or do they want the initial data from which they are derived?
The average journal has about 10 papers most of which utilize data sets of varying sizes. For instance the first paper in that issue (by Douglass and 3 others also on tropical temperature trends) of the journal utilized another seven data sets; should the journal have archived them as well?
In fairness has Steve McIntyre asked for those data sets as well so he can replicate their results, or is he just after Santer?
Re: Brian G (#117), I have read the paper. The number of coauthors is irrelevant: at a certain point these things become petitions rather than papers. When I assembled the data base for my 2007 JGR paper I had to deal with at least as much underlying size and complexity in extracting numbers across lots of different global economic data bases (including UN educational data, the Penn World Tables, the CIA World Fact Book, etc), as well as climatology data bases. Archiving the data doesn’t mean duplicating all these underlying data sets, it means archiving the data as used in the statistical analysis, which I did on my website. A small file of intermediate data is at issue here, not the terabytes of unprocessed GCM output. Santer reported some variance estimates for trends in synthetic MSU data after averaging up to the hemispheric and global level. That means he has a couple of time series for each model which are fed into the statistical algorithm. That’s the data he refuses to release.
Re: Brian G (#117),
The idea that anyone who wants to critically examine a scientific paper is “after” the author is antithical to scientific practice. I spent five years in grad school taking apart papers once a week in lab meeting, regularly in class, and constantly among classmates informally. It’s what scientists do – they read each other’s work word by word, graph by table, searching for flaws. If you can’t take that kind of light (and heat) shining on your work, you need to get out of the scientific kitchen. Given the stakes in this case – a planetary restructuring of civilization to save the very existance of life as we know it – a request for data seems like a small request to me.
I don’t recall anyone being concerned about questions to the cold fusion people. Was the world of physics “out to get them?” As Samuel Clemens said: “If you’re telling the truth, you don’t have to remember anything.” And if you’re doing good science, you don’t have to withhold your data.
Brian G, 17 authors is a farce. That is politics not science for a start.
It seems most of them merely contributed their names. If they even all read the final paper before submission, let alone checked the data and its manipulation, it would be surprising.
If I had published Santer et al, I would have been prepared to archive the data versions that I used (as collated) with detailed provenance; the source code to derive the results: I do this all the time even for blog posts and regret not doing it for every post. If 7 data sets are used, then yes all 7 data sets.
My interest in this paper was, at this time, specific to the statistical test. There are many data sets in the world that I haven’t tried to examine. I can’t do everything in the world. I’ve already got too many topics in play. My own focus is on studies applied by IPCC, not ones rejected by them.
I have an open offer to provide space here for people doing critical statistical studies of any climate study – be it of Douglass et al or anyone else, but do wish to maintain an IPCC focus.
#118. A couple of the co-authors told me that they never saw the Santer data; the NOAA coauthors said that they never saw it (per the FOI response). At this point, I doubt whether anyone outside of Santer’s subordinates ever saw his data set or verified his analyses.
#98 When I reanalyzed the Drought Exceptional Circumstances Report, the data were supplied in Excel files, and Steve prepared some convenient R objects, but I did not use them. As my interest was in the statistical analysis and conclusions drawn from them, it was very important to use the actual supplied files. In fact, the script read directly from them each time, so I did no reanalysis of them. Its in order to eliminate any doubt that you might be working off different data that the original supplied data. It seems pedantic, but then I could go back and run it again on a fresh download of the files, and check the results were the same. (Data sets have been known to change unannounced).
I have written to Prof Paul Hardaker, the CEO of the Royal Meteorological Society, to see what his position is on data archiving.
Prof Hardaker has a blog, here.
The Royal Meterological Society is going to consider adopting a new policy on archiving at the next meeting of its Scientific Publishing Committee.
Link
#125. Progress. I’ll post this up separately.
I wonder if WoW power levelling can be used to mitigate AGW…
I’m reminded that Pat Frank (and my) alma mater solved this problem wayyy back in 1991.
Check out the Reproducible Electronic Documents website, and in particular the white paper linked there.
A money quote:
This, to me, seems to be the key to the dispute with Santer:
“Even if the reader receives the author’s source files (a feasible assumption considering the recent progress in electronic publishing), the results can be recomputed only if the various programs are invoked exactly as in the original publication.”
In a way it’s encouraging to see these people resorting to childish name-calling. It’s a sure sign they know they’re starting to lose the argument.
I see that they still use the term ‘climate change denier’. If you think about it, that’s completely bizarre as one of the main sceptical arguments is that there is always climate change. The essence of the hockey stick is climate change denial over the previous ten centuries. It’s actually the IPCC and their followers who are climate change deniers.
Chris
“Nothing is new below the skies…”
Re 118 and 120. If authors want their names included as coauthors it is their responsibility to check the final paper. Too often they just want the glory without doing the hard yards of reading what is going out in their name. In my experience it is common for a Prof and or supervisor to have his/her name included in the list of papers written by their students. In other cases of multiple authorship some of the authors merely provide data and the inclusion of their name is a shorthand acknowledgement of this fact. It is a pity that so few people have faith in peer review these days but I suppose that is because the ‘peers’ can be carefully chosen or not, as the case may be!
I think email 1233245601 does a good job of explaining why IJC didn’t require data archiving:
There is now an entry under “Data policy” on this website:
http://onlinelibrary.wiley.com/journal/10.1002/(ISSN)1097-0088/homepage/ForAuthors.html
Wiley publication ethics
http://www.wiley.com/bw/publicationethics/
6 Trackbacks
[…] Steve McIntyre på Climate Audit har efter två månaders korrespondens med redaktionen för International Journal of Climatology äntligen fått veta att tidskriften inte kräver att artikelförfattare styrker sina studier med data. […]
[…] A clear example of the struggle to bring data in to public view is seen in these posts at Climate Audit: here and here. There are examples of top-quality practice by journals (here), and some that are less so (here). […]
[…] of articles be made available for public review. This round in the battle is described at “Data Archiving not required by the International Journal of Climatology“, Steve McIntyre, posted at Climate Audit, 28 December 2008. Summary: Dear Dr McGregor { […]
[…] Meteorological Society Considers Data Archiving A little progress on this front since my last post on this […]
[…] this refusal and the progress of several FOI requests in several contemporary posts here here here and […]
[…] Dec 28, I reported on efforts to obtain the data through the journal, reporting that these efforts had also been […]