Recently, I made a request to Thomas Bracegirdle, junior partner of Connolley and Bracegirdle, for the model data used in two recent articles: Bracegirdle and Connolley (GRL 2007) about 20th century Antarctic models; and Bracegirdle, Connolley and Turner (JGR 2008) about 21st century Antarctic models. William Connolley is well known in the blogosphere, especially for his zeal in extinguishing heresy at Wikipedia.
The request for collated model data was very similar in form to my request last fall to Benjamin Santer, senior partner of Santer and Associates LLP, for model data used in Santer et al 2008.
Readers may remember my recall Santer’s rude refusal to provide the data, culminating in the cordial Team salutation “Please do not communicate with me any further” – I guess this means “Hasta la vista, baby” in Team dialect.
I recounted this refusal and the progress of several FOI requests in several contemporary posts here here here and here.
After all of this, Santer’s boss, David Bader, sent me an email purporting to “clarify several mis-impressions” – saying that, aw shucks, they had planned to put the data online all along and that my various FOI requests had nothing to do with it. See here.
Connolley and Bracegirdle (GRL 2007) had said that they had assessed “19 coupled models from the IPCC fourth assessment report archive from the simulation of the 20th century,” from the “Coupled Model Intercomparison Project phase 3 (CMIP3) multi-model dataset at https://esg.llnl.gov:8443/ “. They report the collecting several variables, one of which was sea ice, a variable of considerable recent interest. For sea ice, they said that used data from “15 coupled models”, considering “sea ice fraction (the proportion of grid covered by sea ice and not leads) rather than ice extent (the proportion of grid covered by ice of fraction at least 15%)”.
They noted that
… except for CSIRO the models have essentially zero skill. This is because, apart from CSIRO, all the models have months in which their total area falls well outside the observed range compared to satellite observations from Comiso [1999] using the bootstrap method, which verify best against other observations in Antarctica [Connolley, 2005]. All models produce a seasonal cycle with a peak in approximately the right season, though HadCM3 is a month late and NCAR CCSM two months early. IAP FGOALS has vastly overextensive ice, extending to South America.
Their table showing “essentially zero skill” is as follows:
Bracegirdle et al (JGR 2008) examined 13 models for 21st (rather than 20th century) reporting that:
Projections of total sea-ice area show a decrease of 2.6 ± 0.73 × 106 km2 (33%).
On Mar 30, 2009. I wrote Bracegirdle on the online submission form at the British Antarctic Survey, requesting the collated monthly data used in the articles. The underlying data is online at PCMDI, but to extract the monthly data would require downloading terabytes of data and then figuring out how the monthly composites were made – time-consuming clerical operations with related risk of error, that are irrelevant to statistical analysis.
For the purpose of statistical analysis, I was prepared to use the Bracegirdle-Connolley collation – only verifying the collation if issues arose. Here is my initial request:
Dear Dr Bracegirdle,
I read your interesting articles on AR4 models and would appreciate a digital version of the collation of Antarctic sea ice model projections as used in your most recent articles.
Regards, Steve McIntyre
Bracegirdle promptly replied but not responsively, sending me a PDF of his article, rather than the requested data. I replied:
I already had a copy of the article. My interest was in the DATA: the collation of Antarctic sea ice model projections.
Thanks, Steve McIntyre
Bracegirdle cheerfully replied:
I’d be happy to supply the data. Would NetCDF format be ok?
I thought to myself that the Team seemed to have learned something from the Santer episode. (They had, but not in the way that I had optimistic ally thought when I got the above email.) A week or so later, Bracegirdle emailed:
I have attached NetCDF files of the data that were used for Fig. 8 and described in Bracegirdle et al. (2008). I had to convert to NetCDF from PP format (which is what we and the UK Met Office use). Therefore some metadata, such as variable name (look for ‘unspecified’), does not appear in the NetCDF files. Hopefully these are the data that you were referring to.
Cheers,
These turned out to be tiny files. They did not contain the collated monthly sea ice data used for the calculations, but small files of about 80 numbers showing 21st century sea ice concentration change (difference between 2080-2099 mean and 2004-2023 mean) for (a) DJF, (b) MAM, (c) JJA and (d) SON.
Once again, it wasn’t responsive to the request. So one more request:
This is not at all what I was looking for. You sent me data sets that are only 30K. You state in the article “Model data were retrieved from the data portal at https://esg.llnl.gov:8443/ , from which 19 of the 24 available models were found to have the data required for this assessment.”
This is the data that I was looking for.
Regards, Steve McIntyre
Bracegirdle’s responded that he couldn’t supply the data. After initially asking me “would NetCDF format be ok?” and my answering yes, he now said that they had deleted the NetCDF data and that it “would take some time (more time than I have spare!) to retrieve the data again or convert them back to NetCDF”, with his answer ultimately being the same as Santer’s.
Hi Steve
Unfortunately we have deleted all the NetCDF files that we downloaded after converting them to PP format. It would take some time (more time than I have spare!) to retrieve the data again or convert them back to NetCDF. However, all the data are freely available at https://esg.llnl.gov:8443/home/publicHomePage.do – you just need to register. “
I’ve responded as follows:
Dear Tom,
I find it hard to believe that the British Antarctic Survey would permit the deletion of relevant files for two recent publications or that there aren’t any backups for the deleted data on institutional servers. Would you mind inquiring for me? In the mean time, would you please send me the PP format files that you refer to here for the monthly sea ice data for the 20th century models discussed in your GRL article and the 21st century models referred to in your JGR article.
Regards, Steve McIntyre
We’ll see where this goes.
![Reblog this post [with Zemanta]](https://i0.wp.com/img.zemanta.com/reblog_e.png)
Climate Audit 
57 Comments
So, they deleted the relevant data. Gee, how unfortunate.
-snip –
Bracegirdle is temporizingly evasive, instead of Trouet’s “temporally pervasive.”
I hope the casual reader understands this, scientists I’ve worked with delete NOTHING. Nada. Go ahead and try to delete the data in their engineering papers, but you’d better bring a gun or a bat or something. Erasers are not permitted and the whole discussion of applying an eraser to NON FAULTY lab data is idiotic. It costs pennies to store a megabyte. To even imagine in passing that a recent papers data are deleted is beyond reason.
Put me down as a data deletion skeptic, data deletion naturalist or even a data deletion denier.
Sorry Steve, not much more to add. Snip it if you’re in the mood.
Re: Jeff Id (#3),
I second the motion. -snip- Please show me where data supporting research at the professional level has been deleted with anything more then a freak occurrence, with anything less then a most humble apology from the offender, and recognition from the researcher that the research is now at risk of not being supported.
Re: John Norris (#26), I don’t know about that. I can see downloading something, converting it to a usable format, and then deleting the original download. I think that is unwise, but not gross incompetence or malfeasance. What I wonder is why the files were initially offered to Steve in the NetCDF format if that format was the intermediate format. What should have been offered to Steve was the PP formatted files. Steve shouldn’t have had to ask explicitly for the PP format ones; they should have been offered.
.
Now, if they deleted those, then I would agree.
Kenneth J. Stewart and Stephen B. Reed, “CPI research series using current methods, 1978-98”, Monthly Labor Review,, June 1999, p.29
“No attempt was made, however, to recompute the [indices] by applying hedonic regression analysis to the individual [index] prices collected for the CPI during the 1978–98 period. Such an effort would not have been feasible, in part because the early price data are no longer available.”
Meh. Might be stonewalling, might not be.
To me it reads that they obtained the original data from another source, converted it and kept the conversion.
Then deleted the original….since it’s readily available and wasn’t in the format they used anyway.
Would make sense if they didn’t actually use it in that format, since they still have the data they actually did use it sounds like.
Although, like Jeff, I think I’d be inclined to keep all of it, personally.
Now, the claim of not “having time” to reconvert for you might be stonewalling….
Perhaps you should request the PP format data and convert on your end.
My first thought upon reading the exchange is that you could have been a little more patient.
Certainly the Team’s history of avoiding open data access is well documented and undeniable.
But Bracegirdle has no such history (yet).
As I understand the Bracegirdle’s take on the situation:
1. The files in question were downloaded from a publicly accessible site, to which he referred you.
2. The files were deleted after being converted from a format which Bracegirdle’s group does not use.
I really don’t think that #2 is unreasonable.
Jeff’s comparison is not on point, because these files were NOT “lab data”. They are something downloaded off the internet from another group. Perhaps the nature of the specific files used would provide some useful information. But this is a far cry from deleting data that was measured in a lab or even data that was generated by a simulation or analysis in the lab.
It also seems to me (from the very limited amount written here) that the paper in question does consensus climate models no favors.
Given your history of not getting the information you requests, I don’t think your anger is unreasonable, but I do think it is a little premature.
Re: Jason (#5),
Let’s see Jason – they downloaded gigabytes or perhaps even terabytes of model data, then presumably ran some sort of program to summarise it into a form they found useful. They published a paper based on this collated/summarised data. They then deleted the summary data. As we have seen before many times, data referenced on the web can and does change, and not everyone keeps their old data when they make the changes (and given the volume of data that comes out of CGMs, it’s likely that this particular data is not archived for eternity), and nor do they always even keep a record of what was changed and when (a change log). If someone wants to check the paper or some aspect of, or even try a variation they think will provide some useful metric not reported in the original paper, it can’t be done.
This is, at best, bad practice – even sloppy. It’s somewhat surprising to me that such actions would not result in a withdrawl of the paper, either by the authors themselves with an apology, or by the publisher with a note about the journals archiving policies.
snip
The one thing I would say about these episodes, your initial requests strike me as much too short.
Unlike a FOIA request where you have to be careful to include every possible item of interest these requests seem like they would be better served with a fat more pin-point request.
Steve: The problem isn’t my request; it’s that he deleted the data.
Steve, I have downloaded and fiddled around with some of the AR4 ensemble model data from the CMIP3 portal a while back. As I recall, a century of monthly means is about 1 GB per variable per model run. So, you are in the market for about 20 GB of data for the sea-ice fraction stuff. With a bit more information, i.e. which models and time periods, I can easily link that data to this thread from my server. The native format is netCDF.
#5. Where did my language indicate that I was “angry”? I discourage readers from getting “angry” and I try not to get “angry” myself. Here I’m simply documenting another incident. AS noted below, the “data” is a work product from gigabytes if not terabytes of data and is not simply downloaded from another site as is.
#4. As to your suggestion that I request the PP data, did you not read my closing letter in which I ask for precisely that data.
As to deleting the NetCDF files because they are in a format that they do not use – why put them in this format if they didn’t use it? And having prepared these data sets, like Jeff Id, I find it hard to understand the mentality of deleting them. I’ve got a number of old GISS scrapes and I don’t throw them out. I’m working on a home computer not a public sector workstation.
Anyway he presumably he has source code for the operation in which he converted PP data to NetCDF. How long would it take him to run the conversion? Negligible. Or did he delete that as well?
As to the suggestion that I re-collate from PCMDI, this is what Santer argued (but they eventually archived after FOI as I oberved above). The point, that some have missed, is that PCMDI doesn’t have monthly data just sitting there. If it was, Br could just point me to it. But Bracegirdle has collated the monthly versions from terabytes of model data. It’s not a small job doing this from scratch. I have two objections. Any collation introduces pointless risk of error that is irrelevant to statistical analysis. Plus both Santer and Bracegirdle were paid by public institutions to do this collation; I have no interest in doing so over again.
William Connolley has a webpage here on converting from PP files
http://www.antarctica.ac.uk/met/wmc/misc/ancil/
Note: relevant because the paper describes some other ways to measure sea-ice extant (e.g. Comiso and Nishio, 2008), as well as referencing Connolley and Bracegirdle (2007), and includes T. Bracegirdle as an author.
The British Antarctic survey is apparently publishing a paper based upon recent observations of sea-ice around Antarctica. This dataset
I found the paper in the GRL papers-in-press bin, downloaded, and attached the abstract:
Abstract:
Re: ryanm (#10), Has anyone got a public access version of this paper? Bit more info. here:
http://www.antarctica.ac.uk/press/press_releases/press_release.php?id=838
I thought I saw other comment at CA but can’t refind it. Apologies if a location has been given.
$100 to CA that says that Steve will never get anything out of William Connelley. Since Stoat doesn’t work for BAS any more he shouldn’t be able to block anything (in a rational world of course).
April is not yet over ;-)))
“This is not at all what I was looking for. You sent me data sets that are only 30K…”
I think that if you’re asking for a favour, this is entirely the wrong tone to adopt. I am a physicist, and if I received a query like this regarding one of my papers, I’d be very inclined to ignore it, especially if the data being requested was freely available.
Can I ask for the record: what is the difference between Bracegirdle and Connelley deleting the datafiles for recent papers in which they made substantive claims, and Bracegirdle and Connelley simply making substantive claims without bothering with any actual data?
If Bracegirdle cannot produce the files, then Steve should request that the papers be withdrawn by the journals. “The dog ate my homework” shouldn’t be any more acceptable an excuse in academic journals than it is in junior high.
Steve, you do come across as a bit abrupt in some of those emails, which might be taken as angry/rude.
It shouldn’t matter of course, a request for data is a request for data; I’ll disagree with RW that it should be seen as a favour. It’s part of the job.
(Dammit, Steve would you mind deleting my previous comment, I’ve incompetently put an email format in the website box!)
Well for my tuppence worth any organisation that today does not have 24 hour back up facilities for its data regardless of whether a small glazing business or a government department is nigh on impossible.
Yet here we have vitally important data in as much it goes towards shaping decisions like all western governments are now making; being accidently erased.
That is absolutely outrageous and heads should roll for such poor sloppy practice.
-snip
the BAS is funded by NERC:
http://www.antarctica.ac.uk/about_bas/our_organisation/who_we_are.php
NERC has a policy on data holding:
http://www.nerc.ac.uk/research/sites/data/policy.asp
have fun
per
I was parked outside the BAS recently. It was a mild Spring day; people were strolling past in shirtsleeves. Suddenly a woman appeared dressed in a woolly hat, a thick scarf and an anorak worn over a bulky sweater. She walked into the BAS building. Scary, eh?
Why might the raw data from the web not be useful (as steve noted)? Problems you can encounter: duplicate measurements on the same date (delete one? average them?), missing data, outliers. How you handle these makes a difference. Then there is the question of gridding the data, which has numerous pitfalls.
Valid reasons for “data is gone” — you no longer work at the place where data was collected (hardly likely for a 2008 paper, eh?), the person who collected/stored the data kept it in a format no one can now read (it’s happened to me), AND catastrophic computer crash (which means you don’t back up files often enough).
Responses I’ve gotten to data requests: right away, and never. About 50-50. And this is with much politeness and without the CA reputation.
“The dog ate my homework”.
I wonder if it’s the same dog that ate Jones’ homework.
The are two possible explanations. -snip
My first Post…a little off topic but maybe someone can answer it… of interest to me is the problem of a continuing CO2 buildup (levels on top of Mauna Loa are still increasing at exactly the same rate as of early April,2009…) despite the severe global economic recession. One would expect to see a modest drop in the CO2 levels…very odd or is there a severe time lag…..one would imagine our current recession to be a dream come true to the those amongst us who would have us sitting and freezing in the dark…..I have seen nothing anywhere on this topic…….!!!!!!!!!
Steve: unfortunately, this blog cannot be all things to all people. I realize that most people want to discuss the “big picture”. However, editorially, I’ve taken the position that such threads turn into the same points over and over. I’ve therefore taken the position that this blog will deal with finite issues, focussing on verification and statistical analysis, and do not permit this sort of “big question” to enter specific threads, while having sympathy for people who wonder about it. I wonder about the “big questions” myself and regret that IPCC has not provided an engineering-quality A-to-B exposition of how doubled CO2 leads to 3 deg C. When we see such a document, we’ll discuss it here, but until then, we’ll stick to narrower issues where I think that we can improve our collective understanding.
I got stonewalled before CA existed. It is sometimes used as an excuse for not providing data; however, I do not personally believe that it is the reason for not providing data.
Consider Jacoby’s response before CA. He archived a lot of data, but failed to archive a lot of data. It turned out that he hadn’t given Mann his Mongolia data (Mann digitized it from a squiggle) and said so in also refusing to give it to me. As a primary data collector, it seemed to me that he was annoyed at Mann for getting more publicity than Jacoby without ever doing any data gathering himself.
Added to this was the bizarre “few good men” theory for archiving data that supported your story and destroying the data that didn’t.
Also Team members don’t always share. Apparently Esper can’t get data from Hughes.
For me, I don’t much care about motives. Mining promoters have to report all their drill results. So should climate scientists.
Steve, is there a reason why he couldn’t simply supply you with the PP formatted data? I imagine converting from PP to whatever other format you wanted wouldn’t be too trying. After all, what you really want is the monthly composites – that’s where all the work would be. Did he give a reason for not wanting to supply the PP format?
Re: Ryan O (#24),
I can’t think of any reasonable excuse. He asked me if NetCDF would be all right; I said yes. Then he said he couldn’t supply it because he had deleted the NetCDF. I’ve asked for the PP and we’ll see what happens. In his shoes, I would have spent the half hour (or day) recovering the NetCDF files and cursed myself for deleting the public format data. Or as a very poor fallback, I would have sent the PP data with abject apologies and saying that I would arrange conversion help if it proved problematic. I don’t see how he imagined that any good would come out of simply providing nothing.
#26 and others, I have many data sets in R binary form and it would take a little while to make them into ASCII. If I’d used such a data set in a publication, if someone were not satisfied with an R binary format, I’d view myself as obligated to provide an ASCII version or equivalent. ( I don’t view myself as being under equivalent obligation for blog posts, where I think that readers of this blog should be able to handle R binary data.)
Folks, no editorializing on motives. I’ve snipped some such commentary as unproductive.
per makes an important point.
Reading the policy, it makes it quite clear that it applies to the outputs of computer models as well as field data.
#8 Steve McIntyre As to your suggestion that I request the PP data, did you not read my closing letter in which I ask for precisely that data.
Nope, I did not notice that last night….my apologies for suggesting a fait accompli.
That’ll teach me to not post when I’m tired and half-snapped. 🙂
It will be interesting to see the response, which should answer the stonewalling/not question.
If I may, there seems to be a series of assumptions being made on the part of a number of the posters which are contrary to my own after reading the emails.
Perhaps it’s my lack of knowledge on the formats in question (or perhaps I’m missing something else in the emails)….so a question:
Why is the assumption being made by several posters that there has been data loss from a change in file format?
Does the format conversion somehow imply that?
My assumption (that I described poorly last night) would lead to a chain like this:
1/ Request data from third party. Receive data from third party in NetCDF file format.
1/ alternate 1/ Download directly from web-portal. Software at hand collects in NetCDF format.
1/ alternate 2/ Download directly from web-portal. Software converts to PP format on-the-fly. End of steps, NetCDF is never actually saved.
2/ Convert all received data from NetCDF to PP.
3/ Trash NetCDF files, keep PP files and work with those.
Under this assumption, there is no data loss/deletion/canine consumption….all data is retained, but in the PP file format.
The only thing not being retained is the original file format, which is, after all, readily available per re-request.
As I stated last night, I, personally, would be inclined to keep the original received file….but I’m a pack-rat, and there is always a possibility of that just-in-case moment actually occurring. Keeping it saves me having to re-request. And, yes, agreed, in the circumstances not keeping the original would be sloppy.
But, if the data quantity is massive, I certainly would think twice about keeping it.
This, BTW, may answer your question Anyway he presumably he has source code for the operation in which he converted PP data to NetCDF. How long would it take him to run the conversion?. Conversion of a bunch of 30k files is trivial time-wise….conversion of 20, 50, 100, 1000 gigs of data, not so much.
I guess it needs to be said, I do have issues with the Santer mess, as well as Jones and Steig…..but my issue mostly comes from the belligerent tone and attitude of “nobody should be checking my work”, as well as non-compliance with scientific norms of data provision for review. These are primary issues that led to me being skeptical (and by saying skeptical, I’m being kind here) of their claims in the first place.
I’m afraid I’m not seeing that tone and attitude in the current set of emails….at least, not yet.
Re: MWalsh (#29), the PCMDI data comes in gigantic files, which are processed to yield the monthly sea ice data in question as a work product which is a fraction of the original data size. Maybe 1 MB per model tops.
Yes, it’s possible to re-do the work but, as repeatedly observed, such re-collation introduces pointless risk of error. Plus the work has already been used for a publication and should be available (as even Santer’s boss agreed.)
The tone of these emails is more pleasant, but the result was the same. As compared with Santer, Phil Jones was always very pleasant but nothing happened without resorting to FOI.
Re: Steve McIntyre (#30),
Unless your name happens to be Warwick Hughes…
Steve: That brush-off was not a 9 on a scale of 1-to-Santer. 🙂
Mwalsh asks
One would typically assume that a format conversion will not insert errors, given clean input data fed into a correctly written program running on a stable system. But given the possibility for problems in any of those areas (see Saltzer, Reed, and Clark, End-to-end arguments in system design), why wouldn’t you retain the intermediate data? It seems to be an integral step in the data chain, as worthy of archiving as any other data.
In 1997 Connolley didn’t publish some data but did have it separately available. What has he learned since then?
http://ams.allenpress.com/perlserv/?request=get-document&doi=10.1175%2F1520-0442(2000)013%3C1351%3ADOAARC%3E2.0.CO%3B2&ct=1
“I thought to myself that the Team seemed to have learned something from the Santer episode.”
You think Bracegirdle and Santer are working together on this work? Or is that everyone you deal with in climate science is part of the Team? Gasp.
“Once again, it wasn’t responsive to the request.”
I’m sure Tom Bracegirdle enjoys being referred to as “it”.
Paranoia and rudeness. I’m sure that having read this Tom is really eager to help you Steve.
Re: Plimple (#35),
No, I didn’t.
No such reference. “It” in this context, obviously refers to the email.
I only ask that he fulfil his obligations as part of publishing an article and in accordance with being a recipient of public funds. I think that he should he appreciate the interest, but whether he is “eager” to fulfil his obligations is immaterial to me.
Re: Plimple (#35), you seem to be advancing the “But he didn’t ask me nicely” defense as though it were actually rational and reasonable.
.
Asking a scientist to release his data and methods should never have to be done. The data and methods need to be, and should be, made available as part of the usual process of publication. Replication is the central part of the scientific method, and it can’t be done when people hide how they’ve gotten their results.
.
As such, there is no defense for not handing over your data and methods. You can’t just go “He didn’t say ‘Mother May I’ first! He was mean to me.” You can’t just say (as Phil Jones did) that you refuse because they’ll (gasp) look for errors in your results.
.
So what if someone didn’t ask in the world’s nicest way? Who cares? If you want to be an obstructive jerk and refuse to show your data and methods, you deserve all that you get, and likely more.
.
So whether Tom is “eager to help” or not is totally meaningless. If he wants to be a real scientist and not just play a scientist on TV, he has to follow the rules. His state of mind as he does so means nothing. If you want to play, you have to pay.
.
Having said that, I’ve never known Steve to be anything but polite and cordial in his requests for information, as you can see in the record of his correspondence above.
.
w.
And when can we expect that you will be publishing your findings?
Yeah, I thought so . . .
Re: jay alt (#39), ? what’s that all about?
Re #37
This assertion is fine and would receive little argument from most practicing scientists like myself. The problem is how far it is taken and how it is interpreted. Steve takes a pretty hard line that many on the receiving end find pushes the envelope, exhibits a sense of entitlement and is delivered brusquely, even discourteously.
Providing data is an obligation that most of us meet – methods *should* be described in publications, but I will admit that sometimes this is weaker than we should aspire to: reviewers and journals have a way to go. Where I part company with Steve is in the extent to which I should make available intermediate tools.
The handle-cranking and head-scratching that we see on this website (at its rare best) is intellectual property, as is the code that captures it. Sure – it must be described fully, but if “replicators” need to use their own intellectual horsepower to replicate with somewhat different tools then that becomes a far better test. Not that I often mind sharing tools anyway, but that is a somewhat separate issue, and should be something for which the recipient is grateful. I have asked colleagues for code, on occasion that they worked long and hard to develop. I have been grateful that they shared it rather than simply referring me back to the differential equations and physics that would have undoubtedly strengthened my understanding – I was taking shortcuts and their courtesy helped me to a speedy, if lazy and less principle-based, result. This forum and similar commentators often comment adversely on the “black box” garbage-in/garbage-out problem, but sharing code simply exaggerates this, in my experience.
Finally, there is a real logical problem in this mentality of a publicly funded (therefore all ingredients accessible) mentality – do I need to allow an auditor to visit my lab to check my gas-chromatograph, or mass-spectrometer, or whatever? Indeed should an independent investigator be present on my next field trip gathering pollen, ocean sediments, tree corings or whatever. At some point, tools and methods must be left to careful description. If the result is important enough and the concerns significant then replication will surely be driven by these questions – see for example teh replication of the deep Greenland ice cores, GISP and GRIP.
Re: Datastream (#41), Okay, let’s consider replication of the Greenland ice cores. Replication in this case cost $millions, paid for by tax payers. The data are publicly available, but let’s say the authors would NOT release them. Would that be ok with you? The two key world temperature reconstructions, CRU and GISS, can’t be audited. CRU won’t release anything and GISS finally released its code but no one can make it work (last I heard). Is that ok? These are more than just an individual research project–they are government funded labs creating critical datasets. Last October Phil Jones actually published a paper showing a huge urban heat island effect in China, contra claims in the past of 0.05deg C effect only. Did they redo the CRU analysis? ? ? Why not? I bet they find the same thing in the great red spot of Siberia.
Re: Craig Loehle (#42),
Ross McKitrick’s paper suggests half of the current land surface warming trend may be due to under-estimated UHI effects. This factor clearly needs to be re-assesed – especially in light of the Chinese data. IPCC: this means YOU.
I agree. At the same time I am willing to bet there is a large residual warming effect attributable to changes (natural? anthropogenic?) in the Arctic ocean.
Re: Craig Loehle (#42),
Do not we have to be careful in interpreting the magnitude of UHI found at some point in time versus a changing UHI that would be critical to measuring temperature anomalies over time. I personally feel that micro climate conditions and differences over time due to nonclimate changes at the measuring stations have a larger effect than UHI by itself, but all that needs to be determined — as the Watts team has been attempting to do in the US.
Re: Datastream (#41),
If a scientist has his own peculiar reasons for not releasing data (intermediate) that would make replication much less difficult, I say so be it. My point, and I think perhaps that of others here, would be that that point should be clearly made when reporting results in reviews that are precursors for policy decisions, i.e. use these data and conclusions at your own risk as replication may not be readily or in a timely manner performed. I do not think you can have this one both ways.
Re: Datastream (#41),
I understand your concerns, and I agree with most of them in whole (especially the last two paragraphs) and all of them in part. I think a line would need to be drawn between what is sufficient for science and what is sufficient for policy.
.
Though this might be at odds a bit with many here, for scientific purposes, I do not see it as critical that a researcher must provide the specific code used to arrive at a result. I also agree that, in many cases, better science and more learning results from others attempting to replicate the results using physical principles and descriptions of methods (though sometimes the descriptions provided are terrible – a practice that definitely needs improvement). To me, that would be sufficient for science.
.
When it comes to evaluating science for policy purposes – such as the EPA reviewing and using published climate work to justify marching down the path to regulating greenhouse gasses – the primary purpose is not to learn, it is to assess the certainty of the result. It is an audit function, not a scientific one.
.
For an audit to be conducted properly, the data as used needs to be provided. I might even go so far as to say that the calibration of your gas chromatograph needs to be checked. Code needs to be provided and be audited for mistakes and buggy behavior. An audit is not a time to independently replicate results from first principles.
.
While I understand that many would object to this, when (quite literally) trillions of dollars might be spent, the government has a responsibility to take an engineering approach to the issue that necessarily involves auditing the data, code, and methods as they were actually used.
Re: Ryan O (#44),
Ryan,I did not read your post before making mine but I think we are saying the same thing and I think it is an important point. It would also put more pressure on those scientists and groups supporting those scientists who want to influence policy. I think those posters, who sometimes comment here and imply that we should take those good scientists word for it, need to remember “trust but verify” and all its implications.
Ryan O – unless a paper is written specifically to inform policy or as a result of a policy initiative then I dont see how an author will know that this greater standard must be imposed for their specific paper. Do we have reason to believe the papers under discussion here should require such a standard?
Ultimately the requirement must be to define the minimum standard for data and process description in order to support either review or replication. Once that minimum standard is defined then authors and journals should be encouraged to at least meet and ideally exceed that standard. Steve so far has only desribed the loss of intermediate files that will make review/replication more difficult but has not demonstrated that it will make review or replication impossible.
With regard to this specific incident I share some peoples unease with how Steve has handled this request. I try to look at this from the perspective of all parties and ask myself what is reasonable.
1. Is Steve’s request reasonable. In my opinion yes because he has an interest in the data and it would be helpful to him to have it in that form.
2. Were the first two responses by Bracegirdle reasonable. Yes but clearly he had misunderstood what Steve wanted and that confusion was not helpful.
3. Were the third and fourth responses by Bracegirdle reasonable. In my opinion yes because he thought he was providing Steve with what he wanted and did so in a timely manner. I also note that the intermediate files were in the NetCDF format promised.
4. Was the last refusal email by Bracegirdle reasonable. In my opinion maybe. If the data is unavailable due to deletion then it all hangs on whether anything material is lost by the deletion other than to create extra work for Steve. If equivalent data is made available in the alternative PP format then the response may be reasonable.
5. Up to this point I dont have any issue with Steve’s requests. However Steve has then chosen to make this blog post. In my opinion making the post is premature and an unreasonable escalation at least in the form it has been posted. Steve should have waited until the response to his last request for the files in PP format was received. If the request was refused or if Steve can demonstrate that this prevents reasonable review/replication then escalation via this blog would be a reasonable tactic.
6. The post also emphasises William Connolley who was not involved in the email chain and whose input was apparently not sought. It is reasonable to mention he was a co author but it is not reasonable to place a level of emphasis on him at this point and to try to make him appear guilty by association. Steve may be having a bit of inter blog sport by making the snark (I enjoy amusing and appropriate snark) but in these circumstances I feel the snark has an edge to it.
7. Relating this to the very unpleasant Santer refusal is not justified. Steve had 3 main grounds for complaint against Santer ie the refusal to provide data which was actually available, the appalling tone of the refusal and the subsequent handling of the FOI. None of these apply in this instance. The data is not available due to deletion which is a different issue entirely.
Re: clivere (#47), My point is not to impose a greater standard for certain papers. My point is that before policymakers take action that a proper engineering study (which includes detailed auditing) be undertaken on the science upon which the proposed policy is based. To impose those kinds of standards on science in general would be, in my opinion, stifling.
The burden of demonstrating the certainty of results is on the policymakers.
Re: Ryan O (#50),
I think in the ideal case the burden of establishing CIs would be on the policymakers – if policymakers were ideally and simply interested in establishing uncertainties. The policymakers, with whom I have experience in the US, are much more oriented towards marketing a political solution that often is preconceived.
The thinking person, in my mind, has to take the available evidence into hand while at the same time recognizing the limitations of the systems involved and attempting to take those limitations into account. Blogging (and here one needs to be also cautious of looking only at an ideal version and not recognizing agendas) appears to me to be a potential tool better suited for that process than faithfully and single-mindedly heeding the policymakers, the peer-reviewed literature and the reviewers of that literature that might have demonstrated agendas.
Re: Kenneth Fritsch (#53), Agree entirely. Ideally, we could depend on our government agencies and policymakers to serve this function, as that is what they are supposed to do. What actually happens would indicate the situation is far from ideal.
Re: Kenneth Fritsch (#53),
If their preconceived political solutions do not allow for the possibility of uncertainty or error, then it is not much of a “political solution”, is it?
Re: bender (#56),
Frequently, as it turns out, in the end it is not a solution, but I think the politicians/policy makers, in the US anyway, have a storehouse of general solutions waiting for problems, or better yet emergencies to come along to which they can apply a “solution”.
And, of course, the politician/policymaker has to make a go at establishing the possibility of a calamity as part of this process.
Re: clivere (#47),
Your point #4:
PP is a proprietary format, so the data are not accessible to Steve. Bracegirdle deleted the publicly-accessible data. That is the problem and is what makes Bracegirdle’s last refusal unreasonable. Bracegirdle should convert the PP data to netCDF for Steve and save it somewhere, and not make the same mistake again in the future.
Datastream
you should be aware that for much research that the government relies upon, that is the standard that the government mandates. Look up Good Laboratory Practice.
It is also worth noting that this is an area where paper after paper has been found to have severe problems; look at MBH’98, where the original methods were completely inadequate, and even the corrected version is not enough to allow people to replicate the work.
And isn’t that the point of science ? You put enough in the paper so people can replicate. When people cannot replicate (and that is what has happened many times here), then you have to start asking why. You don’t just put your hands up and say, “oh, i am doing something else now, but the original paper stands even if others cannot replicate it”.
And the point is, this is not just trivial stuff. These papers are being used to justify multi-billion dollar decisions, so it is a reasonable request to find out if the basics can be replicated.
per
Re: per (#47),
And since the EPA is now involved, you might want to Google:
EPA protocols audit.
If research is simply to publish papers, such physical audits would be silly. As pointed out earlier, if it’s important enough to influence policy, the comfy world of the ivory tower no longer exists.
Well, I think this is overblown. Just register for the site and get the data. I do a fair amount of poking around in climate-related databases in my research, this is the norm. It’s not like he deleted all records of his data, right? That would “deserve” some of the comments in this thread but otherwise, this is a non-issue.
On April 22, Thomas Bracegirdle sent me the following explanation of the situation – I apologize for not posting this up earlier due to being busy on non-blog matters for the past week, something that I mentioned elsewhere. Bracegirdle writes:
5 Trackbacks
[…] or that there aren’t any backups for the deleted data on institutional servers. Read more here. […]
[…] Steve McIntyre never tires of holding warmers feet to the fire, and they don’t enjoy it much. Coincidence then that the data he needs was deleted? […]
[…] Steve McIntyre at Climate Audit has been one of the major thorns in prodding those in the climate science community to come clean and shows us what they have. In a recent post, McIntyre discusses yet another adventure into attempting to procure the data that was used for a study. Read his post entitled Connolley co-author: “Unfortunately we have deleted all the NetCDF files…” […]
[…] deleting files”, Examiner with a deeper look, The Air Vent with a real deep dissection, and Climate Audit on deleted […]
[…] Connolley co-author: “Unfortunately we have deleted all the NetCDF files…” […]