Comments on: Nic Lewis on Forest et al 2006

By: Mike Ozanne

Mike Ozanne — Mon, 02 Jul 2012 13:40:10 +0000

“I think the 400Gb almost entirely consists of the output from about 2000 simulation runs, with varying parameter settings, of the MIT 2D model. All should have identical data structure and format.”

So the bit that *actually* needs to be saved is the approx 400Gb/2001 which should represent the raw data that comprises the model input. About 200Mb. Even if that was a bunch of small messy text files, it wouldn’t be difficult to zip/tar it into a single achive and preserve it.

By: NicL

NicL — Sat, 30 Jun 2012 19:27:19 +0000

I think the 400Gb almost entirely consists of the output from about 2000 simulation runs, with varying parameter settings, of the MIT 2D model. All should have identical data structure and format.

By: michaelozanne

michaelozanne — Sat, 30 Jun 2012 14:14:08 +0000

“400Gb for Forest et al 2008, apparently. I imagine a similar size for the 2006 study. If just annual data had been stored (all that was used) then a database only a fraction of that size would have been involved, I think.”

Well 400Gb of what is the question?

400GB of a SQL compliant database based on uniquely keyed records,correctly defined relations set up as foreign key constraints etc. compress the database, generate the backup file, squeeze that with a zip utility, keep a copy on disc burn another to read only media. simples

400GB of spreadsheets, images,csv files, datalogger records with incompatible file formats, different time signature conventions, overlapping data sets, bespoke code in more than 1 language etc etc, not quite so simple.

Of course the trick is to do the legwork at the start of the project so that at the end you have the simples scenario. Hey if your univerity doesn’t have a database specialist, you can find one in the phonebook.

By: michaelozanne

michaelozanne — Sat, 30 Jun 2012 13:33:32 +0000

“My general impression is that the academy is much like business, only with far less oversight and scrutiny.”

There seem to be two distinct paradigms at work. In the commercial world it’s assumed that everyone will take the p*ss given half a chance. hence contract terms , codes of practice, laws and enforcement agencies. Now we are still hard of learning because every time a new market opens up the same standard chain of events happens.

1)New Market opens up, personal restraint is pronounced adequate
2)P*ss taking occurs
3)Market’s main players set up self-regulation mechanism to reassure punters
4)P*ss taking occurs
5)Government re-inforces self regulation by issuing a voluntary code of practice
6)P*ss taking occurs
7)Government sets up legally binding regulations and an enforcement agency but stresses value of “Light Touch” and “Acting Responsibly”
8)P*ss taking occurs
9)Enforcement agency is beefed up, fines are levied, scapegoats and the unswift are jailed
10Level of urine extraction drops to tolerable levels
11)ex P*ss takers to be found on Fox News whining about creativity being shackled by big government

Academia on the other hand is supposedly driven by the noble search for truth, where man’s (insert old joke here about man embracing woman….)better nature is allowed the freedom to improve the lot of mankind.

the chain of events here seems somewhat shorter

1 Make statement declaring academic objectivity, purity of motive, application of highest standards of ethics, proper compliance with sound practice etc
2 P*ss taking occurs
3 repeat 1 ad nauseam
4 appear on the BBC claiming the debate is over.

By: j ferguson

j ferguson — Sat, 30 Jun 2012 11:45:07 +0000

Had not this science become the basis for “policy” and regulation, I doubt that many of us would be worried about the housekeeping – but it has.

I marvel that some of the leading lights in the climatology business are comfortable “informing” policy-makers (as they like to put it) based on science which cannot be replicated and which has not otherwise been confirmed.

Maybe legislation which proscribes basing, even in part, policy or legislation on incompletely published “science” might go a long way to reducing our exposure to this nonsense.

By: Mooloo

Mooloo — Sat, 30 Jun 2012 11:32:39 +0000

My experience is that most back-ups are mirrors, given the relative low cost of storage these days. That is the data is exactly in the form the scientist stored it in, both in file name and location.

I concede that it will not be findable without the good will of both the university’s sysadmin and the academics who stored it, because of the difficulty of deciphering where it is and how it is labelled. But with that it should not be a problem.

(The alternative is to believe that Forest didn’t clearly organise and label his data and his results from the start. Possible, I suppose. But I wouldn’t trust anyone that slack anyway.)

By: Richard Drake

Richard Drake — Sat, 30 Jun 2012 09:34:46 +0000

Interesting idea, Lucy, thanks. Josh has told me I should talk to you about wiki and climate some time too. Nic’s story has some way to run right now and I feel I’m still learning about the latest technical developments, including improvements in e-book readers and open source generally, where Google’s new Android tablet this week is likely to be important – or something very like it. But your comment is noted.

By: Lucy Skywalker

Lucy Skywalker — Fri, 29 Jun 2012 00:02:03 +0000

Richard, perhaps you are the one to write up this story. I agree, it thoroughly deserves writing up. And I think it matches your call.

Dog-ate-my-data-gate: one story of why postmodern Climate Science is not trustworthy.

By: Richard Drake

Richard Drake — Thu, 28 Jun 2012 12:51:56 +0000

But this way there was only one set of data to lose. That could be called a misfortune. To lose ten, when the moment of truth came near ... Lady Augusta Bracknell might well be inclined to call that carelessness.

By: John Silver

John Silver — Thu, 28 Jun 2012 12:33:55 +0000

If there is 400GB of ASCII text files, that will compress to about 10%, 40GB.
This will fit comfortably on 5 (five) Dual Layer DVD discs.