Two New Items on Data Access

There’s a short article “Model verification and documentaiton are needed” in Eos, June 20, 2006, by a geologist, I. Sasowsky, :calling for reviewers to ensure that computer methods are properly documented and archived as part of the review process. Sasowsky notes that prior studies have documented frequent “surprises” and “fundemental errors” in numerical modeling studies, even citing Naomi Oreskes (Oreskes and Belitz 2001) on this:

Trust, but verify”¢’‚¬?this is what editors ask for, and what readers expect, from reviewers of technical articles. As a reviewer, I am growing concerned with the level of trust requested by authors of submitted manuscripts, and the frequent lack of verifiable data and methods. Negative reports in the press [e.g., New York Times, 2005] attest to the worst-case outcomes of such shortcomings ….

Where scientific findings are based on computational analyses, documentation of computer model methods and analyses ought to be a required element of publication. The trust of the public in scientists and our methods depends upon this.

Ian Karucunas, the secretary for the NAS panel, kindly sent me the following reference to a June 20, 2006 announcement by the Research Councils U.K. (said to be equivalent to NSF):

The Research Councils UK Executive Group, the grouping of the eight chief executives of the UK Research Councils, has today published its updated position statement on access to research outputs (http://www.rcuk.ac.uk/access/). …

The paper reaffirms the Research Councils’ commitment to the guiding principles that publicly funded research must be made available and accessible for public examination as rapidly as practical …. and outputs must be preserved and remain accessible for future generations.

I guess no one sent the memo to Jones and Briffa.


14 Comments

  1. David Smith
    Posted Jul 6, 2006 at 9:37 PM | Permalink | Reply

    This is wonderful news. It looks like the grip of certain parties on the process may be weakening a bit. Let the sunlight in!

    The June 20 timing makes me wonder if the NAS and the UK Research Council spoke with each other recently.

    Different GW topic (sorry): The link below, from Woods Hole, mentions a sea creature (the lowly jellyfish) which may sequester CO2. They quote a 4,000 ton carbon per day removal by one large swarm, which I think is equivalent to about 9 million barrels of oil a year. I find that number to be hard to believe: it may be a misquote. Nevertheless, the removal numbers are significant.
    Interesting reading. Perhaps Mother Nature has a few negative-feedback and other dampening mechanisms present in her toolkit.

    Link to a recent study which got no media coverage.

  2. Pat Frank
    Posted Jul 6, 2006 at 10:08 PM | Permalink | Reply

    I wonder if some of the out-speaking, e.g., by Sasowsky, is because of the spotlight you’ve put on the issues of diligence and transparency, Steve. Perhaps some researchers who were troubled, but remained quiet in the past, have decided to speak out because they realized the problem is more pervasive than just the incidences they’ve observed in their own field, and that it needs some public exposure.

    In x-ray spectroscopy, where I work, there are published standards and methods, and everything is in the open. One of my co-workers is in the last stages of producing a data-splining program that he’s going to make open-source. Clearly, standards vary across fields, and if Sasowsky’s comments are to be taken at face value, it appears that calculational geologists have been less than fully transparent of late.

    There may be a ripple-effect of your work going on here, in that the need for transparency and critical diligence in science is coming to the conscious fore all across the board. Lots more people than climatologists are going to owe you a large vote of thanks, Steve.

  3. Steve McIntyre
    Posted Jul 6, 2006 at 10:18 PM | Permalink | Reply

    It would be amusing for someone to run down the Naomi Oreskes citation in Mg Anderson and PD Bates, Model Validation. She’s the person, as I recall, who did the sociological study on AGW support that got Benny Peiser riled. Numerical models surprise me from her.

  4. The Knowing One
    Posted Jul 6, 2006 at 10:52 PM | Permalink | Reply

    In the UK, much paleoclimate research is funded by the Natural Environment Research Council. NERC is currently considering the RCUK position on access, and they state that the “outcome of that consideration … will be made available as soon as is possible”; see
    http://www.nerc.ac.uk/aboutus/access

  5. Lee
    Posted Jul 6, 2006 at 11:06 PM | Permalink | Reply

    re 3:

    Naomi Oreskes (Ph.D., Stanford, 1990) is an Associate Professor in the Department of History and the Program in Science Studies at the University of California, San Diego. Having started her professional career as a field geologist, her research now focuses on the historical development of scientific knowledge, methods, and practices in the earth and environmental sciences. A 1994 recipient of the NSF Young Investigator Award, she has served as a consultant to the U.S. Environmental Protection Agency and the U.S. Nuclear Waste Technical Review Board on the use and evaluation of computer models, and has taught at Stanford, Dartmouth, Harvard and NYU.

    Professor Oreskes’s most recent book is Plate Tectonics: An Insider’s History of the Modern Theory of the Earth (with Homer Le Grand, Westview Press, 2001), which was cited by Library Journal as one of the best science and technology books of 2002. Other publications include The Rejection of Continental Drift: Theory and Method in American Earth Science (Oxford University Press, 1999), “Verification, validation, and confirmation of numerical models in the earth sciences” (Science 263: 641-646, 1994), and “Objectivity or Heroism: On the Invisibility of Women in Science” (Osiris 11: 87-133, 1996). She is currently completing a book on U.S. Navy-sponsored oceanography in the mid 20th century, focusing on the question of how “good science” was done under conditions that including considerable secrecy, and whose motivations were in part explicitly political. Her current teaching covers a diverse range of topics in 20th century science and technology.

    Department of History, 0104
    University of California, San Diego
    La Jolla, California 92093-0104

    http://historyweb.ucsd.edu/oreskes/pages/profile.html

  6. Posted Jul 7, 2006 at 2:45 AM | Permalink | Reply

    The RCUK decisions are mentioned in an article of today’s Science Magazine. The RCUK says that the publicly funded research should be available for free soon after they are published, but let the different research councils decide the details. The Medical Research Council has the most stringent guidelines and asks that peer reviewed research should be deposited in PubMed Central (accessible for free) as soon as possible and anyway within 6 months after publication. The rule applies to grants awarded from October 2006 onward.

    There is no mentioning of publicly funded research of climate change…

  7. Michael Jankowski
    Posted Jul 7, 2006 at 4:17 AM | Permalink | Reply

    Re#3,#5, I’ve always thought it “funny” how nobody criticized her as “not even a climate scientist,” yet that criticism was widely-used against M&M.

  8. Steve McIntyre
    Posted Jul 7, 2006 at 4:45 AM | Permalink | Reply

    Actually, looking at this more closely, they seem to more interested in whether the published articles are publicly available than whether the data is available.

  9. Michael Jankowski
    Posted Jul 7, 2006 at 4:52 AM | Permalink | Reply

    Re#8, that’s the way the data horders would want it interpretted – as long as they’ve got something published for public viewing, they’ll argue that’s enough. It’s quite unclear what specifically “research” covers – just the “output” later discussed, or including all the data generated, etc. And it’s just a “guiding principle” of “publicly funded research,” not a rule. No teeth.

  10. JSP
    Posted Jul 8, 2006 at 12:51 PM | Permalink | Reply

    Sreve,

    Oreskes may be fine hydrologist, but I question her qualifications to perform social science research which requires its own special skills. If I had been the editor of the journal that published her infamous paper I would have subjected it to extra scrutiny because she is an “amateur” in the field. I read her paper and couldn’t understand the categories she chose to classify climate change research. I also couldn’t understand why it had been published since it was at best superficial and at worst misleading.

    It is understandable why Benny Peiser is mad. He’s a professional social scientist, understands how this research is performed, knows junk when he sees it, but can’t get anyone to publish his superior work.

    JSp

  11. TCO
    Posted Jul 8, 2006 at 1:29 PM | Permalink | Reply

    Actually the funny thing is that social scientists (shrinks, “soche girls”, economists, etc.) are better at stats then these climatology physical scientists.

  12. JSP
    Posted Jul 8, 2006 at 4:00 PM | Permalink | Reply

    TCO,

    It is possible. Before I left graduate school I had taken courses in ANOVA, factor analysis, non-parametric statistics, econometrics, research design, multi-dimensional scaling, and philosophy of science. I am very rusty, but it is starting to come back as I get deeper into climate reconstruction.

    It is amazing how these are applicable to climatology analysis.

    JSP

  13. TCO
    Posted Jul 8, 2006 at 4:18 PM | Permalink | Reply

    What are you?

  14. JSP
    Posted Jul 8, 2006 at 5:18 PM | Permalink | Reply

    TCO,

    An ex-college teacher who will soon retire from the New York State Department of Transportation.

    JSP

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 2,897 other followers

%d bloggers like this: