Generally speaking? No.

There are two aspects to checking that software is correct in this (and most other) contexts:

1) Whatever it is I’m trying to achieve, is it the right thing to attempt in the first place? (are my requirements understood, is my choice of algorithm suitable?)

2) When I try to achieve it, is my implementation correct and free from errors and unwanted side-effects?

Colossally difficult to achieve even for people who’s business is software development; in the hands of a visiting programmer or dilettante, they are unlikely even to know what they don’t know.

The *best* you can do in this scenario, IMO, is to set standards for development in general when code is part of a submitted paper – formatting, naming, scoping, documentation, methodology, test scripts and results etc. etc.

If the included code does not adhere to the accepted best practice of the language at hand, the paper should be rejected out of hand until the software *does* pass at least this shallow hurdle for quality.

This has a twofold effect:

1) it forces some degree of reflection and adherence to practices that are likely to avoid the most obvious errors

2) it makes the job of anyone deciding to attempt any in-depth analysis for more subtle errors that much easier, by virtue of excluding the donkey work of parsing random awful code.

And lest one think that the benefit of code publication is just for those who wish to replicate results, I have to second Steve’s observation that clear, well-organized and well-commented code not only serves the reader, but also the author. It’s common, probably universal, that one doesn’t remember the exact steps even in one’s own program, when one returns to it after an interval of a few months. A lesson I learned long ago.

In the first code that we archived, I didn’t try to have the code do more than act as documentation. However, it later became clear to me that you could easily archive code that was fully turnkey and that readers were interested in being able to generate results for themselves and that this helped them understand the article.

There are two things here obviously. Is it practical? Is it desirable?

I think Stodden’s point is that, among all the checks for whether the paper’s basic claims are reproducible a reviewer can perform (which I know for a fact they don’t do anyway), checks on whether the coding works, is the easiest. So why not get it done?

But in reality, – many a time – reviewers don’t know to code, can’t be bothered to put in hard work and offer genuine criticism (why should I do the authors’ work), eyes glaze over code and data, just want to offer generalized criticism, or more commonly, can recognize bad code when he/she sees it, but doesn’t want to go there too much, because their own coding is exactly as bad. Therefore putting this additional burden on reviewers wont be practical – especially for journals like Nature which want ‘efficient’ reviewers and not nosy people who ask too many questions. Secondly, journals might rightly see this as grunt-work which is properly the realm of the researchers and not theirs and therefore, do not want to take on additional responsibilities. In turn, many strongly feel that this should not be the case – given all the money journals make. See this interview with this researcher at Harvard about the dysfunctional scientific journal market(here)

The other reason I can think of, as to why getting journals to run code verification is not a good idea is that, it artificially adds another imprimatur of credibility that scientific findings seem to gain just by publication.

‘Published in Nature? That means the code must have been checked. No need to look there then.’ or even worse, ‘We hold Nature partially responsible for this error. They provided a statement of code verification.’

If skeptics aren’t able to reproduce results, it just means they aren’t qualified to speak on the science.

Yup, just like all those skeptics that couldn’t reproduce cold fusion.

BTW, how’d that argument work for Eric Steig and Michael Mann, the “we’re not statisticians” duo?

“MEATBALL LOGIC, GENERATES SPAGHETTI CODE”… words to live by…

Perhaps a few climate researchers should have this tattooed on their foreheads in such a manner that it is readable in the mirror.

Can any reader who is familiar with both the peer-review process and the character of the code we are contemplating comment on whether peer-reviewers can be expected to perform any sort of useful check on the basis that these reviews are presently done. Is it too much free work? Would such reviews really detect the aberrations?

Shub, I thought understood what Dr. Stodden was getting at. I just wondered if it was practical.