Based on my limited experience, it seems to me that journal peer reviewing faces an interesting challenge with the increased use of Supplementary Information (and I absolutely endorse detailed SI and obviously encourage even more detailed SI). In a very non-random of articles that I know inside-out (Team journal publications), my conclusion is that, in these cases, the journal reviewers didn’t even look at the SI or even verify that the SI actually exists. Whether or not this sort of flaccid reviewing extends to other journals or even to other climate science articles not involving the Team, I can’t say. But I think that the observations are well supported in respect to flaccid reviewing of Team articles.
It’s one thing if the SI is merely data. But combined with the use of SI to provide data, in some cases, authors use SI to derive results that are applied in the main article. In such cases, the reviewers should surely be obliged to examine the SI as an integral part of the article.
I’ll discuss two examples – the MBH Corrigendum and the recent Wahl and Ammann 2007/Ammann and Wahl 2007.
We got some insight into Nature’s practices with SI because we had a submission to Nature under review concurrent with the MBH Corrigendum. One of the reviewers of our submission said in respect of a Mann reply point that this methodological point was not in the original article or original SI and that it should have been included in the Corrigendum SI. Both myself and Marcel Crok pressed Nature and eventually found out that not only was the Corrigendum SI not peer reviewed, but the Corrigendum itself was not peer reviewed (it was handled by editors); the Corrigendum SI was not even examined by the editors. Examination of SI did not, in general, appear to be included in the peer review process. I must say that it seemed odd to me that Nature did not peer review the Corrigendum. I would have thought that, if they felt that a Corrigendum was warranted, then there was at least as great an obligation to peer review the Corrigendum as the original article.
In addition to providing data, the Corrigendum SI also included methodological information, which, unfortunately, remained very unsatisfactory. Given the problems with MBH replication that had already then been demonstrated, it’s hard not to think that some sort of peer review wouldn’t have improved the MBH Corrigendum SI, whose methodological descriptions remained evasive and unreplicable.
A second example arises with Ammann and Wahl/Wahl and Ammann. Both articles make extensive references to Supplementary Information. Wahl and Ammann 2007 makes 8 references to its Supplementary Information, while Ammann and Wahl 2007 make 6 references. The SI references are not limited to the provision of supporting data. In some cases, they refer to figures supposedly illustrating results in the text; in other cases, they refer to statistics and tables in the SI.
Key discussion in Ammann and Wahl 2007 is exported to the SI. For example, they state:
Only the MBH 1600-network is significant at a slightly lower level (89%), and the much discussed 1400- and 1450-networks are significant at the 99% and 96% levels, respectively. (See electronic supplement for further discussion and details, including code and tables with established thresholds for a variety of calibration/verification RE ratios and for the other WA scenarios examined.)
or
The effect of using “princomp” without specifying that the calculation be performed on the correlation matrix (an alternate argument of “princomp”) forces the routine to extract eigenvectors and PCs on the variance-covariance matrix of the unstandardized proxy data, which by its nature will capture information in the first one or two eigenvectors/PCs that is primarily related to the absolute magnitude of the numerically largest-scaled variables in the data matrix (Ammann and Wahl, 2007, supplemental information).
The latter statement is a foolish assertion about tree ring networks that are already standardized to a mean of 1 (under standard tree ring chronology procedures) and which this have the same absolute magnitude. A pseudo-citation for this claim makes it seem more impressive but the pseudo-citation is not to something that itself has been peer reviewed, but to a non-peer reviewed SI.
But where is the SI? No URL is given in either publication, other than the following:
Additional information and illustrations beyond the present text are provided in an electronic supplement and on our WEB site (http://www.cgd.ucar.edu/ccr/ammann/millennium/MBH_reevaluation.html).
The website http://www.cgd.ucar.edu/ccr/ammann/millennium/MBH_reevaluation.html has not been updated in nearly 2 years and contains no mention of Ammann and Wahl 2007 and does currently not contain any of the promised information. This sentence specifically states that there is an “electronic supplement” additional to the UCAR website. The editor and then publisher of Climatic Change were approached about the location of the SI and neither of them knew anything about it. They said to contact the authors.
All of this strongly suggests to me that the peer reviewers did not have access to the SI – otherwise wouldn’t the editor and/or publisher have had access to it in order to supply it to the reviewers?
This says something about the quality of peer review for Team articles – none of the reviewers even bothered to ask for the SI that was cited in the articles or even bothered to determine whether the SI was in existence. I have no doubt that Ammann did up some sort of SI and that the SI will ultimately see the light of day – but shouldn’t the peer reviewers have considered the SI when they were reviewing the article?
BTW I contacted Ammann about this and another question . The other question was:
In Ammann and Wahl 2007, you state without a reference:
Standard practice in climatology uses the red-noise persistence of the target series (here hemispheric temperature) in the calibration period to establish a null-model threshold for reconstruction skill in the independent verification period, which is the methodology used by MBH in a Monte Carlo framework to establish a verification RE threshold of zero at the >99% significance level.
Can you please provide me with supporting references demonstrating the validity of this statement.
Ammann promptly gave a very surly response.
I must say that I find it disturbing that you don’t appear to acknowledge scientific arguments as such, rather you dismiss anything we say and reduce it to below “low level authority”. Under such circumstances, why would I even bother answering your questions, isn’t that just lost time?
I wrote a very measured response urging him to re-consider this refusal, to which Ammann did not initially reply. I then sent a follow-up, copying Nychka, and Ammann said this time that he would answer in a few days but he had other “pressing” business. That was a few days ago, so it will be interesting to see what his response is.
Climate Audit 
85 Comments
If there is no schedule then the trains are never late.
How is “Can you please provide me with supporting references demonstrating the validity of this statement.” a failure to “acknowledge scientific arguments as such”?
Many scientific journals do supply the SI to their reviewers. I recently recommended rejection (or at the least a complete re-write) of a paper because the SI did not, in my opinion, support the conclusions. Admittedly it was a much less prestigious journal than Nature.
#3. It’s interesting that Nature’s review practices involving the Team were weaker than less prestigious journals.
Interesting, but somehow not surprising, eh?
So you asked Ammann for definition #2 below and what you got was #3.
rig·or·ous /ˈrɪgərəs/
–adjective
1. characterized by rigor; rigidly severe or harsh, as people, rules, or discipline: rigorous laws.
2. severely exact or accurate; precise: rigorous research.
3. (of weather or climate) uncomfortably severe or harsh; extremely inclement.
4. Logic, Mathematics. logically valid.
In the journals I mainly use (ecology type), math/stats derivations and extensive tables can be placed in an Appendix, but this is part of the paper. The Appendix serves to shorten the main body of text so the reader doesn’t get lost. Only data are typically relegated to SI, and many of these journals do not even have provision for SI.
Is anyone really surprised by the use of this tactic to get around review? Have we all learned nothing from reading realclimate? It’s one of the team’s favorite tactics. How many times on realclimate have we seen links and citations to sites that are either a)dead b) inaccessible c) don’t say what Gavin and the rest says they say.
These days when there are standards for most things is strikes me as ridiculous that there is no standard for studies that relate to what some say is the most serious problem facing mankind. When we pick up a piece of electronic equipment we expect to see it covered in letters like CE, FCC, CSA UL and so forth. I have suggested it elsewhere but I will repeat that I think the IPCC should ask the International Standards Organisation – who are less than a block away overlooking the lake in Geneva – to provide standards for peer review and archiving, so as to end the puerile argument that “my peer review is better than yours”. As a starting point one might consider the following:
The ISO has worked wonders in various places, indeed. Good idea!
I keep wondering how long climate scientist are going to be allowed (by other scientists) to drag the appellation “scientist” through a muddy pig farm.
9 (David): a supplementary proposal would be to make the peer-review process completely open [ok to conceal names, if the reviewer is afraid – I have served as reviewer of scores of papers and always revealed my name up front] so that the review report(s) be published as ‘supplementary information’; this is free in today’s electronic world. That way, interested parties can check how thorough the review was. I have seen too many rubber-stamped reviews. Even of one of my papers where the SI was not even available because of a screw-up by AGU.
My take on this is that if you don’t have the balls to sign your own name to a review, your wimpy comments are not worth the paper they’re written or typed on. There’s not reason to hide who you are if you’re not scared of your stupid ideas being brought into light. It hasn’t stopped our buddy Jimmy from writing to high school textbook publishers, why should it stop anyone else from archiving their data or making their algorithms available along with their code?
Those that have nothing to hide, hide nothing. Those that hide things have no complaint about others questioning why.
Simple, no?
13 (Sam): Agreed, of course. Simple, Yes! Yet two-thirds of all reviewers wish to remain anon. For me that is OK. They are wimps, but so are most people, but at least the report should be open.
I’ve also reviewed many papers and always, but always, include review of the SI to be sure it supports the authors’ points. There have been times that I’ve rejected a paper because the data in the SI showed the chemistry was wrong, or showed methodological errors. All the colleages I’ve asked (few, admittedly) also review SIs.
“Wimp”iness notwithstanding (13, 14) I prefer to keep my name off reviews to remove the personal from the professional. Even scientists have feelings, and even irrational feelings often trump good sense.
I don’t want my reviews to interfere with my professional relationships, or with my collaborations. I sometimes review the papers of people with whom I’m collaborating on different projects. I wouldn’t want that review to interfere with our joint work, nor would I want the appearance of my name at the bottom to tempt me to hold back on a full and honest review of that colleague’s work.
I also don’t want the reviews of elite figures to carry more weight than the reviews of scientific spear-carriers. Leaving off the name of the reviewer forces everyone to evaluate the review on the merits of the argument. So, I’d go so far as to wonder whether even associate editors should not know the identities of the reviewers. These all seem like valid concerns to me.
About Nature: my own experience and watching Nature respond to Steve’s work and requests for scientific honesty, have caused me to keep forward in my mind that Nature is a science magazine rather than a science journal. The prestigiousness of Nature comes more from its long tradition as the organ of the Royal Society, and less from the scientific power of its pages.
Nature’s editors re-word articles to increase their impact. The chief author on the one paper I have in Nature was pulling his hair out because the editor at Nature persistently inserted re-wordings that distorted the science. Other authors (names in mind but not expressed), may be less reticent about the spectacularizing of their papers by Nature’s editors.
The occasional Nature paper in my field is typically long on claimed significance and short on methodological details. As an experimental scientist, I find such papers largely useless. Nature, however, battens on them.
Steve,
This was posted on Tamino:
“I was a regular visitor and commenter at CA for a couple years. I left because of Steve’s habit of editing what he had said, changing its meaning and seeming intent, without noting that it was a change. It too often left things that had been said, seeming to respond to something different than what they actually responded to.
I left because Steve edited (not scrubbed, snipped part of without notice so that it changed the meaning) a comment of mine, and then denied doing it. This was part of a dispute where Steve was almost certainly wrong – and he deleted the comments after the fact, after responding to some of them.
At least one regular commenter here witnessed much of this.
I no longer trust CA enough to put my words into the care of that site.”
I am a reader of many blogs including yours. I have learned a great deal and find that your site seems to be very good in allowing comments of different viewpoints. So I am very concerned about the above accusation. Please respond
15 (Pat): Again whimpiness aside, the main point of my proposal was that the report itself should be open, for everyone to see, including the response from the authors and any other correspondence associated with the paper [except if commercial interests are involved]
re. 16 (Pat)
I hadn’t considered that side, Pat; thanks for putting it out there.
Since when? The organ of the Royal Society is Proc. Roy. Soc. and has been since before Nature was first published.
Nature was always intended to be more of a magazine than a journal.
Re #19. “Nature was always intended to be more of a magazine than a journal.”
Maybe so, but some years ago ‘The Economist’ published a letter from from Jeff Harvey, who had co-authored Nature’s hostile review of Bjorn Lomborg’s “The Skeptical Environmentalist”. In the course of defending his review, Harvey accused The Economist of ‘smearing the vast majority of the scientific community’ and pffered the following quotation from Ove Nathan, former president of the University of Copenhagen:
“There is no scientific periodical that outshines or is more critically edited than Nature, Science and Scientific American. In science, they speak with almost the same authority as the Bible of Christianity or the Koran of Islam. If all three periodicals pass the same severe judgment upon Lomborg, I personally would take it for gospel truth” (The Economist, 16 February 2002).
#6: 🙂
Very good!
#16. There are two allegations here:
1) that I had edited one of Lee’s posts without marking the snip, supposedly to make him look bad.
2) that I deleted posts on the thread Quantifying the Y2K Error.
You can see Lee’s contemporary complaint here and responses in subsequent posts.
The first allegation is simply untrue. It’s not the sort of thing that I do and particularly with adversarial people. My policy then as now was that if I snipped, I marked the snip. In this particular case, Lee had submitted an off-topic post which, in addition, according to my contemporary comments, had been truncated somehow at his end, although he blamed this on my supposed editing of his post. I denied the allegation at the time. For example here
and here:
Mark T then observed the following in the next post:
I don’t know exactly what happened with Lee’s post. Mark’s explanation sounds plausible to me, but I definitely didn’t edit the post without snipping it as Lee alleged. Lee kept making the untrue allegation over and over. I deleted most of the exchange as a food fight.
Secondly, Lee objected to the deletion of one of his posts on the Quantifying the Y2K Error thread. In this case, the discussion of the quantification of the Y2K error – which was then very topical and in the news, to say the least – had been hijacked by an off-topic discussion of hurricanes, which was highly distracting for any third parties interested in the Y2K issue. TCO made a post, objecting to the discussion as OT. I responded that I agreed with his criticism and proposed that the OT posts be scrubbed and even said that I would wait for his agreement before doing so. TCO then wrote back “scrub away”; so I then deleted the OT hurricane exchange on the Y2K thread. This included the truncated post in controversy.
Lee then objected vehemently to this and TCO responded as follows here:
So it’s a bit much for TCO to support Lee’s indignation.
Returning to Lee’s false claim that I had edited his post, I notice that TCO purports to recollect this matter and now endorses this claim over at Tamino’s. HE says :
and here:
As I noted at the time and again here, I did not edit Lee’s post. TCO’s contemporary observations do not support his present recollection.
Here is my contemporary response on this matter:
Alan Woods at Tamino said the following:
This is precisely the same exchange on hurricanes that TCO asked be scrubbed. Here is an objection by Alan Woods at the time. My response to him at the time was:
In this particular case, as noted above, I discussed the scrubbing of the OT posts with TCO before scrubbing them. So as noted above, it’s a bit much for TCO not to stand up on this matter when others make an issue of it.
22 (SteveMc): I have found Steve’s snipping to be fair and even-handed. Some people say gross things and OT things and posts do sometimes contain special characters, like >, which BTW you can embed by typing “& gt ;” [without the quotes and spaces]. Steve, I have one little request: please do not snip whole posts as that screws up the numbering. Please leave the post in place, but just snip within the post [all of it if need be].
#22,
Steve, I don’t particularly enjoy the threads where people start discussing getting lawyers involved, and that’s not my stand on most issues. But in this case, I think you need to seriously consider having a lawyer send a letter to Tamino demanding that the defamatory information be removed from his blog. It’s one thing to have disagreements with people, but the comments as stated are defamatory and can damage your brand. Sometimes it’s better to let sleeping dogs lie, but in this case, to let unfounded claims lay around, where they will be indexed by Google and live forever, I think deserves more of a response than just a rebuttal comment in your own blog.
Just my .02
#22 I’ve been snipped several times at CA, never, ever unfairly. Each time I deserved it. The irony is that the cause was invariably some bone-headed commentary by some half-witted troll that I should have left alone. Excessively trollish discourse often gets snipped because it’s OT, unproductive, and more annoying than entertaining. It’s a public service.
Getting quantitative – I wonder what be the % volume of snippage at CA? 0.001%. Probably lower than RC. (If each blog posted a quantitative censorship index then ridiculous distortions such as Lee’s could be placed in proper context.)
For my selfish interests, I feel Steve M should conduct the processing of information into and out of his blog as he sees fit and not let the whining of self-proclaimed victims or protocol police alter his basic instincts about the matter. I cannot think of one post of mine that was deleted that did not fit the criterion for a post of delete quality.
I thoroughly enjoy the analyses of papers and comments that are done at this blog and the claims of those who come here to comment on protocol and getting their feelings hurt take too much away from those analyses. While posters such as TCO and Lee can add to the knowledge base here, their diversions into the minutia of manners, for my purposes, at some point outweighs the positive part of their postings. If Steve’s snipping policy led to Lee and/or TCO no longer posting here, I cannot honestly say I miss their presence. Their preaching was certainly overwhelming any of their potential teaching.
I get deleted when I say something stupid, irrelevant, or stupidly irrelevant. It happens far too often, but I can assure everyone the fault does not lie with Mr. McIntyre.
I too am a proud victim of the Zamboni machine.
Multiple times!
28 (JohnM): a few words just got snipped from one of my posts so I too feel qualified to weigh in on this weighty subject
As a retired lawyer I know what occurs when a lawyer provides false information or misrepresents the meaning and scope of law cases cited in brief. The conduct is unethical. If it occurs once the judge or judges involved in the matter will reprimand the culprit.
If the culprit habitually misleads courts in his proof and arguments, a complaint can be filed with the appropriate bar association. The person is liable to official reprimand, suspension or disbarment depending upon the gravity and duration of the unethical practices.
[Steve: snip – please don’t go a bridge too far in terms of imputing motives ]
Steve: [snip – I’ve snipped some offending parts of the previous comment as it uses words that I ask people not to use. I’ve snipped this reply not because I have a problem with the reply, but I’d rather use blog policy to snip the prior post.]
David – ” I have suggested it elsewhere but I will repeat that I think the IPCC should ask the International Standards Organisation – who are less than a block away overlooking the lake in Geneva – to provide standards for peer review and archiving, so as to end the puerile argument that “my peer review is better than yours”….”
Umm… Have you seen the latest ISO debacle? Microsoft has just bought agreement from them that WORD will be counted as an international standard, even though it does not conform to any of the usual standards requirements, and will be a standard that only Microsoft can sell you. Money talks in these circles just as loudly as it does in ‘Climate Science’.
“Flaccid peer reviewing…” I love it! Science, Scientific American, and Nature all need to read Steve’s article.
The general public misunderstands “peer review” because the term is widely misused. Peer review does not prove anything. It only provides a method to disprove [falsify] a hypothesis, such as AGW.
Because the basic purpose of peer review is falsification, strict peer review does not apply to any hypothesis which can not be falsified [e.g., you can’t prove a negative]. And the onus is not on those attempting to falsify a hypothesis; proponents of AGW insist that those questioning its conclusions are the ones who are expected to defend an unstated hypothesis. That is wrong.
[snip]
The primary purpose of peer review is to allow falsification of a hypothesis, if possible. If a hypothesis can withstand peer review without being falsified, the hypothesis is then on its way to becoming an accepted theory. But peer review is no guarantee that a false hypothesis will, in fact, be falsified. It only provides a public forum in which there is an opportunity for peers to falsify a published hypothesis. Refusing to publicly archive the data and methodology of a hypothesis such as AGW directly conflicts with the Scientific Method; how can anyone falsify that hypothesis when the data/methodology is a secret? That is the reason IMO, that Hansen, Mann and others hide their methods and data — hiding the data makes it very difficult to falsify the hypothesis. The same basic motive forces Hansen, Mann, Gore and other proponents of the AGW/CO2 hypothesis to run and hide out from any neutral, moderated debate.
Science, Scientific American, and Nature have learned to game the Scientific Method. Albert Einstein’s revolutionary 1905 manuscript Annus Mirabilis was not peer reviewed. Neither was Watson and Crick’s paper on DNA structure. Conversely, the recent work of Jan Hendrik Schön was peer reviewed — and as a result it was found to be a monumental fraud.
Schön, a former Bell Labs scientist, had authored [or co-authored] one research paper every 8 days in 2001 alone. An astonishing 15 of Schön’s papers were accepted for publication by Nature and Science – two of the most influential journals in the scientific community. But after unrefuted falsification by peers, Schön was proven to have fabricated his results. No doubt Hansen and Mann are well aware of what happened to Schön and Hwang.
Geneticist Hwang Woo-suk was reviewed by the editors of the the AAAS journal Science, after he submitted a paper claiming to have derived lines of stem cells from cloned human embryos. [Hwang also had 25 co-authors! Hmm-m-m]. Science’s referees accepted Hwang’s paper and published it. Later, it was pointed out that Hwang had used a photograph that was the property of another scientist, and his hypothesis began to unravel [eventually Hwang resigned after admitting to fraud].
Moral: Be highly skeptical of unproven hypotheses. For example, the UN’s IPCC presents as fact that a large increase in carbon dioxide will result in a planetary disaster. That hypothesis has been peer reviewed and subsequently falsified. But the pro-AGW media’s agenda demands that a peer reviewed falsification of AGW must not be publicized [or if it is published, it must be buried on page E-37, once, in a 2-inch column, in the lower left corner].
Summary: “peer review” has one central purpose – to allow for the falsification — if possible — of a proposed hypothesis; the Scientific Method collapses without honest peer review, which requires the complete availability of the data and methodology used to arrive at the hypothesis. Remember that, and the agenda of those deliberately hiding their data and methodology from peer review will become apparent.
33 (Smokey): although you say many good things, your basic premise is not correct. Peer-review is not about falsification, but quite the opposite. It is about one or more ‘established’ and ‘respected’ scientist(s) vouching for the methods/results claimed by the author(s). The editor of the journal usually makes the selection of the reviewer(s) based on the editor’s experience or advice from other editors [or sometimes from the author(s) themselves – they are invited to submit contact information for a number of reviewers]. Another method is to select reviewers from authors of papers cited, or from competing groups [the best kind]. this process is not perfect and at times has spectacular failures, but most scientists today accept the process. You are correct that peer-review is a relatively new concept [mostly American from the 1930s]. Before that there were other ways of achieving a similar result: you would send your paper to a famous scientist [e.g. Einstein] and ask for his opinion, or the paper would have been already presented [“read”] at some meeting of a scientific society, and been vetted there. Or you would publish your paper yourself. Crackpots didn’t stand a chance in the tight and small scientific community at the time.
I agree that peer review can never be proof against fraud, hoax or error, but I also accept that for publishers it is a sensible simple and practical quality control step. For authors it may on occasion save them from themselves.
Disclosure, archiving and replication are the key to good science and was the more important part of my suggestion in #6. For the next IPCC assessment there should be a rule that, at least on the three critical chapters, Palaeo, Observations and Attribution, only studies independently certified to be properly archived and capable of replication should be cited. The certification should cover both code and raw data. Independent scrutiny is a fact of life in most industries and it is absurd that climate change science, which potentially is more serious than most industrial products, is not included.
Referring to the rewriting of articles published in Nature.
This from the Wikipedia ( of all places?)
Due to strict limits on the length of articles, in many cases the printed text is actually a summary of the work in question with many details relegated to accompanying supplemental material on the journal’s website.
RE # 15 Pat Frank
Usually I am pretty much in agreement with your comments, but no so much this one –
Surely the reviewer should work without fear or favour. Concealing your name to avoid your feelings hurting the feelings of a friend does not seem the right way to go. It is the correct science, not the friendship, that is important. There are informal workarounds possible.
The “circle of friends” referred to by Dr Wegman as adverse to the review process might have less chance of growth if all names are out in the open.
Heck, you don’t hide your name from your physician, nor he/she from you, yet you can get into some quite personal areas of mind and body. It’s professionalism to accommodate this mutual knowledge, rather than try to conceal it. To conceal is to risk a worsening of the condition. It’s on a different plane to mateship.
p.s. You might recall that I am one of the few people on CA who has always used real names, promoted real names and has given a moderately long c.v. with a few elements of personal philosophy. Has this done more good than harm? I have detected no harm.
Children, children… there is no need to editorialize or vent by using words like “fraud”, which I seldom use and ask others not to use. I’ve deleted a number of posts accordingly.
It’s not that there aren’t important issues. I filed an academic misconduct complaint against Caspar Ammann for, inter alia, failing to disclose adverse verification statistics in their submission to Climatic Change – as Mann had before him – the results of which obviously confirmed some of our key findings. I had lunch with Ammmann and urged him to report these adverse results, but he refused, making it clear that he had no intention whatever of reporting these adverse results.. The complaint was rejected on the basis that, according to UCAR, it was up to the journal to ask for the missing statistics, with Ammann having no responsibility to report them if he weren’t asked. Obviously I think that such a standard is inappropriate, but, if climate science institutions take the view that standard practice in the field does not require disclosure of adverse results, then there’s little point objecting to the behavior of individuals – the issue lies with the practices of journals and institutions. BTW, although UCAR rejected the complaint, the withheld verification statistics were reported in the revised version (These resulted confirmed our statement that the MBH verification r2 value was ~0.) The NAS Panel cited their statement on this (but not ours).
I think that there important differences between “fraud” and “academic misconduct”. People are far too quick to label behavior “fraud” when it may be nothing more than “academic misconduct” – which is serious enough, but which will generally probably fall short of “fraud” as it is defined legally. I see no purpose in people ratcheting up the rhetoric by imprecise and inflammatory use of legal labels.
Enough of such talk for now.
Thank you for your explanation of what happened to the comments.
As for Ammann, what you are saying is that a UCAR scientist can withhold data from a submission to a journal which underminds his hypothesis or conclusion. Furthermore, a UCAR scientist can withhold supplemental information which is cited in his article.
This, to me, are very serious allegations of misconduct. If they are true then I feel I need to go down and pickett UCAR. They are spending our tax dollars and there research is instrumental in shaping extremely important policy decisions about the future of our country. Their research MUST be beyond reproach.
May I get an opinion on the proposition that a specific category be created for “Peer Review”? As we all know, appeals to authority are often invoked under the guise of peer review being infallible, and having a great example stashed away in the W&A Category is tantamount to dropping it in a memory hole in a few months. If specific, credible instances of the more human aspects of PR were cross referenced to their own cat then there would be a ready reference to point to when that argument is used as some sort of trump card. Thanks.
Steve: Category created. I don’t have time right now to catalog past threads, but if there are examples that you want to be so classified, let me know.
Steve,
It looks like the links (internal and external) in your post don’t work (anymore).
Steve: Fixed
I read just about all the top 10 climate oriented blogs, you have do that kind of reading/sampling I find to pull a clear signal from all the noise, in that sense its like temperature proxies, but its tiring to see the discussion wander onto meaningless harping about edited posts and pointless finger pointing on who said what in what context. One can go and read the same discussion at x-box forums with alot less math work if thats what its going to be about.
Why is it just blogs of what I perceive as very intelligent people all taking cheap sideswipes at each other, then each blog master avoids directly posting on other blogs, pretending its a puddle their keyboard simply won’t operate in, operating behind psuedonyms, or having Chiapets do their dirtywork. The only exception is the Peilke’s they march right in where their name has been pasted and make their point. I suppose one could worry about edited posts and what not, so why not have a vote to involve say the 6 top climate bloggers, and have them work out the differences in the math/theories/defintions with no sidechat whatsoever in front of the whole world. It surely would be a whole lot more productive to the world in a enviromental and time sense than to just watch this circular tractionless mudslinging event continue on ad nauseam. The peer review process seems to be broken, so let in unfold in real time with no editing with the chosen. The ultimate fighting of climate if you will..I know Tim Ball has thrown down that gauntlet a few times with no takers on national radio here, if he is such a quack why not take him on?
It seems to be an internet phenomena, watch the core subject of a discussion get braided into a argument about who’s uncle makes the best balloon animals, and what the length on an inch is.
Steve: Great post. I see folks of all persuasions here touting the importance of peer-review. You have highlighted an important shortcoming of the process (at least in some circles). Maybe it’s time the scientific community reviews the peer-review process and determine just what it means and how important it is. Maybe some of the “non-mainstream” articles deserve more attention, in this era…
Oh, and I hear the echos already: Peer-review, for all its faults, is better than other systems. Hmm, maybe so, in normal science, maybe not when we have the Wegman Networks at play. We’ve got an interesting dilemna here, to say the least.
It may be instructive to look at other scientific communities that have largely transcended the problems of poor peer review and SI. One good example is theoretical physics and the highly successful group at CERN (where incidentally HTML was developed to manage their oceans of papers and data). Almost as many theories are produced there as particles, and the vibrant community openly interact to examine each one. High Energy Physics is similar to Climatology in that it often only has a single experimental apparatus available for producing data (the enormously expensive giant accelerator) – however nobody questions that this data is the true test of their theories, and all theory depends on it for validation The data sets are enormous, yet they are made readily available to all researchers.
Thanks much Steve – I came up with a quick few to get the new category up to speed.
http://www.climateaudit.org/?p=732
http://www.climateaudit.org/?p=2068
http://www.climateaudit.org/?p=321
http://www.climateaudit.org/?m=200512
and John A’s contribution http://www.climateaudit.org/?p=572
(direct links omitted due to quantity)
Nice – it gave me an error but accepted it anyway.
Thanks again.
I’d like to add on the topic of fraud in general(hoping not to be snipped), since I researched the subject a couple of years ago. I think that Universities make no difference between “scientific misconduct” and “fraud”. For example, the Swedish definition (taken from Wikipedia) is:
In fact, scientific misconduct has been invented to avoid having to prove an actual fraud (implying, for example, criminal intent, or illegal profit). So instead of having to prove that a researcher has actually intended to publish, say, fraudulent results, it is easier to prove that he/she has not provided evidence for the results when it was required. So, scientific misconduct are actions that can result in fraud. Instead of asking scientists not to fabricate results, you ask them to provide all documentation, and if they don’t, then it’s misconduct.
Keep in mind that publishing a paper with, say, fabricated results, is not a criminal act. However, in one case I know, a researcher was actually prosecuted, and I think even jailed, but on the basis that he got grant money, using invented results. So he actually defrauded the governement, and that is, of course, criminal.
About peer review: also keep in mind that peer review is just a “permission to publish”, based on very specific criteria (that can, however, vary from journal to journal). The criteria usually are: originality, significance, correctness, and of course things like clarity and readability. The “correctness” criterion really only refers to gross errors if, say, an equation is wrong, or some other calculation. On this blog, a lot of “statistics” papers are discussed, where a reviewer can most often reproduce the calculations by him/herself. But many papers report experimental work, and in that case the reviewer can only blindly trust the authors, because experiments are not that easily reproduced. In Shön’s case, for example, no reviewer could have known that the results were fabricated. It was only revealed when others tried to replicate.
If some here think that peer review is not criticized by scientists, think again! The debate on peer review is ongoing and endless.
My experience with peer review is that if you do something mainstream, the review process is fairly robust. If you are the first to use a new technique (wavelets, fractals, Baysian stats) you get reviewers who don’t have a clue but won’t admit it, and you are likely to get rejected. If you stray into controversial waters, watch out, you are going to get unreasonable and even viscious review comments.
As to the question of anonymous reviewing, some scientists are very thin skinned. I have encountered someone who I disagreed with in print 20 years ago and he still held a grudge (and I wasn’t even rude, and it was just part of a review paper). Your career could be damaged if people knew you were the reviewer for their papers. That’s all it takes to get denied tenure or to have a grant turned down.
I think the discussion of peer review is a valuable learning experience and particularly so when it includes insights from those who are actively and currently using the system as a reviewer and as a potential publisher being reviewed. Like any other process it has its limitations of which those people (perhaps more those persons from outside the discipline) using the published information should be fully aware.
I think Steve M by questioning particular applications of the system of peer review adds to our knowledge base and allows us to better judge the outputs of that particular part of the process. In my view of this situation, in order to maintain credibility in this approach Steve M is somewhat obliged to pursue his complaints even when the potential for success is obviously rather limited.
I do admit to cringing when posters take Steve M’s actions and extend the observations beyond what he has intended. I can just see the organizations and individuals at which his criticisms are aimed viewing these “a bridge too far” comments and mixing them with Steve M’s in order to avoid replying to some of those issues head on. I do think something can be learned from the too-frequently used lawyerly replies defending the questioned actions by these organizations, involved individuals and their ardent defenders. That some of these obfuscating actions can be rationalized such that the ruling groups can be excused for doing nothing is not so much the point in my view as is the point why is this group allowing the obfuscating actions to continue?
#37, Goeff, I’ve used my real name, first and last, from the start here; for several years now. My point wasn’t about my own feelings, but rather worries about potentially unsettling responses of a collaborator or colleague to a critical review I might write. Ego fragility does not spare the intelligent, or even the brilliant. Some academics I know do not take well to criticism. I’d rather not have some review I write interfere with my professional relationship with such people.
Craig makes a good point, along similar lines, in #50.
You and I are, of course, above such personal pettiness, :-), but some otherwise intelligent scientsts are not. Anonymous peer review avoids all these social problems.
I don’t disagree with your conclusion here, Pat. However, for every social problem that anonymous review avoids, it creates another. For one, it is much easier to hide an agenda behind an anonymous review than a public one.
But as mentioned earlier, whether benefits outweigh the risks will be an endless debate.
Having been through peer review a number of times, the thing that bothered me is not that the reviewers were anonymous. It was that I wasn’t.
If somebody with a well known name, or somebody who is personally known to the reviewers, writes a paper, he/she will get treated very differently than I do. This sucks, to put it mildly.
My own strong feeling is that, regardless of the anonymity of the referees themselves, that all papers should be presented to referees without the authors name attached. That’s the only way we stand a chance of getting out of the peer review mess we are in now. If James Hansen writes a paper, it seems like every reviewer justs nods their head sagely and passes it on through.
w.
54 (Willis): one knows the lion by its claw.
Willis,
The only problem with that is that they all collaborate on each others work, even providing “informal reviews”, per Judith Curry. I don’t believe any of them would not know if Hansen, or a member of the team was publishing a paper, and would not recognize it on site. This is one of the biggest problems going. There is no independant work being done. If you are with them, you are marked for destruction. Hard to find work in the paleoclimate industry if you are not in the public sector.
Also, I am waiting to see Ammann’s response as promised around April 8, although I am sure I know what it will say already.
Oops – NOT with them.
Pat: “I prefer to keep my name off reviews to remove the personal from the professional. Even scientists have feelings, and even irrational feelings often trump good sense. ”
I’m not saying there’s not valid reasons for anonymous reviews, I’m saying that when circumstances don’t dictate in some way anonymity, choosing to be anonymous when it’s not required can be okay also. However, when you leave your name off and there’s no reason to not use it, can be a show of wimpiness. 🙂
I look at it this way; if you publish, you should look forward to others trying (and take pride in them failing) to falsify or find errors or whatever, to make it a better paper. If your work can’t stand up to scrutiny, what’s the point? Tearing a paper apart should get you thanks, not anger. Isn’t that was it’s review is for, its purpose after all?
Anyway, cowering in fear about commenting on somebody’s work because their feelings would be hurt, well, I don’t know….
FYI: Different journals have different policies. Personally, I prefer the blinded process. That way, as a reviewer, the potential bias associated with knowing the individuals who submitted the manuscript is removed. As someone who is reviewing manuscripts, I have a reasonable expectation that the reviewer will focus on what’s written rather than who submitted it.
Curiously, this process does NOT apply to NIH grant submissions. For NIH grants, WHO submits the grant application has a major (acknowledged) influence on the likelihood for getting funding.
Bruce
Sorry, meant to say, “As someone who is submitting manuscripts for review, I have a reasonable expectation that the reviewer will focus on what’s written rather than who submitted it.”
Bruce
#58 Sam,
As a reviewer, you don’t have to find errors to reject a paper. The “significance” criterion is a very powerful tool. It means you think there are no errors, but that the results are of no interest, and therefore do not “deserve” publication. It’s a totally sujective criterion, and different reviewers will often have a different opinion. For all the papers I’ve reviewed, I don’t recall rejecting any because I had found blatant errors in them. But I’ve rejected tons of papers that I thought were not “significant”. That’s mostly because of the pressure to publish. People try to publish anything, so the system is swarmed with insignificant papers. Lack of originality is also endemic. People re-do stuff that has been published 5-10 years before, because they’re too lazy to do a literature search. There are just too many scientists!… and not enough really good ones, I guess. So peer review is also a filter that restricts how many insignificant papers are published. Of course, very good papers are lost in the process… but not that many. If you’re persistent, you can publish anything. Just pick a different journal and try again.
The ethanol solution to AGW has turned into a disaster with food shortages and food riots occurring. Ethanol substitution for oil was once a favored solution of environmentalists but their hopes have been quashed by reality. One can easily see the relationship between the ethanol disaster and the poor quality of peer reviewing in the sciences. A theory and a solution based on it were widely implemented without adequate analysis. This blog has shown that much of climate science research is of the same sort. If the science is not subjected to adequate review and auditing then disaster can result as evidenced by the ethanol fiasco.
So when some say that AGW remedies are things that would be good to do anyway, then one can easily reply that the same was said about ethanol. AGW is something that is potentially so serious that it should be taken out of the hands of academic scientists and given to people who know how to handle large projects with potentially serious consequences.
re 59
With the blinded review process, one can usually know who or at least which group has written a paper by examining the paper’s references. I know of one person who was so in favor of the blinded process that he omitted references to his own work to prevent this give-away. The reviewers recommended rejection because teh paper failed to reference this well-known work.
I have a friend, a chemist, who when he reviews work if he thinks the work is questionable he goes to the lab to duplicate it. Like Steve M without a blog. Sadly, few do any such thing. It is not entirely a lack of diligence, however. I think a big problem is people reviewing stuff they don’t understand, when they should really send it back and admit it outside their expertise. In these cases they will either pass through c**p or reject valid work, merely because of ego. If the work sounds scientifical and gets the “right” answer, easy pass.
Francois: What is significant or not? What about those publications hungry for anything that either supports the views of the editors or due to a lack of submissions? Why do significant, correct and interesting papers to Nature or Science end up being rejected (I think we may know the answer to that one…) I agree, peer-review can be a gatekeeper in a flood of unoriginal or triffling subjects. On the other hand, it could just be the coked-out bouncer only letting friends or contacts in.
Stan: People don’t grow corn to provide to the ethanol producers. The ethanol producers buy corn at the lowest price they can, depending upon supply, which is highly variable according to price, year to year demand estimates and Mother Nature herself. There is no such thing as ethanol being able to control commodity prices. I think a lot of cattle would agree with me. 🙂
And you’re correct; when you’re familiar with somebody else’s work, it can be easy to know who is writing it from content, if not a knowledge of who’s working on what.
Craig: I think it’s obvious that there is a large social aspect in any field of study with few who are known or thought to be experts in the field. I would guess so at least. And others have said as much.
http://newsbusters.org/blogs/noel-sheppard/2008/04/14/nobel-prize-winning-peacekeeper-asks-un-admit-climate-change-errors
#64 Craig, it’s not always possible to replicate an experiment. In fact, it is rarely possible. Experimental setups can take months to tune and tweak, and quite often involve a large part of homemade gizmos, and rely on intimate, and tacit experimenter knowledge. I’m talking about more experimental science like I was doing (lasers, fiber optics, etc.). Jon Hendrick Shön’s setup, for example, was unique, the kind that only big labs like Bell can have. But even in climate science, all the field work cannot be replicated, or not easily (I’m not talking about drilling a couple of cores here, but stuff like expeditions to Greenland). All the ocean stuff I’ve been reading about these last weeks involves going out on a ship and collecting phytoplankton for days, and doing all sorts of complex and tedious measurements on it. I’m saying this here for the lay audience, whose perception of scientific work may be biased by what they read on this blog: easy and straightforward statistical analysis. I’ve always thought that theoretical physicists had it a bit too easy, especially now with cocmputers and all, they can churn out results by the truckload, whereas an experimentalist must constantly battle with stupid material constraints, and has to always push the limits of what is technically possible.
#65 Sam, Nature and Science are different beasts from most other journals. I never published there, nor did anyone I know in my field, even the top researchers. You publish there if you want to make a big show and attract attention, say from the media. It helps getting big funding, but of course the entry ticket is expensive. It’s all a big game. The whole publication system is what structures the scientific community, sociologically speaking. In theory, all scientists are “equal”, but in practice, there is a hierarchy like in any other field, and it’s the publication system that makes that hierarchy possible. You can be an Editor, and associate Editor, a Reviewer. Sitting on all these committees gives you a lot of power on your colleagues. Power, of course, can be used benevolently, or not. So, in the end, peer review also has that sociological function. That’s why it’s not likely to disappear.
I’m not a scientist or academic so I have no direct experience of publishing in scientific journals. But I offer up the following quote-
“It’s no surprise that blog posts are vulnerable to substandard data analysis.But what most non scientists are not aware of, is the journal publications are too. The quality of work is certainly a lot better than blogs, but it still amazes me what sometimes makes it past peer review into the literature”
The source, Tamino, in a response April 2nd 2008 on ‘How not to Analyze Data – Part Deux’ Of course he was disparaging a paper by Scafetta and West, but what’s sauce for the goose is…..
#68 What’s revealing is that if “Tamino” really thought there were mistakes in that paper, he could write a comment to the journal, and have it peer-reviewed…
WRT Peer review.
part of the catechism of climate science is that peer review is necessary but not sufficient.
In fact, it is neither necessary nor sufficient. The problem with saying that peer review is necessary is quickly understood if we pose a stupid question. Necessary for what? Necessary for truth? If 2+2=4, fell in the woods and nobody was there to peer review it, would it be any less true?
Peer review is neither necessary nor sufficient to preserve the truth function.
It does however keep some junk mail out of your box.
hehe I like that. Peer review is a kinda spam filter.
#54 A growing number of academic and non-academic scientists agree.
#54
If James Hansen writes a paper, it seems like every reviewer justs nods their head sagely and passes it on through.
Well, yes and no. Anonymous peer review allows a less well known researcher to criticize an established one without fear of retaliation. But then it’s the Editor’s job to pick the right reviewers. My first ever review, fresh out of my Ph.D., was of a paper by very well known MIT guys, who did refer to one of my papers, but strangely (or not), not the one that had the key idea on which they had based their own work, presumably to get more credit… the funny thing is that (1) I was asked to review, and (2) I had met the author a couple of months before, and had told him about my Ph.D. work, and even handed him a reprint of my paper! He could hardly plead ignorance!
So well known researchers in top labs do not always get a free pass. The system isn’t all that bad, you know. But as I said, a lot depends on whether the Editor does a good job or not.
But then, in another instance, I had a paper stalled for months by a stubborn reviewer, and only found out a couple of years ago that he was at that time applying for a patent, and didn’t want that paper published, as it could have been interpreted as prior art… Of course I cannot be sure it was that person, but you often suspect who the reviewer is by the way the comments are written. Research fields are quite narrrow, so we always get to review each other’s papers.
Although the ‘submitted’ date wouldn’t be changed by a reviewing delay so priority wouldn’t be affected.
Having reviewed many papers and acted as a conference organizer/editor I found the review process to be very variable, if I got a very superficial review I’d pass the paper to another reviewer and not use the original one again. Likewise when asked if I could review a paper but didn’t have the time then I’d decline.
Re#67
I agree with the comment about the ability to replicate experiments, in many fields this is impossible (and would severely limit the possible reviewers if required).
#74 It was a bit trickier that that in the case of the patent. It can take a couple of years for a patent to be granted, and so there’s quite a delay before it becomes public. And what I had published was not the patent per se, but some observations that could have contradicted some of the patent claims. Those were people from a big corporate lab, and they have very good patent attorneys over there. They don’t take any chances. And that’s exactly how I became aware of it: years later, as a consultant on a patent infringement suit.
I’d say most editors do a decent job, but then papers are rarely controversial (I did say most of them were insignificant junk…), and even more rarely do they have to deal with politicized issues like AGW. Since it’s mostly a volunteer job, most editors are not necessarily prepared to deal with such issues.
The priority date of a patent in respect to prior art is either the date that it is submitted to the patent office or in the case of the US, Canada and Mexico the date of the invention or 1 year prior to the submission date which ever is less. So the time it takes a patent to be issued (the prosecution) is not relevant to prior art.
Your description seems to be directed towards the utility of the patent. A patented invention has to be both new and useful. If the invention in the patent does not work then it would be very difficult to show that someone else’s product is using it. So the patent would be invalid for indefiniteness (it did not describe an invention so that someone of ordinary skill could reproduce it. It just doesn’t work) and could not be infringed by a working product (It just doesn’t work so a working product could not be using it..)
re 76
Thinking about it some more, a better possibility is the “duty to disclose”. It is required of any inventor to disclose to the patent office any fact that is pertinent to the validity of his/her patent. So if this reviewer had seen a published paper that contradicted some of the observations in his/her patent then he/she would have to disclose these to the patent examiner. Anything else would be considered a fraud on the patent office and a failure to disclose would render the patent invalid. Prior art is not an issue here just the requirement to disclose all facts whether they are adverse or not. These facts could come to light after the submission date.
I’m not a lawyer but this does seem to be a very tricky point. If a reviewer sees pertinent information in a confidential reviewing process, what should he/she do? I know that a paper submitted for review is not a public disclosure. Public disclosure comes on the date of publication or presentation at a conference.
.
Consensi (consensuses?) come and consensi go
Th link above points to a New York Times story in which a famous and influential psychology experiment from the 1950s has been shown to have resulted from mistaken mathematics.
I think that this shows that peer review is not just review prior to publication but continues with the examination of reported research by other researchers. In other words, the auditing performed by Steve McIntyre is an essential aspect of peer review. And the assertion by Peter Webster above that his results will await the process of peer review before publication does not contain the full process of peer review. When a result is published, peer review has only just begun and as the New York Times story reveals even a result presented in textbooks may be invalidated 50 years after it is reported.
Ammann’s webpage http://www.cgd.ucar.edu/ccr/ammann/millennium/MBH_reevaluation.html now says in bold red letters “UNDER REVISION” – something that it did not say when I made this post or when I took a screenshot. It also continues to say “Last modified: Sat Mar 25 13:55:00 MDT 2006”, even though it was modified this week to insert the “UNDER REVISION” banner.
If it’s under revision right now, what does that say about the SI availability as at the time of acceptance and publication?
Re #79
Nothing really until we see what the revisions are, for all we know they’re only going to add some pictures.
#80. Is it your surmise that the SI was available to reviewers when they considered the article? If it was, then where is it?
Have you recieved a response to your two requests for SI, or a reference to Ammann that he said would be in a couple of days? It has been nearly two weeks now.
no, I don’t assume that the so is the subject of the comment either.
Re #83
Should be:
no, I don’t assume that the SI is the subject of the comment either.
my fingers are obviously too fat for texting on my ipod!
Steve, Are you still awaiting Caspar Ammann’s response to your letter sent some days before 11 April, to which he promised an answer ‘in a few days’?
Steve: Yes. I sent him a reminder without acknowledgement.
On December 11, 2008 the online and print issues of Nature issued a retraction for a study which had been published in a January 2006 edition. Was this due to a lack of proper “peer review”?
Several articles on this appeared in western Canada newspapers between Dec 3 and Dec 11.