I’m posting up our July 2005 review of Wahl and Ammann. The recently accepted version is here. I’m posting this up for a variety of reasons. Mann relied heavily on Wahl and Ammann in his NAS panel testimony (which wasn’t even online as accepted last week) and so it’s hard to finish off the discussion of NAS without wading through Wahl and Ammann. Also the associated UCAR Media Advisory that all our claims were “unfounded” had been widely cited.
Comparing W&A against my previous comments in a peer review capacity is an interesting object lesson in what peer review is and isn’t. It’s also interesting to speculate as to whether IPCC considerations may have affected how it’s been handled.
Differences between Versions
First, the information about verification r2 statistics and the various arguments for r2 versus RE, referred to a couple of posts ago, was not in the original submission. The following is new in W&A Revised: Section 2.3 entitled “Calibration and Verification Statistics — Validation; the first few paragraphs of Section 5. Summary and all four appendices, plus Tables 1S, 2S and Figure S1. Some sentences have been changed here and there, but otherwise the rest of the text looks pretty much word-for-word what we saw last June.
I suspect (but do not know) that these sections have been added since December 14, 2005. When I met Ammann in San Francisco, he indicated that he’d not had to make any significant changes. We discussed the disclosure of adverse verification r2 results. I strongly urged him to get in front of the issue or else he’d be in an even worse situation. He didn’t agree. Something’s changed his mind since December 14. Maybe it’s my pounding away at climateaudit. Or maybe there were strings attached to the “provisional acceptance” received on Dec. 12 , 2005 (which I was unaware of in San Francisco). It doesn’t really matter. The sections are new.
Obviously, the table of verification statistics completely vindicates our claims on this matter in McIntyre and McKitrick [GRL 2005] and makes the UCAR Media Advisory that all our claims were “unfounded” in need of retraction.
Background to our Climatic Change Review
This was the second article that I’d reviewed for Climatic Change. Stephen Schneider, who is prominent in the climate debate and a critic of ours, had particularly criticized E&E back in 2003 for not allowing Mann et al. to review our 2003 article. Whether that particular criticism was merited, Schneider then had the virtue of consistency and, when MBH submitted a supposed rebuttal of MM03 to Climatic Change, I was invited to be a reviewer. I’ve found Schneider to be an engaging correspondent, although I’m not very happy about what he’s done here.
Obviously, I approach things from a different angle. In my capacity as a reviewer of the MBH 2004 submission, I asked for the residual series, cross-validation statistics and source code that Mann had refused to provide to me as an opponent in controversy.
This provoked some soul searching at Climatic Change. Schneider said that no one had asked for such things in 28 years of his editing the magazine. He was worried that, if reviewers were expected to examine source code, no one would ever review for his journal. I said that I didn’t expect everyone to examine source code, but that I wanted to and needed to.
Their editorial board met and decided that they would not even ask Mann for source code, but agreed that the journal would have a policy requiring authors to provide supporting data.
OK, I asked for supporting data such as cross-validation statistics and residual series under the newly established data policy. Mann refused. I sent in a review that Mann had flouted the newly established data policy and therefore the article could not be accepted. I never heard anything more about this article and it never appeared.
Of course, by this time, Jones and Mann [2004] had cited the article (in review) as trashing us and did not withdraw the comment in Jones and Mann [2004] when the underlying article was rejected. Jones and Mann [2004] in turn has been quoted on a number of occasions.
On May 12, 2005, I was asked to review the Ammann and Wahl submission. Schneider knew what to expect from me. Equally, I had been given a surprisingly fair shake by Schneider in 2004. We had much correspondence while Clim Chg was deciding on a data policy, much of which was amiable fencing. You run into some skilled defenders of indefensible positions (Karl Ziemelis of Nature is like that), whose casuistry has a certain flair that appeals to me.
The other part of the background pertains to the IPCC 4AR process. While realclimate had issued several denunciations of us and these denunciations assuaged most climate scientists, nothing of this type from the Hockey Team had appeared in the peer reviewed literature (the prior MBH submission not making into print).
So from an IPCC 4AR point of view, as Luboà’¦à⟠Motl has pointed out, we were still “state of the art” — hardly where IPCC WG1 wanted things. While they may have said that they had “moved on”, it would be hard for IPCC 4AR to say nothing about MBH98 and simply pretend that it had never happened.
By this time, the deadline for IPCC 4AR to be in press was looming up — December 2005 was the cut-off. It’s not far-fetched to think that IPCC WG1 was pretty anxious that Ammann and Wahl make the cut-off for inclusion in IPCC 4AR.
For some time before this, Ammann had said that they had “exactly” replicated MBH98 and, on May 11, 2005, the date of one of my rare public appearances, UCAR issued a national Media Advisory saying that Ammann and Wahl had shown that all our claims were “unfounded”.
This “Media Advisory” was quickly inhaled into IPCC processes. Sir John Houghton cited it to a Senate Committee to show that we had been refuted; Mann mentioned it about 6 times in his letter to the Barton Committee; even the European Geophysical Union cited this to the Barton Committee. Although the article was unpublished, Ammann and Wahl sent it to IPCC 4AR (described there as Wahl and Ammann 2004, J. Clim) in the same version as we were asked to review. IPCC 4AR in turn relied on Wahl and Ammann to launder MBH98.
Our Review
A bunch of things strike me on re-reading our review from last June.
First, I had been able to reconcile our code to W&A code almost immediately (I reported in real time on climateaudit), so I knew that they had got the same adverse verification results as we had obtained. As far as I was concerned, it was completely unacceptable both for Ammann and for UCAR to issue a press release that all our claims were “unfounded” knowing that our claims about verification statistics were true. Accordingly, in my capacity as a reviewer, I requested this information.
I’ve posted up the correspondence to prove that the information was requested and then refused. In our review, I made very severe criticisms of this refusal. At a minimum, it was a flouting of Climatic Change data policies that had been established during a similar process with Mann. But it also evidenced that the authors knew that the results were bad and were withholding the data. Since the revised version responded to these criticisms, obviously my criticisms were not taken as being unwarranted. I don’t think that the arguments in the new sections are any good, but that’s a story for a different day.
As far as IPCC 4AR goes, there’s a huge difference between the revised version and the unpublished version sent to IPCC 4AR — the revised version, however unwillingly, vindicates our claims that MBH98 verification r2 and other statistics fail and has longwinded excuses that look like special pleading. There was not a whiff of that available to IPCC 4AR.
Second, I objected strenuously to the many misrepresentations, especially concerning “our” reconstruction. Here nothing has changed in the Revised W&A. I made point after point in our review about misrepresentations. Notably, we categorically denied that we had ever “presented” an alternative reconstruction, showing explicit statements as far back as the time of MM03. Even realclimate has recognized this. Nonetheless, they spend pages arguing over a straw man, arguing that “MM” reconstructions (which are just demonstrations of MBH98 without bristlecones or with centered PCs or such) fail a verification RE statistic. For us the salient point was that MBH98 failed verification r2 (as did the non-bristlecone variation). We were merely illustrating non-robustness; statistical verification of a variation merely illustrating non-robustness is pointless (a position also implicitly taken in BàÆà⻲ger and Cubasch). We objected vociferously to the misrepresentations. I further objected to them when I met with Ammann in San Francisco. But hardly a comma has changed in response.
Third, their Response Letter of June 10, 2005, the one in which they refused to disclose the adverse verification r2 statistic, concluded as follows:
Concerning whether RE is “at issue” The requester mentions that the RE statistic is at issue, a claim that Dr. Ammann and I have shown is made moot by the results of our indirect tests in ms #3321. In addition, Dr. Ammann and I have shown in other material referenced in mss. #3321 that the analysis of McIntrye and McKitrick in GRL (2005)–which claims RE significance levels are improperly determined by Mann, Bradley, Hughes–is itself deeply flawed. Thus, the argument in the request is incorrectly put in this regard, and it also ignores that we do use an entirely separate statistic–the deviation from verification period mean.
The “other material [by Ammann and Wahl] referenced in mss #3221” was their concurrent submission to GRL. We pointed out that we had been informed on June 6, 2005 that this article had been rejected by GRL and that, in any case, their GRL submission contained no discussion of RE significance levels, much less a demonstration that our results were “deeply flawed”. As you will see, I called a spade a spade in the review.
Aftermath
In correspondence in June 2005, Schneider mentioned that sometimes an impasse arose between reviewers and authors, in which case, Climatic Change might issue a “springboard editorial”. While our review was severe, it was, for the most part, expressed in objective terms and in copious detail. Although this presumably created an impasse, I heard no more from Climatic Change. Later I noticed that W&A was “provisionally accepted” in December 2005 and recently we heard that it was finally accepted. While some comments were implemented in the revision, most comments were simply ignored.
Someone has pointed out that Wahl and Ammann was “provisionally accepted” by Climatic Change about 1 day before the IPCC deadline expired — was that a coincidence? Seems unlikely to me, but you never know. An interesting issue arises from the big differences between the version accepted by Climatic Change and the version on file at IPCC WG1. IPCC WG1 reviewers would be unaware that MBH98 failed verification statistics, as we had alleged. Did the W&A version as it existed in December 2005 meet IPCC policies for being finalized? An interesting question.
I’m as aware as the next person that there’s a controversial aspect to all of this. But, for example, our relations with non-Hockey Team authors, Huybers and von Storch and Zorita are completely cordial, even though we disagree with them. I had pleasant dinners in San Francisco with Huybers and with Zorita; von Storch congratulated me when we met in Washington for what we were doing. I’d rather have comments than no comments. It’s the sludge that bugs me.
So when we come to responding to Ammann and Wahl [2006], we have to still wade
through all the misrepresentations contained in this “peer reviewed” article. The r2-RE arguments were not there before, are a form of response to our review and I’ll discuss them separately.
As to the rest, while we have a controversial interest, I think that every point about mischaracterization and misrepresentation is objectively verifiable and could have and should have been sorted out last summer in editing rather than simply adding to more controversy. (In my opinion, one of the reasons that Saiers rejected their GRL submission was a similar difficulty in wading through misrepresentations, every one of which we were contesting.)
Why weren’t these matters ironed out? Did Climatic Change make any attempt whatever to resolve these issues? What were those attempts? It looks to me like they did nothing. I really can’t think of a whole lot to add in respect to the rest of the article to what we said in our review of July 2005. I wish that at least a little of it had been dealt with back then.
The new material about r2 and RE, probably added since December after the “provisional acceptance” , looks very weak to me. There are no third-party statistical sources. Their only statistical reference is Wahl [2004] and Lytle and Wahl [2005], articles on pollen, not time series – hardly imposing foundations to build their theory of verification statistics. in our review, we listed many statistical references that needed to be considered. None of them are referred to. Who reviewed this stuff? Did anyone review it? (Look I’m cynical – the MBH Corrigendum was never peer reviewed.)
Wahl and Ammann [2006] should have been cleaned up last summer to extract whatever points did not apply to straw men. I gave some concrete suggestions on how this could be done. So what do Climatic Change peer reviewers do?
Climate Audit 
37 Comments
I would hope that the IPCC would not report any conclusions from an unpublished (provisionally accepted) work. They would be open to criticism if a conclusion was a controversial one that was based on details excluded from the final report. I’d like to think that they were using the “provisionally accepted” paper as a placeholder so that they could eventually quote the final (published) paper.
Even though AW has replicated MBH98 mathematically, neither paper represents anything physically because of the use of bristlecones, the use of stationary PCs, and other reasons that you have stated. They are worthless in telling anyone anything about past climate.
Steve, having read your earlier (July 2005) review letter, what is the next step wrt to the publication of W&A in Climate Change. Going by the July review and the changes made since, W&A still contains serious misrepresentations and mischaracterisations. Do you get the opportunity to publish a reply in CC ? I would imagine you must be getting extremely irritated with the constant misrepresentation applied to your work, and by the failure of what I guess can be called the “Hockey Team” to actually address criticisms. Surely if r2 (or R2 if you like) is essentially always applied in such cases, someone will notice ?
“Who reviewed this stuff? Did anyone review it? (Look I’m cynical – the MBH Corrigendum was never peer reviewed.)
“Ammann and Wahl [2006] should have been cleaned up last summer to extract whatever points did not apply to straw men. I gave some concrete suggestions on how this could be done. So what do Climatic Change peer reviewers do?”
Given the content of your description, Steve, it wasn’t a problem with peer review, but with Schneider as editor. No process can survive what amounts to deliberate subversion.
Again given your description, Schneider failed abysmally in his job to have let AW through unchanged in light of your criticisms of the substance and of the misrepresentations.
I can’t imagine that Schneider didn’t understand the substance of what was in contest, and so it wasn’t just honest ineptitude. It’s likely the other one or two reviewer(s) weren’t expert in statistical methods, and so applied themselves to other aspects of the paper. If that were true, Schneider would have had even less reason to put aside your criticisms.
It’s not peer review that has the blot, Steve. After all, you were a reviewer and you did your job (likely, as do most peer reviewers). In this case it’s Schneider himself.
Don’t be so hard on the editor of Climatic Change. The editor is trying to allow everyone to have their say. He is assuming that people are generally honest, and that what underlies the dispute between Steve M and A&W is how they interpret things. This is standard in most of academia.
There is a serious problem here, but the problem is not with Schneider. The problem is systemic: there needs to be greater disincentive to dishonesty in research.
All this poses a very interesting problem for the NAS Panel on Temperature Reconstruction. If they come out in support of the Hockey Team, they will have to provide detailed argument that counters the MM view of things – not an easy task I would have thought.
Surely the smart thing for them to say is that their investigations show that practices by the Hockey Team and their publishers leave quite a bit to be desired, and put out a demand that climate scientists lift their game.
If they don’t do this, the crescendo will just get louder and louder until they finally accept the reality that what is at stake here is good science.
I think Schneider’s behavior is reprehensible but far too commonplace.
My own attitude from outside academia is that if I had a paper which had large sections of statistical analysis, I’d want someone like Steve to make sure I’d done my sums correctly and not made glaring errors. The last thing I’d want is to end up being dissected on a blog somewhere or in a reply to the journal in which I’m forced to defend poorly done work.
But of course I’m naive enough to have such an attitude when my career advancement does not depend upon journal publication or thoroughness.
I cannot understand why Ammann or Wahl put their professional reputations at risk with such poorly done work unless they are animated to defend personal or political beliefs.
What are peer reviewers for? To rubberstamp the scientific worldview of the editorial board. But it appears to me that they act like partisans rather than expert reviewers, allowing poorly done work whose conclusions support their beliefs and blocking or blackballing those who opposed them.
Most editors will openly state that they do not view judging dishonesty as part of their job. That’s the way the current system works (not just in climatology). Don’t blame Schneider.
As for “I cannot understand why Ammann or Wahl put their professional reputations at risk with such poorly done work”—the premise is wrong, unfortunately. A&W are not putting their reputations at risk. Most likely, they will never admit what they did. The most that will happen is A&W will be shown to be incorrect due to an “honest but incorrect interpretation” or some such. That’s the point: there are rarely sanctions for dishonesty, even when blatant.
What’s worse is that the system of scientific publication is almost certainly not going to reform itself. When the NAS can just ignore a request from the House Science Committee on replicability and data availability, the seriousness of the problem is manifest. And look at some of the things the current Editor-in-Chief of Science does: e.g. refusing to uphold his journal’s stated policies and blaming Katrina on global warming.
Reform, if it comes, is going to have to come from outside. And establishment scientists will fight it with everything they can, because they are fighting to maintain their own self-image.
#2. The AW and MM replications are now identical to 8 9’s, using apples and apples. AW knowingly failed to acknowlede this. AW do not replicate confidence interval calculations. Both versions can replicate the spliced MBH hockey stick as archived in broad strokes, but intresting and releveant details especiallly in the 15th century remain still to be reconciled. Does final reconciliation “matter”? At this stage, I’d like to see the final reconciliation. Also sometimes ironing out little discrepancies is important to understanding what they are doing.
Re: # 8
Most competent science journals should employ competent scientists to be their content editors, rather than emplying competent journalists for the job…a competent journalist gets “the story” and “sells copy”.
#4. Pat, I think that the issue here is two-fold. I’ll bet that the other reviewers did virtually nothing. I’m finding it pretty hard to see any changes made for other reviewers. Second, I can’t see anything incorrect in my review comments when I look at them 9 months later. It looks like Schneider made them respond to a few of them, but not all of them.
Were they studies or were they Isle of Man corporations? Selective disclosure ….. it’s so 9/10.
RE: #2. Well it is *possible* that they might be able to tell us something vague about long term availability (or lack thereof) of a certain type of snow pack at a certain critical annual time window, and / or, about whether or not the Southwestern US Monsoon started early and penetrated well north more often than not, or it didn’t, during past segments of time. But I’d agree that they tell us essentially nothing about past global temperature regimes.
Steve
Your review of AW is hard-hitting, clear, direct and fair. My guess as to what happened is what you suggest, that Schneider said to them that they did not have to answer all of your demands (of course, they could not have done so, because there is no “there” there), but that the final paper had to report the verification R2.
I bet that Amman even knew that when you were talking to him in SF, because you say he already knew that the paper was “provisionally” accepted. He would not have wanted to say, Yes, we are going to report R2 in the final version, because that would have led to your asking more.
#5,#8 The professional duty of a science journal editor is *not* to let everyone have their say. The professional duty of a science editor is to ensure that everything appearing in the pages of the journal meets the standard of science. If a reviewer has a persistent problem with the objective content of a submission, it is the duty of the editor to resolve that problem in terms of the objective criteria. Either the reviewer is wrong, or the author (or both). If the editor is not competent to judge and the argument is not resolved by dialogue, the usual expedient is to bring in another reviewer who is expert in the point of contention.
Schneider failed in his professional duty. Given the version of events rendered by Steve’s posted correspondence, I see no ambiguity on that judgment of failure.
#7 John, the object of peer review is not, “To rubberstamp the scientific worldview of the editorial board.” The process in the rest of science has not been corrupted by advocist politics. It’s true there is some arrogance in some journals — some math and physics journals especially. But in general the object of peer review is to vet the science. It generally does a very good job. As evidence I offer the technological trajectory of modern society. That technology would not be possible without a solid substratum of valid science. Anyone working in science is aware of continual and real advances. They would not be possible without proper peer review of pre-published results.
Here’s a big secret about the physical world. It doesn’t care squat about politics or biases. Politically motivated, world-view-imposing editorial boards would quickly stumble and fall as their field moved away from fact, and further and further into some imposed fantasy. The fact that hasn’t happened with 99% of science, including medicine, is that the large majority of peer reviewers and editors do their jobs with integrity.
Here’s a bit of advice: Keep your larger perspective even while gazing on the mess of climatology. No matter that one is mucking a cistern, clean fields and streams are still the larger reality. Sanity demands one keep a grasp on that truth.
“As evidence I offer the technological trajectory of modern society. That technology would not be possible without a solid substratum of valid science.”
Just a minor arguement on that point. Technnology is a different matter, sureley it is based in science, but it offers tangible, unarguable results. When The labs at AT&T developed the transistor, technology progressed. But there was little argument equal to 1000 year paeleoclimate reconstruction. That cannot be checked. When you make a transistor you can show, without doubt, that it is doing what you say it does.
Climate science is not capable of doing that. A. We cannot go back to 1400 AD and set up a myriad of weather stations and develop a Global Mean temprature. NOr can we go back and determine if the MWP was localized or global. B. we cannot verify the output of the models without sitting around for 50 or a hundred years and seeing if their predictions are valid.
Yes science in general is safe, but that is because most of the science falls within the real of provable here and no (to a greater or lesser degree). Sure there was load of controversy and discusion over quantum theories in the early part of the centruy, but eventually experiments were designed to prove/disprove these things. Once done the math was re-examined and errors corrected, as it should be. With climate science this is much more difficult. And that would be fine if it weren’t for the policy ramifications.
Hear, hear.
Errors can be forgiven; covering up of errors cannot. Schneider now has a relatively short period of time in which to rectify his error.
Re #16, ET SidViscous
What worries me is the number of areas where a well-financed campaign of totally bogus statistics can be used to change policy. The anti-smokers are managing to get smoking banned throughout much of Europe via this route (relative risks of 1.13 = 13% more likely to get lung cancer from 2y smoking, and suchlike rubbish). Smokers are generally a minority: if the AGW crowd win, they will be able to harm practically everyone.
What will be the next target ?
Not to take the discussion down that route. But we already know what’s next. Food.
It’s already been started, there have even been proposals to limit how often one can eat at a fast food restaurants, seems absurd now, but it won’t be in 20 years.
Re #15 and #17: Is this the same Stephen Schneider who made this famous quote (google took me to http://johnquiggin.com/index.php?p=1861 who claims this is the full quote):
“On the one hand, as scientists we are ethically bound to the scientific method, in effect promising to tell the truth, the whole truth, and nothing but – which means that we must include all the doubts, the caveats, the ifs, ands and buts. On the other hand, we are not just scientists but human beings as well. And like most people, we’d like to see the world a better place, which in this context translates into our working to reduce the risk of potentially disastrous climate change. To do that, we need to get some broad-based support, to capture the public’s imagination. That, of course, entails getting loads of media coverage. So we have to offer up scary scenarios, make simplified, dramatic statements, and make little mention of any doubts we might have. This “double ethical bind’ we frequently find ourselves in cannot be solved by any formula. Each of us has to decide what the right balance is between being effective and being honest.I hope that means being both).”
#20 That’s our boy.
I don’t know how much harm would be done if the warmers convince all the policy makers that AGW is real and is problematical. Nations will not cut their economic throats to make the kind of cuts in CO2 emissions that would be necessary to do any good. There will just be a lot more grandstanding, like what is going on in Canada and Europe today. Lot’s of talk, but no real action. And the Third World countries will stand by, laughing all the way to the bank.
jae
You might think differently when you go to fill up at the pump.
23> Yes, you are probably correct. Has this happened yet in those countries that have adopted the Kyoto crap?
The whole AGW thing is very frustrating and hard to approach scientifically, due to such strong unscientific reasons for assuming it’s true (note that I am NOT saying there are no scientific reasons for assuming it’s true, and I am not saying it is untrue). Vast numbers of scientists and organizations depend financially on “proving” the theory, so these entities naturally tend to keep on pushing the idea and strongly resist any skeptics. If the policymakers ever start doubting, there could be a lot of scientists and universities without grants, lots of curtailed travel, lots of defunct committees, etc. I wonder what the UN budget is for all things AGW.
Re #15,17: “The professional duty of a science editor is to ensure that everything appearing in the pages of the journal meets the standard of science”. Sorry, but it’s not like that.
I agree that the requirements of duty should be higher than they presently are. If they were raised to be as high as you indicate, though, the whole system would clog up. It is impractical.
A better way to improve the system would be to do what Steve M has been pushing for: greater transparency, which allows others to readily check the work after publication. This ought to be easy to implement. If this were coupled with a modicum of accountability when bogus research is uncovered (not so easy to implement), the integrity problem would be largely addressed.
Lastly, about sending the manuscript to a third reviewer, do you know that this wasn’t done?
#26, “Sorry, but it’s not like that.” I don’t know where you’ve published, KO, but where I’ve published it is like that.
It is also like that in every specialist journal of which I am aware. Except, it appears, where Stephen Schneider is editor.
It’s rather ludicrous and self-contradictory to suppose that holding authors to the standards of science would “clog up” science (whatever that means).
Science has standards of objectivity that are unique among the human intellectual disciplines. If those standards are breached, one may as well publish poetry and polemics.
Has anyone tried publishing a science journal in the form of a blog? Just wondering.
Schneider has really screwed up, with his “agenda,” and I really believe it will catch up with him. Soon.
Re #27. Medical journals generally have relatively high reviewing standards. So consider the following.
(Source: The Economist, 03 June 2004.) Where I said “clog up”, the BMJ says “grind to a halt”.
I do not know of any journals where the editors actually read the reviews. Editors almost always read only the review conclusions (overall recommendations). Thus the particulars in the review that Steve M wrote for the A&W paper might not have been considered by the editor.
Also, the other reviewer of the A&W paper, presumably, recommended acceptance. We don’t know what happened next. For all we know, a third reviewer was brought in to settle things, and the third reviewer said “accept after inclusion of r2”.
RE#30:
No…but there have been things in the news here and here (just a couple of MSM examples) to show that the studies in medical journals are very problematic–data being made up, poor statistical analysis–the very same types of problems that are in the paleoclimate studies.
We can either have things “grind to a halt” or we can have bogus studies. Sometimes I wonder if grinding to a halt wouldn’t be better.
#30-Specious argument, KO. I wrote that editors should uphold the standards of science, not, as you’d have it, that they themselves should recalculate results.
Any recalculation done is done by reviewers. Editors rely on reviewers to do their jobs, and good editors retain groups of reliable and knowledgeable reviewers.
Your comments about “might” this and “for all we know” that merely reiterate my main point about Schneider. A good editor would have informed his reviewer that a third (or fourth) reviewer had been brought in. The editor would also have allowed the prior reviewer to see the presumably conclusive analyses made by the new reviewer. Doing these things amounts to a professional obligation.
The fact that so much remains mysterious is itself an indication that Schneider failed his editorial responsibilities miserably.
Of course if raw data and source code were made available to reviewers, it would be much easier for them to check the studies more thoroughly. They could at least do a fairly quick check to see whether the method as described in the paper actually matches the method used. Then, if the data and code were also made available to “auditors” post-publication, and auditors had a forum for discussion and publishing problems found in published papers (which does not seem to be currently available in many journals) then at least flawed studies would eventually be brought to the scientific community’s attention.
Is it really so hard to ALLOW people like Steve to check the studies if they volunteer to do so?
#31 seems to suggest that I said medical science journals had high standards. I actually said that the standards of all science journals are unacceptably low (#5,8) and that those of medical journals are relatively high (please look up the definition of “relatively”).
#32 goes further and claims that I said editors themselves are supposed to recalculate statistics.
Both comments are straw men. I don’t believe that people would resort to using straw men if they had reasoned ways to rebut a point. And I didn’t think that this blog was supposed to be a forum for rhetoric.
Re: #34
In the quote you supply, an EDITOR claims that they do not recalculate results. Since there is no context for that quote it is quite reasonable for one to assume that he is talking about himself and other editors. You then link your previous discussion about “clogging up” science with the quoted editor’s statement about “grinding to a halt”. It’s not much of a stretch to assume you are also talking about editors since you don’t further clarify that you are referring to reviewers. Therefore I find your claim that Pat was making a Straw Man argument to be unfounded.
re: 34, 35
Whatever the merits, the problem is that there are too many papers published which are of poor quality.
In stock market terms, it’s a speculative science bubble being offered to the public and to politicians. When the bubble bursts, the promoters (science journals) will find it very hard to compete for the very small number of articles “post-crash” which are of good quality, but don’t offer eyewatering returns of sensational headlines, worrying climate forecasts of temperature rise and high pressure sales tactics for reports with even scarier headlines.
Expectations of returns from a few good articles will then revert to the long term average after a bear market of prolonged understatement and retrenchment.
At the moment, the sky’s the limit for climate modellers and multiproxy authors. Every statistical run is a potential new article to foist onto a public desperate to buy into the next rush of apocalyptic fever, although it has to be said that the number of people willing to buy these stories has reduced recently, causing some commentators to wonder whether a “top” in the bull market has been reached…
To clarify my last two posts, the entire quotation was included. The way the peer-review system works, though, is with editors overseeing the detailed work, which is done by outside reviewers. So I think it is clear that the “we” in the quotation must refer to editors+reviewers, with the recalculations done by reviewers. The alternative interpretation being suggested does not seem sensible.
3 Trackbacks
[…] I was asked to review the Wahl and Amman submission in May 2005 and recently posted up my review here. The first recommendation in my review was that all Wahl and Ammann remove all arguments that […]
[…] […]
[…] […]