Wahl and Ammann 2007 and IPCC Deadlines

In a previous post, I’ve observed some oddities in connection with the dating of Wahl and Ammann 2007 and with Schneider’s obfuscation when asked to explain how an article supposedly accepted on March 1, 2006 could cite an article that had not even been submitted until August 2006. (BTW, I note that Journal of Climate uses the term “In final form” in its timelines, which, applied to the Wahl and Ammann case, would preclude any dates prior to August 2006, the date of submission of Ammann and Wahl 200x, which is cited in Wahl and Ammann 2007, and, more likely, preclude any dates prior to the June 2007 acceptance of Ammann and Wahl 200x – a date which appears to have triggered the actual acceptance of Wahl and Ammann 2007.)

Why would they go to this trouble? Why wouldn’t they just put the acceptance date for Wahl and Ammann 2007 as June 2007, concurrent with the acceptance date of Ammann and Wahl 2007, which unblocked the publication process?

To answer this, we need to look at the IPCC Publication Deadlines, where we shall see a compelling and incontrovertible link between Wahl and Ammann timelines and attempts to at least appear to meet IPCC publication deadlines.

IPCC Working Group 1 Publication Deadlines

Three main milestones were set out in the IPCC Publication Deadline timetable:

May 10-13, 2005 – Literature to be cited will need to be published or available in draft form by this time (the date of the Second Lead Author meeting).
Dec 13-16, 2005 – Literature to be cited will need to be published or in press by this time (the date of the Second Lead Author meeting).
Late Feb 2006 – the TSU must hold final preprint copies of any unpublished papers that are cited in order that these can be made available to reviewers. This means that by late-February 2006 [the date for preparing the Second Draft] if LAs can not assure us that a paper is in press and provide a preprint we will ask them to remove any reference to it.

There are several different versions of the WG1 Publication Deadline Timetable. An early version is here; The most recent version is online here. Also see the IPCC document entitled “Deadlines for literature cited in the Working Group I Fourth Assessment Report”, preserved as of Apr 30, 2006 (and unchanged as at June 21, 2006).

The above deadlines were well publicized and guided the behavior of climate scientists. For a huge study like AR4, for better or worse, there have to be some cut-off dates or the study would never end. For example, Judy Curry has mentioned here her understanding that there was an end-2005 deadline, an understanding that presumably guided her publication schedule.

Wahl and Ammann 2007 Chronology

The first deadline cited above was May 10-13, 2005. On May 10, 2005, Ammann and Wahl submitted Wahl and Ammann 200x to Climatic Change and Ammann and Wahl 200x to GRL -see here – and issued a press release on May 11, 2005 here . The submission date on the published version of Wahl and Ammann 2007 is May 11, 2005.

The second deadline was December 13-16, 2005 and “literature to be cited will need to be published or in press by this time”. The
UCAR website states that Wahl and Ammann had been “Provisionally Accepted” on Dec. 12, 2005 – on the eve of the expiry of the second IPCC deadline. Has anyone ever seen a paper report the date of its “Provisional Acceptance”? Clearly “provisional acceptance”, whatever it is, is not the same thing as being “published or in print”. Wahl and Ammann had clearly missed an important deadline – a deadline that Judith Curry and others were adhering to.

The end of February 2006 was a drop-dead date by which “the TSU must hold final preprint copies of any unpublished papers that are cited in order that these can be made available to reviewers. This means that by late-February 2006 if LAs can not assure us that a paper is in press and provide a preprint we will ask them to remove any reference to it.” The UCAR website states that Wahl and Ammann was supposedly “accepted for publication” on Feb 28, 2006, this third date, like the other two, being on the eve of an IPCC deadline. So there is an incontrovertible pattern.

The timing of the Climatic Change decision was very curious, to say the least. GRL turn times are very short. The GRL decision on the contingent GRL submission would be available in 2 weeks; within the four corners of journal administration, there was no reason for Climatic Change to make a hurried decision ahead of GRL – especially given that Schneider had already been burnt once by Ammann on his GRL submission. Why wouldn’t Schneider just wait a couple of weeks? Or is it possible that UCAR/Ammann, with an eye on IPCC deadlines, put pressure on Schneider to “accept” the article before the end of February? Sure seems likely. And I wonder if Schneider’s acceptance still had some strings attached – something that seems highly likely given the subsequent delays. Ammann and Wahl 2007 was published over a year after any other Chapter 6 journal citation.

Ammann placed Wahl and Ammann 2007 online in early March 2006. It was brought to my attention and I immediately made a number of posts – see here on March 10, 2006 and there are a number of contemporary posts, including ones where I placed my 2005 Review and correspondence online. This new version included a table showing that the MBH verification r2 for the AD1400 step was 0.018 (we had reported 0.02). Despite this confirmation of at least one of our results (and they confirmed many of our results), Wahl and Ammann did not withdraw their allegation that all our claims were “unfounded”.

On March 16, 2006, GRL notified Ammann and myself that the Ammann Comment had been rejected, with Famiglietti stating:

The simple message, with which I concur, is that there is nothing significant that is new here, so that there is no need to publish the present exchange.

Despite this GRL rejection over two years ago on March 16, 2006, the UCAR webpage on this matter states (as of today):

We are working on an update of this site and a cleaned-up version of the code since the Wahl and Ammann paper is accepted by Climatic Change and in press. Once GRL has reached a decision, we will also post the appropriate codes and illustrations here. (Note, the current version is fully functional, and no results will change.)

They were obviously very quick to announce the Climatic Change acceptance, but not so quick to announce the GRL rejection. Securities commissions would make mincemeat of mining promoters that did this.

A version of the webpage was scraped on Feb 5, 2006. The only change between that version and the present version is the insertion of the above paragraph. The last recorded change to the webpage (as of today) was on March 25, 2006. Ammann’s webpage presently states:

Caspar Ammann
Last modified: Mon Mar 25 13:57:32 MDT 2006

Thus, there was one and only one change on this webpage between Feb 5, 2006 and today: the insertion of the above paragraph. A change to the webpage is attested on March 25, 2006. The only plausible conclusion is that the above paragraph was inserted on March 25, 2006. If so, the paragraph was untrue and overtly deceptive at the time it was published on the UCAR website. I would welcome any interpretations of this information (and will post any such interpretations in this paragraph).

Returning to the IPCC schedules. On April 7, 2006, according to the IPCC Schedule, the Second Order Draft was to be made available to reviewers and the 8-week review period for expert and Government reviewers was to start. I downloaded the Wahl and Ammann version then online at the WG1 website for unpublished papers; my download is dated Apr 7, 2006 on my computer, which I take to be the date that I downloaded the pdf. (On Apr 9, a CA reader noted that online that the stale version was still online at WG1). The pdf document properties stated that the pdf had been created by Oyvind Paasche, a Chapter 6 Contributing Author, on March 10, 2006. Curiously this version was identical to the May 10, 2005 version that I had received as a reviewer (converted to pdf form.) So the version online at WG1 (and presumably the version available to Chapter Author Paasche) was not the “accepted” version or even the September revision, but the original submission which failed to disclose the failed verification results. Recall the requirement “the TSU must hold final preprint copies of any unpublished papers that are cited in order that these can be made available to reviewers.” The Wahl and Ammann version then held by the WG1 TSU was not a “final preprint” – even the March 2006 version was not a “final preprint”. But the situation was even worse, the TSU was not even in possession of a current version of the paper.

TSU presumably became aware of the problem as a result of CA attention. Instead of placing a notice of the substitution on their website and sending a notice of the substitution to Reviewers so that any Reviewers that had downloaded the stale version could download the current version, they quietly substituted the new Wahl and Ammann version at the WG1 website without any notice whatever. I don’t know the exact date on which they made the substitution. I downloaded the newer version on June 2, 2006, which, again by coincidence, was the end of the 8-week review period for expert and Government comments.

The IPCC schedule provided that Review Comments be collated and sent to Chapter Lead Authors on or about June 16, 2006.
This schedule was closely adhered to. At the present website, the front page of the online Review Comments states that these are from “Batch AB (15 June 2006)”.

A couple of IPCC reviewers (with whom you are familiar) noticed the non-compliance of the Wahl and Ammann article (and a couple of other articles) with published IPCC publication deadlines and objected to the inclusion of these articles. See, for example, comments 6-1115, 6-1116, 6-748, 6-1158, 6-750, 6-1215 and 6-735.

Review Comment 6-1158 (B 29:41 29:41) stated:

Wahl and Ammann 2006 did not meet several publication deadlines. Is it fair to use this study when other studies also not meeting publication deadlines were not used? It was not accepted by December 13-15. TSU did not have a preprint by late February. The version available for review was not the same as the accepted version – in particular, the version made available omitted critical information that MBH98 failed cross-validation r2 and CE statistics. [Stephen McIntyre (Reviewer’s comment ID #: 309-119)]

or 6-1115 which stated:

It seems very unwise to me to waive IPCC WG1 policies on publication guidelines, especially for lead authors. For example, Osborn and Briffa 2006 did not meet the December deadline for being published or in print; it was not even mentioned in the First Draft nor was it available from TSU as part of the First Draft process. Other citations in the chapter did not meet the December deadline for being published or in press as at the December draft meeting (Osborn and Briffa 2006; Wahl et al 2006; Wahl and Ammann 2006; Hegerl et al “accepted”); several did not meet the February drop-dead date for providing TSU with a preprint (Wahl and Ammann 2006; Hegerl et al “accepted”). The version of Wahl and Ammann 2006 as accepted differed dramatically from the version provided to TSU for both the First Order and Second Order Drafts, notably in respect to the inclusion of their calculation of MBH verification statistics confirming the results of McIntyre and McKitrick showing failure of MBH verification statistics that had previously been denied. [Stephen McIntyre (Reviewer’s comment ID #: 309-12)]

The CLA Replies were, in a way, predictable. But there’s something very interesting in the wording of these rejections that I didn’t notice when the Review Comments became available in 2007. See if you notice.

Rejected, guidlines used for preparing the draft have been followed and new guidlines do not pose problems 6-1115)

Papers cited are within the guidlines for in press papers 6-1116)

Rejected- the citation is allowed current rules. 6-748

Rejected- the citation is allowed under current rules. 6-1158

See response to comment 6-1158. (6-750).

Rejected – revised deadlines mean that these papers are citable. 6-1215

Rejected -… all papers cited are within current IPCC publication deadline rules. 6-735

Notice that they didn’t contest the validity of the Review Comments under the Publication Deadlines that existed at the time that the Review Comments were made. Watch the pea under the thimble as the Chapter Authors purport to justify these Replies.

A more typical sort of Review Comment was the suggestion by authors that their own 2006 publications be cited. Here’s a typical example by D. Stone (9-476):

9-476 I conceitedly suggest adding the following recently accepted papers: Stone, Allen, Selten, Kliphuis, Stott. 2006. Stone, Allen, Stott. 2006. A multi-model update on the detection and attribution of global surface warming. Journal of Climate; Crooks, Allen, Stott. 2006. An update on the detection and attribution of observed temperature change – a space-time separable approach. Journal of Climate

Another example Review Comment 9-816:

9-816 A 49:1 49:2 This has been more thoroughly tested in: Pall, Allen, Stone. 2006. Testing the Clausius-Clapeyron constraint on changes in extreme precipitation under CO2 warming. Climate Dynamics.

Two weeks after the delivery of the above Review Comments to Chapter Lead Authors, pursuant to prior IPCC schedules, the Fourth Lead Author meeting took place in Bergen, Norway from June 25-30. The schedule states:

This meeting considers comments on the second order draft and revisions to produce the final draft start immediately afterwards.
June 25 chapter meetings
June 26-28 LA4 meeting
June 29-30 Meeting of the TS/SPM writing team

Most institutions make minutes of their meetings and I presume that IPCC Lead Author meetings are no different. It would be interesting to see the minutes of this meeting.

On July 3, 2006, IPCC WG1 sent an email to reviewers (which I received), a scraped version of which is onlinehere, a document the name of which indicates that it was written on July 1, 2006.

Guidelines for inclusion of recent scientific literature in the Working Group I Fourth Assessment Report.

We are very grateful to the many reviewers of the second draft of the Working Group I contribution to the IPCC Fourth Assessment Report for suggestions received on issues of balance and citation of additional scientific literature. To ensure clarity and transparency in determining how such material might be included in the final Working Group I report, the following guidelines will be used by Lead Authors in considering such suggestions.

In preparing the final draft of the IPCC Working Group I report, Lead Authors may include scientific papers published in 2006 where, in their judgment, doing so would advance the goal of achieving a balance of scientific views in addressing reviewer comments.

However, new issues beyond those covered in the second order draft will not be introduced at this stage in the preparation of the report. Reviewers are invited to submit copies of additional papers that are either in-press or published in 2006, along with the chapter and section number1 to which this material could pertain, via email to ipcc-wg1@al.noaa.gov, not later than July 24, 2006.

In the case of in-press papers, a copy of the final acceptance letter from the journal is requested for our records. All submissions must be received by the TSU not later than July 24, 2006 and incomplete submissions can not be accepted

A short turn time of July 24, 2006 was required because the IPCC schedule required that Chapter Lead Authors return annotated responses to all comments on the second draft to TSU by August 4 (which would be the approximate date that one can ascribe to the Chapter Lead Author Replies presently online). Given the well-publicized deadlines – ones that Judy Curry believed to have still existed – by what authority were these new guidelines issued? Are there minutes of the meeting? Did someone make a motion? Did someone second the motion? Was anyone opposed?

I presume that the Chapter Author Replies presently online date from approximately August 4, 2006. In response to Stone’s first suggestion, they replied:

The papers that are in press have been included.

Consulting the papers, the first timeline and second timeline for the two suggestions read:

Manuscript received 10 June 2005, in final form 21 April 2006)

Manuscript received 13 June 2005, in final form 24 April 2006)

So these references fall within the July 2006 variance – they were suggested by reviewers, they did not introduce a new topic and they were in final form. One wonders whether the TSU observed the protocol set out in the July letter to reviewers and obtained the required letter of acceptance from the Reviewer. IPCC policy requires that all written reviewer comments be placed in an open archive. The July 2006 correspondence would obviously be “written comments” under this definition, but wasn’t placed online with the other comments.

It does seem possible that Chapter 9 authors did check such matters. The other Stone suggestion referred to above online here was turned down as follows:

This paper is not citable here, since its status is ‘submitted’.

Its publication timeline shows that it was accepted on July 21, 2006 – a date which did not entitle it to be cited.

25 May 2005 / Accepted: 21 July 2006 / Published online: 30 August 2006

So you can see why Ammann and Wahl are so determined to preserve the fiction of a March 2006 acceptance date even Wahl and Ammann 200x was not “in final form” and “editorial components” were “incomplete”. I’ve browsed through the IPCC references and so far I’ve been unable to locate any other article that has a comparable timeline to Wahl and Ammann 2007. I’ll post on this separately.

The July variance is very limited and did not permit Chapter Lead Authors to include any 2006 article whatsoever. The policy required that a Reviewer “suggest” an article and “submit” it to the TSU. In-press articles had to be accompanied by the acceptance letter. In most walks of life, this sort of variance would be very strictly administered to avoid favoritism.

However, as we’ve seen above, the Chapter 6 Authors applied this rather specific variance to qualify Wahl and Ammann 200x,
Osborn and Briffa 2006 and Hegerl et al 200x.

In many contractual situations, they wouldn’t have been allowed to do this. The following argument would be made: had the Chapter Authors excluded these articles in the Second Draft, as IPCC policies required, Reviewers would be entitled to suggest their citation to clarify an issue under the July 2006 policy. But the July 2006 policy could not be used to salvage prior breaches of the policy by Chapter Authors, especially where the policy breach was in favor of their own article (Osborn and Briffa 2006) or other IPCC Chapter Authors (Hegerl et al 200x; Wahl and Ammann 200x).

Note that if Wahl and Ammann was supposedly qualified under the July policies, then a Reviewer was obligated to submit a copy of the Wahl and Ammann 200x acceptance letter to the IPCC TSU. This would be a “written comment” and should be on file. Will they have one? I doubt it, but we might as well find out.


  1. jeez
    Posted May 25, 2008 at 2:17 PM | Permalink

    Nice sleuthing Steve. You might find this humorous. An officer in my company attempting to cover her tracks or implicate me–I haven’t quite figured out the motivation yet, recently sent me a memo with an altered header, as if she was resending something from the past.

    This is what was altered as if automatically generated by the email system. Her supervisor, our CEO was cc’d on this email.

    Sent: Monday, November 06, 2007 1:12 PM

    Aside from noting that I had never received the purported original email. I took a second look at the header, glanced at a calendar and noted that November 06, 2007 was a Tuesday.

  2. Dennis Wingo
    Posted May 25, 2008 at 2:59 PM | Permalink

    Some people went to jail over something similar in Silicon Valley. It was called backdating stock options.

  3. MrPete
    Posted May 25, 2008 at 3:10 PM | Permalink

    second-to-last paragraph, typo: “July 2007” should be “July 2006” I think. Twice.

  4. David Holland
    Posted May 25, 2008 at 4:08 PM | Permalink

    The Guidlines for inclusion of recent scientific literature is an odd document. It is undated, unsigned and does not look all that official. Its pdf properties say it was authored by Martin Manning and is dated 1, July 2006 which is after the date by which the official time table with a file date of 12 April 2006 says:

    “Deadline for expert and Government comments on second draft: June 2” and “Comments distributed to CLAs by June 16”

    manning’s memo is in effect reopening the review process since it was only issued a month after the scheduled end.

    The official time table, which is also undated apart from its properties, filename and server file date, was updated with a new version dated 24 August 2006 , but there is no mention in it of the changes made by Martin Manning’s memo. Both time tables were authored by Roy Miller. If Manning’s memo was properly authorised I would have expected the time table to be updated to reflect it.

    By Manning’s memo, Wahl and Ammann (2007) doesn’t cut it as he says papers published in 2006 may be included. In Chapter 11, comment 11-781 Rasmus E. Benestad (Reviewer’s comment ID #: 18-34) puts in a plug for Fan et al (2005,2006) and was told –

    “Rejected. [discussion] But Fan et al. (2006) will not be published this year to meet the AR4 requirement of the deadline.

    So the CLAs of Chapter 11 knew the deadlines.

    For Wahl and Ammann to be in according to the rules Climatic Change would have had to have given a copper bottomed guarantee of 2006 publication – and if they did they would have some questions to answer.

    Incidentally, the email address in the Manning memo is ipcc-wg1@al.noaa.gov and must be subject to US Freedom of Information Law. I will be asking some questions this side of the pond, hopefully someone in the US will also ask a few. One thing we really need is to have reviewers comments and authors responses dated. The dates must have been there to start with so why would anyone strip them off?

  5. Jon
    Posted May 25, 2008 at 4:44 PM | Permalink

    Steve, although you have a clear writing style: perhaps a time-line graphic with numbered notes would be appropriate.

    e.g. (http://www.levenez.com/unix/history.html)

  6. Armand MacMurray
    Posted May 26, 2008 at 1:25 PM | Permalink

    Minor nitpick: the use of the word “trolling” to describe capturing a snapshot in time of a website or document grates on me (because of its very different common internet-related meaning). Perhaps you might switch to a word such as “scraping”?

  7. Henry
    Posted Jun 4, 2008 at 6:46 PM | Permalink

    The world gets murkier.

    New Scientist ran an article on 1 May 2008 saying that IPCC 4AR is already out-of-date since it was only based on research up to 2005. UK government scientists John Mitchell and Robert Watson wrote in (published 29 May 2008) contesting this and saying that 4AR had articles published up to early 2007. Wahl and Ammann 2007 was in fact published online late in August 2007 and printed in the November edition of Climatic Change, so “early” needs to be interpreted flexibly.

    The odd thing about the Ammann and Wahl 2007 article The importance of the geophysical context in statistical evaluations of climate reconstruction procedures published simultaneously in Climatic Change was that it seems to have been submitted in August 2000 (or so the Springer site states) despite not being accepted until June 2006. This seems an extremely long time – had the world not moved on? It would be interesting to see how it changed over that time. Clearly it was added to, to meet the need to replace contents of the rejected GRL Ammann and Wahl ‘Comment on “Hockey sticks, principal components, and spurious significance” by S. McIntyre and R. McKitrick’.

  8. David Holland
    Posted Jun 7, 2008 at 3:41 AM | Permalink

    I said in #4 that Manning’s 1 July 2006 memo, which invited new citations, in effect reopened the review process. I was wrong. It only reopened the drafting process since their was no subsequent expert and government review of added citations. As such the step must fall foul of Clause 3 of the Principles Governing IPCC Work which states,

    Review is an essential part of the IPCC process. Since the IPCC is an intergovernmental body, review of IPCC documents should involve both peer review by experts and review by governments.

    This Manning memo in my view invalidated the process.

    Re #7,
    The copy I have seen of Ammann and Wahl 2007 says accepted 13 June 2007 not 2006. Steve has reported that the ‘received 22 August 2000’ was a printers error – should have been 2006

    Steve: There are two articles: Ammann and Wahl 2007 and Wahl and Ammann 2007. It is WA that is cited in AR4. However, it cites AW 2007.

  9. John F. Pittman
    Posted Jun 7, 2008 at 6:06 PM | Permalink

    Steve McI commenting on #8. Are these

    Ammann and Wahl 2007 and Wahl and Ammann 2007

    available for free yet? If so can you link (them again, if you already have), so that we can read them? I keep getting the North report when I google.

    Perhaps they (it???) should be posted on “CATEGORIES” on your site so that we could reveiw them (it??)?

    Steve: Check http://data.climateaudit.org/pdf/others/. There is a Wahl and Ammann category under MBH/

  10. John F. Pittman
    Posted Jun 8, 2008 at 6:44 AM | Permalink

    Thanks Steve. Could you comment on this quote from the Climatic Change DOI 10.1007/s10584-007-9276-x? I am curious with the claim that MM05c procedure “trained” a low frequency climate signal into what was (should have been?) non-climatic noise. Is this really possible?

    To generate “random” noise series, MM05c apply the full autoregressive structure
    of the real world proxy series. In this way, they in fact train their stochastic engine
    with significant (if not dominant) low frequency climate signal rather than purely
    non-climatic noise and its persistence. The resulting verification RE significance
    thresholds for rejecting the likelihood of spurious, random “success” derived from
    reconstructions using these pseudo-proxies are therefore too large by construction,
    precisely because they contain at least some climate signal along with purely
    stochastic processes. Such thresholds thus enhance the danger of committing Type
    II errors (inappropriate failure to reject a null hypothesis of no climatic information
    for a reconstruction).

4 Trackbacks

  1. By Mosher: The Hackers « Watts Up With That? on Jan 26, 2010 at 3:37 PM

    […] and are supposed to reference only works that are published in the peer reviewed science that met specific time deadlines. The deadlines are critical to have a fair and open process, ensuring that papers referenced have, […]

  2. […] comments. There is no simple take away. Except this: the Climate scientists corrupted the IPCC process, the science journal process, the statistical process, and the FOIA process to silence one […]

  3. […] British Manipulation Of Scientific Publishing, circa 1964 13 07 2010 Worried about some dodgy behavioural traits of some prominent British scientists? Astonished at the cavalier attitude regarding publications and dates by IPCC Lead Authors? […]

  4. […] access), the IPCC staff allowed the chapter authors to refer to it, even though the paper had not even been accepted for publication by the official IPCC deadline! See here for a detailed summary of exactly why the Wahl & […]

%d bloggers like this: