The Dog That Didn't Bark

This is the title of a famous Sherlock Holmes story and not intended as a slight to any individual.

Take a look at the Review Comments for AR4 Second Draft Chapter 6 online here.

While I was reviewing these comments, I noticed that there are no reported comments on chapter 6 from Caspar Ammann, one of the major participants in the paleoclimate debate. In the Chapter 6 Review Comments, there are many comments about his article, but none by him. In the IPCC roster., he is shown as a Contributing Editor but not a reviewer [Note - this has been modified in light of #7 below].

I’ve just gone through the exercise of searching all the First Draft and Second Draft Review Comments for any on-the-record comments by Ammann and found none. Perhaps the online Review Comments do not include comments by reviewers if they are Contributing Authors to another chapter. If anyone feels like perusing the Review Comments for examples, I’d be interested. This is a bit surprising in a way, since Susan Solomon’s review comments were submitted through the IPCC review process and are on the record, so why wouldn’t Caspar Ammann’s comments on chapter 6 also not be on the record?

Terminology from a then unsubmitted paper turns up in the Chapter Author replies to Review Comments. Compare the terminology in the answer to Review Comment 6-735 (which would have been final around Aug 4, 2006) to the language in Ammann and Wahl, which, as we recently learned, was submitted on August 22, 2006. Some language points track exactly and do not occur elsewhere in the literature. So there is no doubt that Ammann made written comments to the Chapter Authors of Chapter 6, but these are not on the record despite IPCC policy which states:

All written expert, and government review comments will be made available to reviewers on request during the review process and will be retained in an open archive in a location determined by the IPCC Secretariat on completion of the Report for a period of at least five years.

Update: Here is an amazing parallel between the Replies to Review Comments for chapter 6 and Ammann and Wahl 2007. The Reply to Review Comment 6-735 stated:

Rejected – the text is based on the authors’ interpretation of the current literature (and all papers cited are within current IPCC publication deadline rules). The text gives a balanced view.

Please note the following –
The MM05d benchmarking method is based on an entirely different analytical framework than that used by MBH98.

MBH used the standard method in climatology of making a random time series based on the low-order AR characteristics of the target time series during the calibration period. here the N. Hemisphere mean. This random process is repeated in Monte Carlo fashion and its skill in replicating the actual target time series is evaluated according to any measure of merit in which the investigator is interested.

MM’s method instead uses the full order AR characteristics of one of the proxies used in the reconstruction to create pseudoproxies in a Monte Carlo framework. These are then input into the reconstruction algorithm along with white noise pseudoproxies for all the n-1 remaining proxies. This is, in theory a statistically meaningful procedure, which asks what kind of apparent skill is available in the reconstruction simply from one proxy’s noise. However this procedure is not general and would need to be repeated for each proxy set to be examined. Also, it would need the subjective choice of which single proxy should be modelled according to its red noise characteristics each time.

Finally, it does not take into account that some of the verifications seen as “skillful” are associated with very poor/exceedingly poor calibrations, which would be rejected on first principles in real world reconstruction applications. This consideration indicates that the 0.51 threshold cited by MM is actually, at least somewhat, overstated.

I defy anyone to show me any support for these comments in peer reviewed literature as at August 4, 2006. But here are some quotes from Ammann and Wahl 2007:

Standard practice in climatology uses the red-noise persistence of the target series (here hemispheric temperature) in the calibration period to establish a null-model threshold for reconstruction skill in the independent verification period, which is the methodology used by MBH in a Monte Carlo framework to establish a verification RE threshold of zero at the >99% significance level.

Rather than examining a null model based on hemispheric temperatures, MM05a,c report a Monte Carlo RE threshold analysis that employs random red-noise series modeled on the persistence structure present in the proxy data (note, noise here is meant in the sense of the ‘signal’ itself, rather than as an addition to the signal). … RE performance thresholds established using this proxy-based approach have the disadvantage of not being uniformly applicable; rather, they need to be established individually for each proxy network.

Furthermore, the MM05c proxy-based threshold analysis only evaluates the verification-period RE scores, ignoring the associated calibration-period performance. However, any successful real-world verification should always be based on the presumption that the associated calibration has been meaningful as well …

Interestingly, the first paragraph cited above caught my eye in Ammann and Wahl and I asked him for a supporting reference, justifying this argument. To which, Ammann, a federal employee, answered:

why would I even bother answering your questions, isn’t that just lost time?


Update Jan 2009:
This particular mystery was resolved by the Climategate Letters, which contain off-the-record correspondence between Wahl and Briffa.


23 Comments

  1. Nathan Schmidt
    Posted May 24, 2008 at 9:38 AM | Permalink

    Ammann?

  2. kim
    Posted May 24, 2008 at 9:41 AM | Permalink

    Heh, in the middle of ‘Hound of the Baskervilles’ right now.
    ===================================

  3. steven mosher
    Posted May 24, 2008 at 9:55 AM | Permalink

    SteveMc look at the dogs who did bark and bite.

    I think Susan Soloman batted 1000 in getting comments accepted.

    It might be tough to do but a hitting percentage would be fun.

    Comments are either rejected, noted or accepted.

    who had the highest acceptance rates, who had the highest rejection rates.

  4. A Camp
    Posted May 24, 2008 at 10:15 AM | Permalink

    Silver Blaze is the story where the dog doesn’t bark. There is no SH story by that name. Just saying.

  5. Steve McIntyre
    Posted May 24, 2008 at 10:16 AM | Permalink

    #1. Right first time.

    Ponder this. In the Second Draft Review Comments, “Ammann” occurs 64 times (!) but Ammann doesn’t comment himself. How odd is that?

    Caspar Ammann is listed by IPCC as an Reviewer, but he didn’t make any review comments. I wonder how many other people are listed as reviewers, but didn’t make review comments.

    Now I think that there’s overwhelming evidence that Ammann communicated with Chapter Authors – I’ll post up some evidence later today. It’s just that he seems to have done so off the record and not to have done so on-the-record as Ross and I were required to do.

  6. Thor
    Posted May 24, 2008 at 10:37 AM | Permalink

    I had a look at the AR4 WG1 annexes, and it seems Caspar Ammann is listed in Annex II (Contributors) on page 955. However, he is not listed in Annex III (Reviewers) which start on page 969.
    Could that be why?

  7. Bill Drissel
    Posted May 24, 2008 at 11:16 AM | Permalink

    Sherlock Holmes: “… the curious incident of the dog in the night-time.”
    Inspector Gregory: “The dog did nothing in the night-time.”
    SH: “That was the curious incident.”
    Silver Blaze, A. C. Doyle

  8. Steve McIntyre
    Posted May 24, 2008 at 1:21 PM | Permalink

    #7. Ammann was a Contributing Author to chapter 9 (but not chapter 6). So you have a point and I’lll modify the post accordingly. But I think that it raises an interesting question. Did Ammann submit written comments on chapter 6?

    IPCC procedure states:

    All written expert, and government review comments will be made available to reviewers on request during the review process and will be retained in an open archive in a location determined by the IPCC Secretariat on completion of the Report for a period of at least five years.

    Even if Ammann were a Contributing Author of chapter 9, if he submitted written comments to the Chapter Authors of chapter 6, shouldn’t those be included in the open archive? Susan Solomon’s comments are on the record. It is inconceivable that Ammann didn’t submit comments on chapter 6. Shouldn’t they be on the record?

  9. Barclay E. MacDonald
    Posted May 24, 2008 at 1:46 PM | Permalink

    I can’t wait to find out if there is a point to all this that we should care about.

  10. Steve McIntyre
    Posted May 24, 2008 at 2:01 PM | Permalink

    #10. We’re getting to the main issues. Still laying a little groundwork. This is a blog after all. Glad that your interest is whetted.

    I’ve added a section to the above post showing the remarkable parallelism in language between the Reply to Review Comments for Review Comment 6-735 and language in then unsubmitted Ammann and Wahl. The Chapter Authors asserted that they were giving a “balanced” view of the literature, while relying on unpeer reviewed opinion from Ammann to supposedly rebut Review Comments.

  11. Max
    Posted May 24, 2008 at 2:35 PM | Permalink

    I thought with that title it would foster a different kind of debate. SM, I think these people see you as adversarial, however listening to your radio interview, you laid out your viewpoints very carefully to not bring on that kind of response, and I noticed that you don’t give up meaty sound bites for the media to run away with. It could be the just the way these people read the tone out of your emails, have you ever tried phoning, instead of writing?

  12. Michael Jankowski
    Posted May 24, 2008 at 3:31 PM | Permalink

    All written expert, and government review comments will be made available to reviewers on request during the review process and will be retained in an open archive in a location determined by the IPCC Secretariat on completion of the Report for a period of at least five years.

    Maybe that’s the catch…his communications weren’t written and therefore weren’t required to be available and archived.

  13. Craig Loehle
    Posted May 24, 2008 at 3:32 PM | Permalink

    I believe what we are seeing is the belief among some scientists (and it is not just in climate science) that there is an inner circle of anointed scientists who can dispense with certain formalities required of lesser mortals. Formalities like proving your point, citing authoritative references, and taking criticism seriously. Once you reach this exalted station, you can speak from authority and everyone is supposed to listen to you. Steve M. violates the code by not bowing down with sufficient obsequiousness.

  14. Will J. Richardson
    Posted May 24, 2008 at 3:49 PM | Permalink

    Re: Max in Comment 11

    “I think these people see you as adversarial . . .”

    “The wicked flee when no man pursueth: but the righteous are bold as a lion.” Proverbs 28:1, KJV

  15. Steve McIntyre
    Posted May 24, 2008 at 3:54 PM | Permalink

    #12. The turn of phrase between Ammann and Wahl 2007 and the Reply 6-735 matches too precisely for a written comment not to have been made.

  16. Pat Keating
    Posted May 24, 2008 at 6:30 PM | Permalink

    15 Steve

    …unless he read them a section of the manuscript….

  17. jae
    Posted May 24, 2008 at 8:55 PM | Permalink

    13, Craig:

    I believe what we are seeing is the belief among some scientists (and it is not just in climate science) that there is an inner circle of anointed scientists who can dispense with certain formalities required of lesser mortals. Formalities like proving your point, citing authoritative references, and taking criticism seriously. Once you reach this exalted station, you can speak from authority and everyone is supposed to listen to you. Steve M. violates the code by not bowing down with sufficient obsequiousness.

    LOL, sad but true, even when big money and politics are not involved.

  18. Geoff Sherrington
    Posted May 24, 2008 at 10:02 PM | Permalink

    Re Sir Arthur Ignatius Conan Doyle – horrible trivia

    It is on record that the novel hero was nearly named “Sherrington Hope” but was finally cast as “Sherlock Holmes”. Numerous Google references. Also note that Toronto Public Library has an extensive collection of Doyle’s works, eh Steve?

    Re Ammann. Is Caspar the Friendy Ghost writer?

  19. steven mosher
    Posted May 25, 2008 at 9:34 AM | Permalink

    The texts are teleconnected.

    Now I get to say this because I am a textual scientist, having caught out many an undergradute on “inappropriate attributions”

    Things that you look for. Odd locutions repeated. structural similarity in paragraphing. Structural similarity in argumentation.

    Minor edits that seek to disguise the source. basically, trying to hide the source by adding words or deleting words

  20. Bob B
    Posted May 25, 2008 at 3:40 PM | Permalink

    Steve, I know of an expression—“That dog don’t hunt”
    I think it would apply better

  21. Posted May 25, 2008 at 11:10 PM | Permalink

    #19 is proof positive that Mosher is really me. Why you ask?

    I’m notorious for short one sentence paragraphs. Rafts of them. A rather unusual style.

    And here I am breaking my own rule except for this paragraph, however the others are short.

    So there. :-)

  22. Posted Jul 27, 2008 at 11:40 PM | Permalink

    This is really interesting. When he said:

    “would I even bother answering your questions, isn’t that just lost time?”

    I wonder about what Ammann was trying to hide…

  23. Steve McIntyre
    Posted Feb 6, 2012 at 2:15 PM | Permalink

    A CG2 email from Tim Osborn expressing a somewhat disdainful attitude toward my “”proof” that Ammann corresponded directly with us and not via the official review comments process, and thus his correspondence is apparently not included in the official set of comments/responses”
    The correspondence was between Ammann coauthor Wahl rather than Ammann himself, but the surmise was 100% correct. This makes me question the degree to which Osborn (and perhaps even Jones) were aware of the surreptitious contact between Wahl and Briffa and that Wahl had re-written IPCC language that had been sent to external reviewers to suit his own ends.

    1701. 2008-05-26 10:12:07
    ______________________________________________________
    cc: t.osborn@
    date: Mon, 26 May 2008 10:12:07 +0100 (BST)
    from: “Tim Osborn”
    subject: FOI stuff
    to: k.briffa, p.jones
    Hi Keith and Phil,
    I see on ClimateAudit that there are now many posts on Wahl & Ammann, its citing by Chap 6 of IPCC and how this fits with the publication deadlines etc., late changes to the IPCC publication deadline rules (a memo from Manning), plus “proof” that Ammann corresponded directly with us and not via the official review comments process, and thus his correspondence is apparently not included in the official set of comments/responses:

    McIntrye: “I’ve added a section to the above post showing the remarkable parallelism in language between the Reply to Review Comments for Review Comment 6-735 and language in then unsubmitted Ammann and Wahl. The Chapter Authors asserted that they were giving a “balanced” view of the literature, while relying on unpeer reviewed opinion from Ammann to supposedly rebut Review Comments.”

    David Holland comments that he will follow up some of these issues:
    “Incidentally, the email address in the Manning memo is ipcc-wg1@al.noaa.gov and must be subject to US Freedom of Information Law. I will be asking some questions this side of the pond, hopefully someone in the US will also ask a few.”

    So I’m sure that Holland will be pursuing or even expanding his FOI request! I can’t remember what emails I do or don’t have, but it would be useful if we do have the one from Wahl/Ammann confirming that their article received final acceptance, to demonstrate that we followed the rules concerning the deadlines for material that we cited.
    Tim

3 Trackbacks

  1. By Mosher: The Hackers « Watts Up With That? on Jan 26, 2010 at 3:37 PM

    [...] exists outside of the climategate mails. The mails, merely confirm what Steve McIntyre had already figured out.  The mails of course add some background that strengthens the case, including the hacker’s [...]

  2. By “Keith Should Say…” « Climate Audit on Apr 2, 2010 at 5:31 PM

    [...] unsubmitted paper get applied in Review Comments purporting to dismiss Mc-Mc results? On May 24 here, I surmised that, contrary to Mitchell (and Solomon’s) representation that all the review [...]

  3. By Acton “Tricks” the ICO « Climate Audit on Feb 6, 2012 at 2:14 AM

    [...] as his “borrowed (stolen)” language was noticed in a CA post of May 24, 2008 (The Dog That Didn’t Bark), leading me to postulate surreptitious non-open non-transparent contact between Wahl and Ammann [...]

Follow

Get every new post delivered to your Inbox.

Join 3,203 other followers

%d bloggers like this: