As I mentioned the other day, it’s very interesting for me to re-read the responses of Team members to the publication of MM2003. While Mann, Briffa and Bradley all start shooting bullets in every direction, Osborn’s reaction is thoughtful and nuanced and I urge readers to read it in full. Unfortunately, the Team paid little attention to Osborn’s suggestions. Had they done so, much time and effort would have been saved.
Mann’s response to MM2003 is, in retrospect, all too predictable. For example, Mann here:
The important thing is to deny that this has any intellectual credibility whatsoever and, if contacted by any media, to dismiss this for the stunt that it is..
Or in an email a few days later, one that provides an interesting insight into Mann’s idea of the “high road”:
Takinig the high road is probably very important here. If *others* want to say that their actions represent scientific fraud, intellectual dishonesty, etc. (as I think we all suspect they do), lets let *them* make these charges for us!
Bradley wanted CRU to make a statement on whether our study was “truly an audit” and whether we did it “right”. However, he didn’t want a statement that actually involved due diligence. He wanted a “quick and forceful statement” from the “Distinguished CRU Boys” to “quash further arguments”:
if an “independent group” such as you guys at CRU could make a statement as to whether the M&M effort is truly an “audit”, and if they did it right, I think that would go a long way to defusing the issue. ..
If you are willing, a quick and forceful statement from The Distinguished CRU Boys would help quash further arguments
Briffa’s reaction was to get Nature to write an editorial (perhaps presaging Nature’s later response to Climategate.) Briffa wanted a response along the lines of:
some cool statement can be made saying we believe the “prats have really fucked up someway” – and that the premature publication of their paper is reprehensible.
The only person with a reasoned response was Osborn. I read his October 31 response with considerable regret. Had Osborn’s direction been followed, an enormous amount of time would have been saved.
(1) The single worst thing about the whole M&M saga is not that they did their study, not that they did things wrong (deliberately or by accident), but that neither they nor the journal took the necessary step of investigating whether the difference between their results and yours could be explained simply by some error or set of errors in their use of the data or in their implementation of your method. If it turns out, as looks likely from Mike’s investigation of this, that their results are erroneous, then they and the journal will have wasted countless person-hours of time and caused much damage in the climate policy arena.
(2) Given that this is the single worst thing about the saga, we must not go and do exactly the same in rushing out a response to their paper. If some claims in the response turned out to be wrong, based on assumptions about what M&M did or assumptions about how M&M’s assumptions affect the results, then it would end up with a number of iterations of claim and counter claim. Ultimately the issue might be settled, but by then the waters could be so muddied that it didn’t matter.
I disagree with the premise that we had not attempted to investigate the differences with MBH results (our inquiries to Mann had been blown off). See our contemporary account here. However, today I don’t want to re-litigate this point, but instead to focus on new points arising out of Osborn’s response.
Osborn was far more aware than his associates of the potential for “claim and counterclaim” that could be set off by an injudicious response, resulting in a circumstance where by the time of an eventual settlement, the “waters [would be] so muddied” that settlement wouldn’t matter. Osborn was worried about on October 31, 2003 that “countless person-hours” had already been “wasted”, but I doubt that even he realized where the Team’s injudicious decision not to follow his recommendations would take them.
Osborn was also the only member of the Team that had actually looked at MBH methodology. The Climategate Letters show that Osborn had looked at MBH methodology in the previous summer with Mann even sending Osborn, as a “trusted colleague”, his residuals, which Mann described as his “dirty laundry”. Perhaps that was in Osborn’s mind in the composition of his comments.
Later in Osborn’s draft reply, he would urge “collaboration with M&M” to avoid such the eventuality of “claim and counterclaim”. We would have been very willing to do so and made offers to Osborn to involve CRU in reviewing the issues. Unfortunately, the Team did not follow Osborn’s recommendations.
Osborn continued with a discussion of whether CRU was compromising its own “independence” by signing on to support MBH at this early stage. (This isn’t reviewed here – see original.)
Skpping ahead a couple of paragraphs, Osborn urged a “very careful reading” of MM2003, as opposed to a kneejerk reaction:
I really advise a very careful reading of M&M and their supplementary website to ensure that everything in the response is clearly correct – precisely to avoid point (2). I’ve only just started to do this, but already have some questions about the response that Mike has drafted.
Osborn’s first technical question related to principal components. Although the use of stepwise principal components – a different issue than decentered principal components – is not mentioned in MBH98, Osborn was aware that Mann had used this procedure and reasonably wondered whether this might account for differences in our results. If so, matters might be resolved fairly expeditiously without the need for opprobrium on the Team’s part.
(a) Mike, you say that many of the trees were eliminated in the data they used. Have you concluded this because they entered “NA” for “Not available” in their appendix table? If so, then are you sure that “NA” means they did not use any data, rather than simply that they didn’t replace your data with an alternative (and hence in fact continued to use what Scott had supplied to them)? Or perhaps “NA” means they couldn’t find the PC time series published (of course!), but in fact could find the raw tree-ring chronologies and did their own PCA of those? How would they know which raw chronologies to use? Or did you come to your conclusion by downloading their “corrected and updated” data matrix and comparing it with yours – I’ve not had time to do that, but even if I had and
I found some differences, I wouldn’t know which was right seeing as I’ve not done any PCA of western US trees myself? My guess would be that they downloaded raw tree-ring chronologies (possibly the same ones you used) but then applied PCA only to the period when they all had full data -
hence the lack of PCs in the early period (which you got round by doing PCA on the subset that had earlier data). But this is only a guess, and this is the type of thing that should be checked with them – surely they would respond if asked? – to avoid my point (2) above. And if my guess were right, then your wording of “eliminated this entire data set” would
come in for criticism, even though in practise it might as well have been.
Osborn was right in his surmise that we had not used the unreported stepwise principal components method – we had described what we had done in our paper and attached source code to clarify this sort of question. “Stepwise” principal components is not a “conventional” methodology – at the time, I couldn’t locate other precedents nor have I seen any since. However, as most CA readers know, there was another shoe to drop. Even using stepwise principal components, it was not possible to directly replicate Mann’s principal components, because he had used decentered principal components – something that Mann had possibly not realized. In any event, it wasn’t mentioned in MBH98. This has been the topic of endless later discussion (and I don’t propose to even touch on these issues here.) This then led into discussion of how many principal components to retain and bristlecones and all that stuff – none of it particularly difficult, had the Team elected not to “muddy the waters”.
Osborn continued by expressing concern that we will “use the email record” if they mention “ftp sites and excel files” in their response, as he is aware that Mann’s allegations were contradicted by the actual email record. Osborn:
(b) The mention of ftp sites and excel files is contradicted by their email record on their website, which shows no mention of excel files (they say an ASCII file was sent) and also no record that they knew the ftp address. This doesn’t matter really, since the reason for them using a corrupted data file is not relevant – the relevant thing is that it was corrupt and had you been involved in reviewing the paper then it could have been found prior to publication. But they will use the email record if the ftp sites and excel files are mentioned.
This refers to Mann’s claim, then widely publicized, that we had asked for “excel” files, that an error had been introduced in the preparation of the “excel” files and various other untrue allegations that Mann had disseminated through David Appell on Oct. 29, 2003. Here’s an excerpt, but there are numerous other fantastic assertions:
In short, here’s what happened: M&M asked an associate of Mann to supply them with the Mann et. al. proxy data in an Excel spreadsheet, even though the raw data is available here. An error was made in preparing this Excel file, in which the early series were successively overprinted by later and later series, and this is the data M&M used….
The spreadsheet file they used was a complete distortion of the actual Mann et. al. proxy data set, and was essentially useless, particularly in the earlier centuries. The authors had access to the full data, which has been available on a public ftp site for nearly two years. When they noticed, as described in their paper, some signs of problems with the Excel spreadsheet version of the data, one might think that they would have bothered to check the data available on our public ftp site.
See our contemporary response here, showing that we had never asked for an Excel spreadsheet (something that is obviously foreign to my desire to work with original data) and that the file at Mann’s website to which we were directed was dated to 2002, long before our request. And far from “not noticing” the errors in this file, MM2003 contains a detailed listing of defects in the file.
Further color was added to events when, a few days later, Mann deleted the dataset in controversy. (The University of Massachusetts poured gasoline on the fire by deleting MBH99 data from their website a few days later as well. Upon protest, they restored their data a few days later, but Mann didn’t.) Osborn’s awareness of the deletions is evident on Nov 12 here when he emails Jones and Briffa:
I do wish Mike had not rushed around sending out preliminary and incorrect early responses – the waters are really muddied now. He would have done better to have taken things slowly and worked out a final response before publicising this stuff. Excel files, other files being created early or now deleted is really confusing things!
This incident was my first introduction to the Team just making stuff up. I couldn’t believe that someone described as a “Scientific American Visionary” would either stoop to making up stories or to be so foolish as to make up stories that were readily refuted by the email record. I was even more amazed that the climate science “community” acquiesced in this sort of fabrication. This was totally outside my experience.
Osborn’s response here shows that he is acutely aware that Mann’s characterization of the email record is untrue. However, Osborn does not show any awareness of any obligation on his part to ensure that the record is accurate – something that one would have expected of him as a scientist. Instead, his thinking is entirely tactical – mentioning the “excel files” will open them up to criticism. Osborn was not acting here as a scientist whose responsibility was to ensure that the record was correct, but an advocate.
Skipping ahead a couple of paragraphs – I am skipping commentary on all items only because this is already a long post – Osborn observed that we had mentioned the “decline”. Again, his recommendation on how to “handle” the point is primarily based on advocacy rather than science:
(f) The recent tree-ring decline they refer to seems related to tree-ring-width not density. Regardless of width of density, this issue cannot simply be dismissed as a solved problem. Since they don’t make much of an issue out of it, best just to ignore it.
Skipping further ahead, Osborn says that it would be incorrect for the MBH response to allege that we “used neither the data nor the procedures of MBH98″. Osborn observed that we had at least approximately done so with the materials available to us.
(h) To say they “used neither the data nor the procedures of MBH98″ will also be an easy target for them, since they did use the data that was sent to them and seemed to have used approximately the method too (with some errors that you’ve identified). This reproduced your results to some extent (certainly not perfectly, but see Fig 6b and 6c). Then they went further to redo it with the “corrected and updated” data – but only after first doing approximately what they claimed they did (i.e. the audit).
Despite Osborn’s objection, CRU published the MBH statement making the claim that he had objected to here as being incorrect.
Osborn then once again expressed concern that a quick reponse would simply “muddy the debate for most outsiders”. An accurate prediction. Osborn then proposed a very different response than the one eventually published. Here is Osborn’s very different draft:
The recent paper by McIntyre and McKitrick (2003; hereafter MM03) claims to be an “audit” of the analysis of Mann, Bradley and Hughes (1998; hereafter MBH98). MM03 are unable to reproduce the Northern Hemisphere temperature reconstruction of MBH98 when attempting to use the same proxy data and methods as MBH98, though they obtain something similar with clearly anomalous recent warming (their Figure 6c). They then make many modifications to the proxy data set and repeat their analysis, and obtain a rather different result to MBH98.
Unfortunately neither M&M nor the journal in which it was published took the necessary step of investigating whether the difference between their results and MBH98 could be explained simply by some error or set of errors in their use of the data or in their implementation of the MBH98 method. This should have been an essential step to take in a case such as this where the difference in results is so large and important. Simple
errors must first be ruled out prior to publication. Even if the authors had not undertaken this by presenting their results to the authors of MBH98, the journal should certainly have included them as referees of the manuscript.
A preliminary investigation into the proxy data and implementation of the method has already identified a number of likely errors, which may turn out to be the cause of the different results. Rather than repeating M&M’s failure to follow good scientific practise, we are witholding further comments until we can – by collaboration with M&M if possible – be certain of exactly what changes to data and method were made by M&M, whether these changes can really explain the differences in the results, and eventually which (if any) of these changes can be justified as equally valid (given the various uncertainties that exist) and which are simply errors that invalidate their results.
Obviously, we would have eagerly co-operated with CRU (or anyone else) to reconcile results. After publication of MM2003, we asked CRU to review our results (in the form of what would become our Nature submission.) I’ll review this in a separate post but here is one email from me to Osborn to give a flavor of our contemporary attitude:
Dear Dr. Osborn and others,
We have entered into discussions about a possible review by UEA/CRU in complete good faith. We do not have the slightest interest in presenting incorrect or defective results or to create debate which is merely at cross-purposes. Regards, Steve McIntyre
We continued our efforts to reconcile methodological particulars with Mann (for example here copy to Osborn, an email also occurring as a trailer in a Climategate Letter here) but these were repudiated by Mann, apparently without any objection from Osborn or CRU who elsewhere had urged that results be reconciled.
This is another interesting backstory which I’ll discuss in a forthcoming thread. If the Team had adopted Osborn’s recommendations instead of Mann’s, perhaps it would still have be conceivable to use the Bradley’s term – the “Distinguished CRU Boys” – without irony.