Climate Audit

The Oxburgh “Report” states that the eleven CRU publications that they examined are “representative”, “were selected on the advice of the Royal Society” and that CRU agreed that they are a “fair sample” of CRU’s work.

The eleven representative publications that the Panel considered in detail are listed in Appendix B. The papers cover a period of more than twenty years and were selected on the advice of the Royal Society. All had been published in international scientific journals and had been through a process of peer review. CRU agreed that they were a fair sample of the work of the Unit.

A “fair sample”?

To the extent that Climate Audit posts are a yardstick for CRU articles/documents that are at issue, the Oxburgh “fair sample” is almost mutually exclusive of the CRU articles discussed here. Nor were the Oxburgh articles at issue in, for example, my submission to the Parliamentary Committee and the Muir Russell “Team”.

Aside from CRU activities at IPCC (the sections in AR3, AR4 and AR4 Review Comments), the most prominent CRU articles criticized here are the following nine: Briffa et al 1992 (the Tornetrask chronology and “Briffa bodge”); Briffa et al 1995 (Polar Urals), Briffa 2000 (passim introduction of Yamal, Taimyr); Briffa et al 2002 (the famous cargo cult “assumption”); Mann and Jones 2003; Jones and Mann 2004; Osborn and Briffa 20056; Rutherford et al 2005; Jones et al (1990) on UHI.[Oct 27: also obviously Jones et al 1998].

The Oxburgh Eleven includes five tree ring articles (Briffa et al, Nature, 1998; Briffa et al, Roy Soc Lond 1998; Briffa 2000; Briffa et al 2001; Briffa et al 2008). Four of these articles were noted relatively favorably in my May 2005 post in which I first drew attention to the “trick” A Strange Truncation of the Briffa MXD Series (see image below). I had no complaint with the original Briffa articles – it was the IPCC spaghetti graph with its false rhetorical effect that bothered me. Even for veteran watchers of peas under thimbles, it’s pretty amazing that four CRU articles – that not only had had not been the subject of criticism, but had been used to locate the trick – were chosen as somehow “representative” of the CRU corpus, while the articles that had actually been criticized here were for the most part excluded. Every CRU hockey stick article (Jones et al 1998; Mann and Jones 2003; Osborn and Briffa 2006) was excluded.

So, in addition to not taking any evidence from CRU critics or targets, the terms of reference for the execrable Oxburgh “inquiry” diverted its attention away from articles that actually were at issue to other articles that had actually been used as source material to identify the trick in the first place.

Excerpt from IPCC 2001 “trick” spaghetti graph.

This raises a few obvious questions. The Oxburgh Report states that the eleven articles were “selected on the advice of the Royal Society”. However, they did not provide any information on how the Royal Society determined that these eleven publications were “representative”. Nor did they mention who at the Royal Society actually made the selection. The report says that UEA agreed that the Royal Society selection was a “fair sample”. I wonder who at UEA actually agreed that the selection was a “fair sample” and what their criteria were.

One of the recommendations of every inquiry so far is that methodologies be properly disclosed. Oxburgh didn’t disclose how they selected their supposedly “representative” and “fair sample”. “Fair sample” and “representative” are statistical terms – terms were used in a report coauthored by a very senior professional statistician in a context where statistics are very much at issue. So I presume that the Royal Society took some care to ensure that the eleven publications actually were “representative” and a “fair sample” – and not ones that were pre-selected by UEA, rather than the Royal Society.

I’ve written to both UEA and the Royal Society seeking clarification on the selection process and will keep readers posted.

Update Apr 16: Here is my request to the UEA under the Environmental Information Regulations:

Dear Mr Palmer,

Pursuant to the Environmental Information Regulations, I hereby request correspondence between the University of East Anglia and/or its officers and the Royal Society between December 1, 2009 and April 12, 2010 concerning the selection of publications considered in the Oxburgh “report”. Thank you for your attention.

Regards, Stephen McIntyre

I received acknowledgment of this request this morning.

I sent the following inquiry to the Royal Society yesterday:

The Oxburgh Report lists 11 CRU papers in Appendix B stating: “The eleven representative publications that the Panel considered in detail are listed in Appendix B. The papers cover a period of more than twenty years and were selected on the advice of the Royal Society.”

Can you tell me who at the Royal Society was responsible for providing this advice and what their criteria were for selecting these particular 11 papers?

I have not yet received an acknowledgment.

See Andrew Montford on this as well/.

The Oxburgh “report” on proxy reconstructions is about 1.5 pages long and doesn’t take long to parse – which I’ll begin below. No one should construe the fact that I’m commenting on these nine paragraphs as endorsing the idea that nine paragraphs – especially these none paragraphs – constitute a thorough review of CRU proxy reconstructions. None of the issues raised in my submission to the Parliamentary Committee (see here) was considered.

1. Tree growth is sensitive to very many factors including climate. By piecing together growth records from different trees, living or dead, it is possible to determine the temporal variation of growth patterns going back many hundreds of years. The dendroclimatological work at CRU seeks to go beyond this and to extract from the dated growth patterns the local and regional history of temperature variations. The Unit does virtually no primary data acquisition but has used data from published archives and has collaborated with people who have collected data.

No particular objections to this paragraph. The Climategate letters also show that CRU directly funded the collection of some data (e.g. Yamal).

2. The main effort of the dendroclimalogists at CRU is in developing ways to extract climate information from networks of tree ring data. The data sets are large and are influenced by many factors of which temperature is only one. This means that the effects of long term temperature variations are masked by other more dominant short term influences and have to be extracted by statistical techniques. The Unit approaches this task with an independent mindset and awareness of the interplay of biological and physical processes underlying the signals that they are trying to detect.

Obviously I agree that the data sets are “large”, “influenced by many factors of which temperature is only one” and that the “effects of long term temperature variations are masked by other more dominant short term influences”. These sound like Climate Audit points – I question whether these caveats have been as clearly disclosed in IPCC reports as they have been here. They say that the “effects of long term temperature variations … have to be extracted by statistical techniques” – I’d phrase this a little differently and more conditionally. If the effects can be extracted, then they have to be extracted statistically. Surely the efficacy of the statistical techniques needs to be demonstrated. I’m not saying that they were unaware of this nuance.

3. Although inappropriate statistical tools with the potential for producing misleading results have been used by some other groups, presumably by accident rather than design, in the CRU papers that we examined we did not come across any inappropriate usage although the methods they used may not have been the best for the purpose. It is not clear, however, that better methods would have produced significantly different results. The published work also contains many cautions about the limitations of the data and their interpretation.

It’s nice that they concede that “inappropriate statistical tools with the potential for producing misleading results have been used.” Unfortunately, they don’t provide a citation, merely attributing the error to “some other groups”. Surely a panel of esteemed academics can provide citations rather than using Gavin-esque evasions like “some other groups”. Hand’s comments indicate that he’s referring to Mann – who, as everyone except Oxburgh seems to know, is one of the most attested correspondents in the Climategate letters.

They say that they didn’t “come across any inappropriate usage”. But then again they didn’t actually interview any CRU critics or targets. Obvious examples of “inappropriate” statistical usage include the use of calibration-period residuals to estimate confidence intervals, the “Briffa bodge” in adjusting Tornetrask data and of course the trick to hide the decline – both in its IPCC form (deletion of data past 1960), its WMO form (splicing instrumental and proxy data) and its Rutherford 2005-Mann 2008 form (substituting “infilled” data for divergent data.)

They say that “it is not clear that better methods would have produced significant;y different results”. They took no evidence on this. This has been contested in many blog posts and was specifically contested in my submissions to the Parliamentary Committee and Muir Russell Committee – a readily accessible submission that Oxburgh ignored.

4. Chronologies (transposed composites of raw tree data) are always work in progress. They are subject to change when additional trees are added; new ways of data cleaning may arise (e.g. homogeneity adjustments), new measurement methods are used (e.g. of measuring ring density), new statistical methods for treating the data may be developed (e.g. new ways of allowing for biological growth trends).

These caveats are hardly ones that would offend a CA reader. The question is whether these caveats are properly disclosed in IPCC reports.

5. This is illustrated by the way CRU check chronologies against each other; this has led to corrections in chronologies produced by others. CRU is to be commended for continuously updating and reinterpreting their earlier chronologies.

One of the most serious criticisms of CRU – made both at CA and in my two UK submissions – has been CRU’s failure to update the Polar Urals chronology of Briffa et al 1995 (not on the short review list.) The Polar Urals Update resulted in a ring width chronology with an elevated MWP. This was never reported in the literature and this failure sticks in my craw. CRU deserves no “commendation” for their non-reporting of the impact of additional Polar Urals data on the chronology. That Oxburgh would “commend” them on this shows ignorance. But of course, they didn’t take any evidence from critics.

6. With very noisy data sets a great deal of judgement has to be used. Decisions have to be made on whether to omit pieces of data that appear to be aberrant. These are all matters of experience and judgement. The potential for misleading results arising from selection bias is very great in this area. It is regrettable that so few professional statisticians have been involved in this work because it is fundamentally statistical. Under such circumstances there must be an obligation on researchers to document the judgemental decisions they have made so that the work can in principle be replicated by others.

Much of this paragraph reads like Climate Audit. “The potential for misleading results arising from selection bias is very great in this area”. Couldn’t agree more. The work is “fundamentally statistical” – something that I’ve said for several years. “There must be an obligation on researchers to document the judgemental decisions they have made” – yup.

7. CRU accepts with hindsight that they should have devoted more attention in the past to archiving data and algorithms and recording exactly what they did. At the time the work was done, they had no idea that these data would assume the importance they have today and that the Unit would have to answer detailed inquiries on earlier work. CRU and, we are told, the tree ring community generally, are now adopting a much more rigorous approach to the archiving of chronologies and computer code. The difficulty in releasing program code is that to be understood by anyone else it needs time-consuming work on documentation, and this has not been a top priority.

Surely Oxburgh is straying into Muir Russell territory here. Weren’t they asked to assess the science rather than the conduct? Most of this paragraph is an excuse for not doing things. What does this have to do with assessing bodging, cherry picking and their like?

8. After reading publications and interviewing the senior staff of CRU in depth, we are satisfied that the CRU tree-ring work has been carried out with integrity, and that allegations of deliberate misrepresentation and unjustified selection of data are not valid. In the event CRU scientists were able to give convincing answers to our detailed questions about data choice, data handling and statistical methodology. The Unit freely admits that many data analyses they made in the past are superseded and they would not do things that way today.

At CA, I’ve criticized CRU for “unjustified selection of data” while refraining from speculating on their motives. Blog policies require readers not to speculate on motives. “Unjustified selection of data” is a different issue than “deliberate misrepresentation”. CA readers and myself have generally been concerned about confirmation bias and unintentional or negligent misrepresentation than “deliberate misrepresentation”.

Oxburgh says that CRU gave “convincing answers to our detailed questions about data choice, data handling and statistical methodology.” Unfortunately, Oxburgh did not follow the advice of the Parliamentary Committee to do things in the light of day. As a result, none of us knows what the “detailed questions” were or what the CRU answers were.

“The Unit freely admits that many data analyses they made in the past are superseded and they would not do things that way today.” I am unaware of any such admissions on the public record. I, for one, would be interested in knowing which data analyses are involved and precisely which methods the Unit have abandoned.

9. We have not exhaustively reviewed the external criticism of the dendroclimatological work, but it seems that some of these criticisms show a rather selective and uncharitable approach to information made available by CRU. They seem also to reflect a lack of awareness of the ongoing and dynamic nature of chronologies, and of the difficult circumstances under which university research is sometimes conducted. Funding and labour pressures and the need to publish have meant that pressing ahead with new work has been at the expense of what was regarded as non-essential record keeping. From our perspective it seems that the CRU sins were of omission rather than commission. Although we deplore the tone of much of the criticism that has been directed at CRU, we believe that this questioning of the methods and data used in dendroclimatology will ultimately have a beneficial effect and improve working practices

Again Gavin-esque evasiveness – “some of these criticisms”. OK, cite two or three. Are they throwing spitballs at Climate Audit here? Or are they criticizing someone else? If they are criticizing Climate Audit, I’d like to know what they regard as “selective and uncharitable” criticism. If they aren’t criticizing Climate Audit, who are they criticizing? As to a supposed “unawareness of the difficult circumstances under which university research is sometimes conducted”, of their “funding and labour pressures” and the “need to publish”, well, boo hoo. These studies have been cited in documents being presented to the public to justify major public expenditures and policies.

They “deplore the tone of much of the criticism that has been directed at CRU” – again without providing any examples. Hypocritically, they ignore the “tone” of remarks within the Climategate letters, the language that has appalled the public. If they are going to complain about the mote in their brother’s eye, they should first remove the beam in their own.

The Oxburgh report ” is a flimsy and embarrassing 5-pages.

They did not interview me (nor, to my knowledge, any other CRU critics or targets). The committee was announced on March 22 and their “report” is dated April 12 – three weeks end to end – less time than even the Parliamentary Committee. They took no evidence. Their list of references is 11 CRU papers, five on tree rings, six on CRUTEM. Notably missing from the “sample” are their 1000-year reconstructions: Jones et al 1998, Mann and Jones 2003, Jones and Mann 2004, etc.)

They did not discuss specifically discuss or report on any of the incidents of arbitrary adjustment (“bodging”), cherry picking and deletion of adverse data, mentioned in my submissions to the Science and Technology Committee and the Muir Russell Committee. I’ll report on these issues later in the day as they’ll take a little time to review. First, let’s observe Oxburgh’s trick to hide the “trick”. Continue reading →