The Oxburgh “Report” states that the eleven CRU publications that they examined are “representative”, “were selected on the advice of the Royal Society” and that CRU agreed that they are a “fair sample” of CRU’s work.
The eleven representative publications that the Panel considered in detail are listed in Appendix B. The papers cover a period of more than twenty years and were selected on the advice of the Royal Society. All had been published in international scientific journals and had been through a process of peer review. CRU agreed that they were a fair sample of the work of the Unit.
A “fair sample”?
To the extent that Climate Audit posts are a yardstick for CRU articles/documents that are at issue, the Oxburgh “fair sample” is almost mutually exclusive of the CRU articles discussed here. Nor were the Oxburgh articles at issue in, for example, my submission to the Parliamentary Committee and the Muir Russell “Team”.
Aside from CRU activities at IPCC (the sections in AR3, AR4 and AR4 Review Comments), the most prominent CRU articles criticized here are the following nine: Briffa et al 1992 (the Tornetrask chronology and “Briffa bodge”); Briffa et al 1995 (Polar Urals), Briffa 2000 (passim introduction of Yamal, Taimyr); Briffa et al 2002 (the famous cargo cult “assumption”); Mann and Jones 2003; Jones and Mann 2004; Osborn and Briffa 20056; Rutherford et al 2005; Jones et al (1990) on UHI.[Oct 27: also obviously Jones et al 1998].
The Oxburgh Eleven includes five tree ring articles (Briffa et al, Nature, 1998; Briffa et al, Roy Soc Lond 1998; Briffa 2000; Briffa et al 2001; Briffa et al 2008). Four of these articles were noted relatively favorably in my May 2005 post in which I first drew attention to the “trick” A Strange Truncation of the Briffa MXD Series (see image below). I had no complaint with the original Briffa articles – it was the IPCC spaghetti graph with its false rhetorical effect that bothered me. Even for veteran watchers of peas under thimbles, it’s pretty amazing that four CRU articles – that not only had had not been the subject of criticism, but had been used to locate the trick – were chosen as somehow “representative” of the CRU corpus, while the articles that had actually been criticized here were for the most part excluded. Every CRU hockey stick article (Jones et al 1998; Mann and Jones 2003; Osborn and Briffa 2006) was excluded.
So, in addition to not taking any evidence from CRU critics or targets, the terms of reference for the execrable Oxburgh “inquiry” diverted its attention away from articles that actually were at issue to other articles that had actually been used as source material to identify the trick in the first place.
Excerpt from IPCC 2001 “trick” spaghetti graph.
This raises a few obvious questions. The Oxburgh Report states that the eleven articles were “selected on the advice of the Royal Society”. However, they did not provide any information on how the Royal Society determined that these eleven publications were “representative”. Nor did they mention who at the Royal Society actually made the selection. The report says that UEA agreed that the Royal Society selection was a “fair sample”. I wonder who at UEA actually agreed that the selection was a “fair sample” and what their criteria were.
One of the recommendations of every inquiry so far is that methodologies be properly disclosed. Oxburgh didn’t disclose how they selected their supposedly “representative” and “fair sample”. “Fair sample” and “representative” are statistical terms – terms were used in a report coauthored by a very senior professional statistician in a context where statistics are very much at issue. So I presume that the Royal Society took some care to ensure that the eleven publications actually were “representative” and a “fair sample” – and not ones that were pre-selected by UEA, rather than the Royal Society.
I’ve written to both UEA and the Royal Society seeking clarification on the selection process and will keep readers posted.
Update Apr 16: Here is my request to the UEA under the Environmental Information Regulations:
Dear Mr Palmer,
Pursuant to the Environmental Information Regulations, I hereby request correspondence between the University of East Anglia and/or its officers and the Royal Society between December 1, 2009 and April 12, 2010 concerning the selection of publications considered in the Oxburgh “report”. Thank you for your attention.
Regards, Stephen McIntyre
I received acknowledgment of this request this morning.
I sent the following inquiry to the Royal Society yesterday:
The Oxburgh Report lists 11 CRU papers in Appendix B stating: “The eleven representative publications that the Panel considered in detail are listed in Appendix B. The papers cover a period of more than twenty years and were selected on the advice of the Royal Society.”
Can you tell me who at the Royal Society was responsible for providing this advice and what their criteria were for selecting these particular 11 papers?
I have not yet received an acknowledgment.
See Andrew Montford on this as well/.
Climate Audit 
73 Comments
Seems like a trick to hide the trick. Select some uncriticized papers that did not trigger the inquiry in the first place. Report the happy news that they seem good. This may even be true in a narrow sense, though a trick, since an irrelevant finding.
I have asked the question on the BBC’s Richard Black blog:
“
Could these be the people who selected the papers for the panel to review:
http://royalsociety.org/The-science-of-climate-change/
Does anyone know?
“
and specifically… how was Phil Jones involved in the selection process?
Well we have known for a long time that dendros have a unique interpretation of the words ‘random sample’. It looks like UEA and the Royal Society are simply following the well established standards of the dendro community.
Mispelling. It’s ‘reprehensive’ and ‘far sample’.
============
Steve: the first link does not take you to the “Oxburgh ‘Report.'” It’s a memorandum from Acton.
So, after all your experience, do you now go out and take a long walk before writing such temperate criticism, Steve? No sarcasm meant. The criticism is civil and restrained.
Admirably so, I do add.
Were decisions made by either CRU or the Royal Statisitcal Society or both
as to which CRU publications were not fair samples or representative
examples of East Anglia/CRU output ?
One can envision a listing of publications with either CRU or the Royal Society allowed
to arbitrarily strike out the publications to not be considered as “representative”.
The “sides” then come to an agreement on what reports will finally be selected.
This process would then be a standard precurser format to arbitration, not investigation.
“The potential for misleading results arising from selection bias is very great” in such slap-dash inquiries.
Oxburgh hung by his own petard.
A petard is a bomb–you are hoisted (blown into the air) by a petard, not hung. Poor Shakespeare…
Craig: they use to hang them on doors and walls to blow holes in them. Shakespeare is using it as a metaphor to mean that you are destroyed by the devices you meant to use on others.
This argument is petarded.
Steve
This matter must be a suitable subject for a formal peer reviewed paper, but perhaps in a leading sociological or social phychology journal, rather than in Science or Nature.
It should just be a straight report of the facts, with no conclusions drawn.
Just a statement of how criticisms are answered in climatology.
BUT this must be widely diseminated.
Are there any UK mass media newspapers interested,
or is everybody just content to be led down the slippery slope to a carbon taxed, impoverished future?
I would have thought, on the basis of his comments, that Professor Hood would be very concerned if he knew his credibility was being used in a “cherry picked” study. It would be very interesting to hear his views on the issue of whether was an accurate representation of the work that was at issue.
Sorry I meant Professor Hand in the post above
Further to my earlier post – is there a mainstram journalist reading all this?
Stsve has done all the hard work.
It does not need a Bernstein(?) & Woodward to investigate.
It just need an every day good journalist and an upright editor to write it up.
BingO!
Steve – please do take heart.
After reading the comments to your earlier posts, I see that you have received a much larger dose of criticism by bloggers, than is normal at CA.
You have touched on a VERY raw nerve.
Throughout the world there is an orchestrated attempt at whitewash over climategate.
The attacks on you and you analysis just show how much you have hit the bulls eye and how strong their determination to kill the truth.
The problem for them is that, in the long run, the truth is stronger still.
Please do keep it up.
Agreed. You’re doing a great job Steve. Thank you.
Flak is heaviest when you’re over the target.
Perhaps not a fair sample but a sample of fair papers.
Yes, Minister!
Barclay, if its not too much trouble, would you please spell the clan name the correct way?
Thanks! But my response would be OT:)
It’s been a case of cherry pickers for the defence representing client cherry pickers.
Please show some respect, the impartial Royal Society are ideally placed to provide objective advice on this issue;
From their Copenhagen climate conference press release
http://royalsociety.org/Preventing-dangerous-climate-change/
“It is certain that GHG emissions from the burning of
fossil fuels and from land use change lead to a warming of climate, and it is very likely that these GHGs are the dominant cause of the global warming that has been taking place over the last 50 years. ”
No principled positions to compromise their advice.
The Devil is in the detail:) Thanks, Steve, for keeping the details in focus.
Kind Regards
Michael
I find this all sickening. I’d hoped that Climategate et al would lead to a cleansing of the stables, though I also opined (somewhere) on the basis of my public service experience that those responsible for distortions and fear-mongering would not suffer serious consequences if and when exposed. Yesterday I taught a group of Queensland Public Servants studying for a Master of Public Policy degree, most fairly senior, all dedicated, all concerned at the appalling standards of departmental heads and government ministers. As with AGW, the insiders who gain mutual benefits from sticking together and controlling processes can withstand what should be damaging revelations by outsiders.
It should be noted that about two thirds of the income of the Royal Society of London nowadays comes directly from the UK parliament – therefore, in effect, the Royal Society is currently a branch of the UK Civil Service.
http://www.publications.parliament.uk/pa/cm200102/cmselect/cmsctech/774/77404.htm
Paragraph 10. “Parliamentary grant-in-aid, accounts for two thirds of the Royal Society’s income. The Royal Society estimated that its total expenditure in 2001-02 would be £36.184 million of which £10.2 million was private funding.[9] The Royal Society could make more effort to attract private funding from appropriate sources. We urge the Royal Society to investigate more streams of funding, using its prestige and grant-in-aid as leverage.”
Steve: For the purposes of FOI, it does not appear to be a branch of the civil service. It is sort of off-balance sheet.
72 per cent, in fact, nearer three-quarters than two-thirds. (Well, this is a statistics-dominated blog.)
I had not realized that things were so bad – could you provide a reference for the 72 percent please?
The point is that the Royal Society is, in many circles, regarded as the most prestigious (because oldest, original) scientific society in the world. And it used to be fearlessly independent of government.
Nowadays the Royal Society is a fundamentally dishonest organization; based on a pretense of independence, and trading on its past reputation.
If you use a random selection process and CRU has thousands of pubs, you will for sure not select the papers that have the problems CA has found. Random = “Fair”? but I doubt it was even that scientific.
The following update was added to the above post.
Here is my request to the UEA under the Environmental Information Regulations:
I received acknowledgment of this request this morning.
I sent the following inquiry to the Royal Society yesterday:
I have not yet received an acknowledgment.
See Andrew Montford on this as well/.
Steve:
Does it make any sense to copy the members of the committee, particularly Prof. Hand, on your concerns and your FOI requests?
Andrew Montford has an interesting update. See http://bishophill.squarespace.com/blog/2010/4/16/actons-eleven-the-response.html
Time for Dr. Wegman to check the membership in this social club. Their focus is so narrow they would not see a freight train if it was on top of them.
Good work Steve!
‘Fair’ contrasts with ‘biased’ in statistics, and other usages that imply reasoning and justification. In regard to temperature, however, most people would take ‘fair’ to mean not too low, not too high, just like Baby Bear’s porridge. Perhaps Oxburgh means simply that his sample was not too small for his purpose and not too large either. It’s another matter whether that sample size makes for a statistically significant conclusion.
The correct interpretation, but the wrong bear.
Steve:
Andrew has some new news. He spoke with someone at the RS and it appears that there is a further twist to the tale!!
See: http://bishophill.squarespace.com/blog/2010/4/16/actons-eleven-the-response.html
The Royal Society at least is beginning to respond to the queries and they may be taking exception to the misuse of their name to give the appearance of weight to Oxbrough report.
Well, looks like UEA investigation was based on material suggested by UEA.
What could be wrong with that ? Duh…
yep, Oxburgh’s claim that the eleven papers were “selected on the advice of the Royal Society” appears to be untrue. It appears that the Royal Society had nothing to do with identifying or selecting these eleven papers. My, my. I wonder what Kerry Emanuel has to say about this.
Steve —
Montford’s inquiry has knocked a few apples out of the cart, but your query was more direct, so we still await an answer to it.
Perhaps if David Hand, President of the RSS, is also a member of the RS, he could be prevailed upon to urge the RS to expedite a direct answer to your question?
Re: Hu McCulloch (Apr 16 13:03), Hand is an FBA not an FRS.
Imagine that someone doing cloning experiments is accused of making up data, or someone is accused of plaigarism, or if anyone remembers the David Baltimore scandal, and an investigation is called, and they investigate OTHER WORK and say it looks ok more or less. Really? When has that ever happened before?
Or how about the investigation of Bernie Madoff, and he gets to bring in only witnesses from before he got caught, who all made money?
horrible examples Craig.
The Panel was set up by the University in consultation with the Royal Society
to assess the integrity of the research published by the Climatic Research Unit
in the light of various external assertions.
it had a pretty wide scope.
perhaps Steve should had simply made it more public, that he thinks that the majority of the articles published by the CRU are absolutely fine. without the witch hunt, they might have been able to focus on the important articles.
Madoff didn t invest at all. a look at his latest deals would have blown up his scheme. try another example Craig.
Steve: I sent submissions to the Parliamentary and Muir Russell inquiries identifying major issues. Oxburgh made no attempt to obtain input. That’s their fault, not mine. I have no opinion on the majority of CRU articles, but the dendro articles considered by Oxburgh were ones that enabled me to identify the trick in the first place. Much, if not most, of the criticism of CRU here arose out of their actions at IPCC , which Oxburgh conveniently neglected to report on – although they did reject the UEA “explanation” that the trick was a “good” way to do something. They said that it was “regrettable” “neglect” – the emails indicate that the omission of data was not because of “neglect” however.
Typically in a Ponzi scheme, early investors get their money back plus a handsome return, which induces other, at first more wary, investors to put their money in as well. So although not all the investors make money, it is ordinarily not hard to find some who did. If these are the only ones who are called to testify, the jury could easily be persuaded that no harm was done.
So I think it’s a valid analogy. However, Craig’s first example, of defending against charges fabricated data or plagiarism by pointing to selected works that do not have these problems, is even more to the point.
I wonder what the odds are of Sean Connery volunteering “Highlander II: The Quickening” to a panel reviewing his career by examination of a subset?
How about the 1970s Zardoz?
Steve: I sent submissions to the Parliamentary and Muir Russell inquiries identifying major issues. Oxburgh made no attempt to obtain input. That’s their fault, not mine.
your input. that is pretty legitimate, and most likely not a fault.
my quote above is from the report. they did take a look at the big picture. CRU research is fine. (in stark contrast to what the sceptics blogs and media claimed)
Typically in a Ponzi scheme, early investors get their money back plus a handsome return, which induces other, at first more wary, investors to put their money in as well.
i know what a Ponzi scheme is. but an investigation wouldn t just look at whether other people got paid. the equivalent in checking scientific papers, would be, whether there were papers retracted for integrity reasons before.
the cloning example is completely wrong as well. in contrast what you are trying to tell us, the attacks on the CRU unit were extremely broad. destroying data, faked code, exaggerating warming. such broad misconduct would show in a sample of their work. but it does NOT.
Because they looked at the wrong papers, Sod. Cannot you read what Steve has said and understand it?
sod, the fact that none of the articles pointed out by “skeptics” was used by the inquiry doesn’t bother you at all ?
I mean you roughly have two sides : “skeptics” that say there is bad science going on and pointing to some very specific papers, and “warmists” saying all is fine.
Now wouldn’t the first action of an inquiry be to take a look at the papers the skeptics are pointing at ?
I mean if someone tells you there is a dead body in the bedroom, do you only look in the kitchen then leave the house and say all is fine ? I’m not saying that looking in the kitchen won’t be interesting, I’m just saying you maybe should not start by and limit yourself to the kitchen…
“they did take a look at the big picture.”
No they didn’t : “The Panel worked by examining representative publications by
members of the Unit […]The eleven representative publications that the Panel considered in detail are listed in Appendix B.”
The misconduct by Enron was related to off-balance sheet entities that allowed Enron to produce misleading profit and loss statements. The analogy here would be that the auditors, who were charged with investigating the off balance sheet transactions, instead chose to only look at the on-balance sheet transactions and based on those transactions gave Enron a clean bill of health.
Enron executives would have loved to have an Oxburgh type of “independent” inquiry. They would have spent a lot less time away for home in recent years.
Sod,
If we accused them of plagarism in papers A, B and C, and the commission looked into the matter by examining papers X, Y and Z,
EVEN IF papers X, Y and Z were selected randomly, EVEN IF X Y and Z were a fair sample, would that not strike you as a bit odd. They needed to look at the offending work. That they didnt should upset everyone who believes in AGW as I do.
I see your dilemna. If they had looked at papers A, B and C and said “no problems found” that would have been affirmation for AGW. That they looked at X, Y and Z, then nothing they said would disprove any of the allegations against A, B and C. At best they would find what they found – “No problems”. But that still leaves open the possibility that there are problems with A, B and C which have not been addressed or answered.
That is the 2nd worst outcome for CRU that anyone could have forseen. The only thing worse would have been if they had found problems in the papers they did look at (when apparently no one else was questioning them).
Sod,
On what basis did you make the claim that “they did take a look at the big picture?”
What evidence that we have been able to glean from the public disclosures made to date indicate to most people that they did they did anything but look at the big picture.
Perhaps, you were present during Oxburgh’s investigation. If so, please illuminate us as to how they conducted a big picture investigation without looking at the issues which Steve and others have been raising for years?
Can these “correspondences” be obtained through the FOIA?
Environmental Information Regulations is like a FOIA
Clearly the UEA(CRU)Standard Operating Procedure(SOP) of selecting data that proves the hypothesis is quite widespread and could be considered a creed for all who believe in AGW. Otherwise this behaviour of the panel beats my dumb mind. This panel was criticised from the word go and we were all alerted to the likely outcome. Now it has all come to pass!
Quoting Sir Humphrey:
“If people don’t know what you’re doing, they don’t know what you’re doing wrong.”
> Steve McIntyre
> Posted Apr 16, 2010 at 12:35 PM | Permalink | Reply
> yep, Oxburgh’s claim that the eleven papers were “selected on the advice of > the Royal Society” appears to be untrue. It appears that the Royal Society
> had nothing to do with identifying or selecting these eleven papers.
Steve,
I’m not sure this is true.
From the report…
“and suggested that the review begin by looking at key publications, which were chosen to cover a broad range of subjects over a wide timescale. ”
..implies that the RS might have selected a few ‘key publications’ and presented them to the panel for review.
These could have been accepted without investigation into their selection.
Nial
“The Panel was set up by the University in consultation with the Royal Society to assess the integrity of the research published by the Climatic Research Unit in the light of various external assertions.” Unfortunately, the UEA may not feel that “external assertions” posted at ClimateAudit are worthy of investigation. With 20/20 hindsight, the main “external assertions” concerning the UEA are those arising from Climategate that have appeared in major British newspapers or discussed in mainstream scientific publications. If you ever have another opportunity to submit allegations, you might consider featuring references to such sources, supplemented with additional information at ClimateAudit . (Even a citation to RealClimate would be evidence that a public controversy exists.) It is grossly unfair that sources with inferior accuracy and detail should be given prominence, but it should be clear by now that the establishment sees itself in a political battle for public opinion about AGW, not scientific integrity. From their perspective, readers of ClimateAudit are a lost cause.
It is possible that some or all of these papers were selected by a citation search for the most referenced papers produced by CRU during various periods of time. However, work with the IPCC certainly represented the highest impact contribution from CRU.
Interestingly, two of the papers reviewed concerned the Wang allegations and a change in the estimated impact of UHI’s. It would have been interesting to hear Jones explain how he got one answer in 1990 and another in 2008.
Steve, thanks for posting the nice “spaghetti” graph that hides the decline! Would you also post the exploded view that pulls this apart, to show how badly they misrepresent the data with this technique?
Here, this one is historic:
I’d love to see the graph pulled apart, I’ve seen that graphic before! Cheers!
To what misinformation is SMc dedidcated?
just another drive-by smearing. pay it no mind.
Wow. It just keeps getting worse.
Booker:
http://www.telegraph.co.uk/comment/columnists/christopherbooker/7601929/Climategate-a-scandal-that-wont-go-away.html
Could it be the primary objection to the Oxburgh report is that the folks here don’t agree with its conclusions? I doubt it would be subject to this level of scrutiny had it concluded that Phil Jones et alia were indeed being deliberately misleading/fraudulent/nefarious.
Given the Royal Society’s involvement in this debacle, how ironic that its motto is Nullius in Verba, Take no ones word.
Steve says:
“Every CRU hockey stick article (Jones et al 1998; Mann and Jones 2003; Osborn and Briffa 2006) was excluded.”
Thus proving that the sample was in fact UNrepresentative. Argument over. (Whose idea was it to choose a “sample” anyways?)
The claim in the Oxburgh report that the eleven papers were “selected” by the Royal Society is now known to be untrue.
Unfortunately, neither the Royal Society nor the Oxburgh panel have seen fit to correct their misrepresentation.
One of the things I like about the blogosphere is the culture of correction (exemplified perhaps by Wiki). If an allegation of “misrpresentation” sticks (and many don’t) honorable authors correct, and note the correction. Ancient institutions that do not share this culture of correction are the dinosaurs of the information age. Long live the blogospheric culture of accountability as practised by Steve M.
9 Trackbacks
[…] A “Fair Sample”? […]
[…] Al menos tres investigadores del equipo han revisado las publicaciones más representativas (once artículos) y después les han preguntado al respecto. Tampoco es que tengan que ser drogados con el suero de la verdad o sometidos al polígrafo, y a mí no se me ocurre otra manera mejor. No obstante, hubiese preferido que hubiesen hecho una revisión más a fondo y que hubiesen trabajado, al menos, más de 15 días por persona. A Steve McIntyre también le parece una muestra de publicaciones manifiestamente mejorable. […]
[…] I notice – pure co-incidence no doubt – that these are exactly the ones McI choses as “Every CRU hockey stick article “. Well all right: this isn’t co-incidence. […]
[…] looking into possible research misconduct at the Climate Research Unit of East Anglia University got it all wrong. So, in addition to not taking any evidence from CRU critics or targets, the terms of reference for […]
[…] panel because Oxburgh ‘had not looked at the right papers’ and [McIntyre] came up with 5 papers they hadn’t looked at. But none of these papers were highlighted (or even mentioned) in McIntyre’s submission to […]
[…] le souligne aussi Steve McIntyre, pratiquement aucun des 11 travaux sélectionnés ne figure parmi les travaux vivement critiqués, […]
[…] to provide as little damning evidence as possible – none of the papers reviewed were those that skeptics have issues with. We already knew that NOT ONE SKEPTIC of note was interviewed to discuss possible issues to be […]
[…] should have been on the list. On April 15, 2010, the day after the release of the Oxburgh Report, I reacted immediately to their stupid list with a list of papers that had been at issue as follows: Aside […]
[…] panel because Oxburgh ‘had not looked at the right papers’ and [McIntyre] came up with 5 papers they hadn’t looked at. But none of these papers were highlighted (or even mentioned) in McIntyre’s submission to […]