My fear is that by keeping the public focussed on irrelevancies, you are excluding them from the discussion of what we should do about climate change

But it’s not Bishop Hill that Myles Allen should be criticizing; it’s John Houghton who more or less made the Hockey Stick the logo of the IPCC. Mann was told that IPCC higher-ups wanted a visual that didn’t “dilute the message” and they got one: they deleted the last part of the Briffa reconstruction – Hide the Decline. If, as Allen now says, it’s an “irrelevancy”, then Houghton and IPCC should not have used it so prominently. And they should not have encouraged or condoned sharp practice like Hide the Decline.

In the run-up to AR4, I suggested that, if the topic was “irrelevant”, as some climate scientists have said, then IPCC should exclude it from the then AR4. Far from trying to keep the topic alive in AR4, I suggested that it be deleted altogether. I guess that there was a “consensus” otherwise. If Allen wants to complain, then he should first criticize IPCC.

Bishop Hill links to a presentation by Myles Allen to a 2011 conference on Climategate, which like every other such handwringing introspection by climate “communicators”, notably failed to invite any of the major CRU critics – people who might actually have given them some insight into Climategate. In his presentation to climate communicators, Allen gave his own version of Hide the Decline. Allen showed the graphic below, sneering that the entire effect of Climategate was 0.02 deg C in the 1870s.

Needless to say, Allen’s graph has nothing to do with Hide the Decline and the Climategate dossier. Allen’s graph shows the CRUTEM temperature index from 1850, not the 1000 year reconstructions in which Hide the Decline occurred. CRUTEM was only mentioned a couple of times in the Climategate dossier. Climategate was about the Hockey Stick, though this point was misunderstood by Sarah Palin and now, it seems, Myles Allen.

Figure 1. Allen in front of temperature history.

In contrast, here’s a graphic from Richard Muller’s 2011 lecture. Unlike Allen, Muller understood Hide the Decline, which is shown here in one of its manifestations. (This is the WMO graphic; the more important Hide the Decline was in the IPCC Third and Fourth Assessment reports.) Hide the Decline is not 0.02 deg C in the 1870s; it was Briffa, Mann and Jones deleting the inconvenient portion of the Briffa reconstruction after 1960. And it wasn’t a microscopic difference. This difference is large enough that it might well have “diluted the message” that Houghton and others wanted to convey.

Figure 2. Muller in front of WMO hockey stick – a 1000 year reconstruction. Left – Hide the Decline; right – actual data.

While one would hope and expect that Myles Allen would have had a better factual grasp on Climategate issues than Sarah Palin, it seems that we’ve been disappointed.

Allen’s decision to show temperature data rather than Hockey Stick reconstructions cleverly draws attention away from the problems of those reconstructions. The Climategate emails have a apt phrase for Allen’s technique. Showing an unrelated dispute about a temperature graphic rather than the decreasing Briffa reconstruction is itself just another …. trick to hide the decline.

Update: Lucia responded to Myles ALlen in the comments as follows:

[Myles Allen said]

I appreciate that people like yourself who have devoted a lot of time to the analysis of paleoclimate data find it irritating when scientists who don’t work in that area dismiss it as uninformative.

First: communication tip: You need to learn to post complete thoughts. Uninformative about what? Everything? Climategate? Or the thermometer record? Or the strength of evidence for AGW? Depending on how I read your mind, you may be saying something true or utterly false. If you are going to lecture people on communicating science you might want to stop making readers guess which you mean.

Second: It seems to me you are misunderstanding what SteveMc writes. He’s not saying he is irritated that someone thinks paleo data is uninformative. He is saying that you suggest the “whole affair” (i.e. climategate) is an argument about the thermometer record. The fact is: climategate is not merely or even mostly about the thermometer record.

And I stand by the assertion that, thanks to the sloppy coverage the affair received in the media, it wasn’t just Sarah Palin who got the impression that the instrumental temperature record was seriously compromised

I would suggest that the main reason for this “sloppy coverage” was that reporters turned to people trying to rebut those discussing climategate at blogs and in forums. Some people people who (like you) might prefer to discuss the thermometer record rather than misbehavior of scientists or what “hide the decline” meant, diverted the discussion to the thermomeber record.

I strongly suspect the behavior of the scientists who wanted to suppress discussion of climategate succeeded in giving the media the incorrect impression that climategate was about the thermometer record is one of the reasons much of the media, some politicians, and Sarah Palin developed the impression climategate is about the thermometer record. That you can show they were confused about what people at blogs and forums were posting about merely shows you don’t know what it was about.

I would also suggest the only thing that can come of you continuing to try to convince people it was about the thermometer records is for people to explain that which you do not wish to be discussed: The Hockey Stick, misbehavior or scientists and the various whitewash investigations.

OTOH: If you simply wish to communicate that the topics that are central to climategate are not important to our understanding of climate change- that would be fine. But if you wish to make the case that the hockey stick doesn’t matter, then you need to make that clearly. Unfortunately for you, clear exposition requires discussion of the hockey stick!

A proper exposition might be to
a) Discuss what the hockey stick “is” with a little history.(Accuracy would be useful here. Mention it was used as background at IPCC meetings, and in Gore’s talk.)
b) Discuss why this shape is not important to our understanding of climate change. Show versions with and without the decline– and explain why even if the decline exists we do believe the world is warming. Do this by
c) Explaining the thermometer record.

Don’t try to take the tack of inaccurately claiming that climategate is actually about the thermometer record. If you take that tack, you’ll find yourself trying to defend your position– downgrading much of what you seemed to present rather strongly as your opinion, and burying your arguments in favor of your opinion deep in comments at a blog. (I’d note: I think much of your argument amounts to “changing the subject”– but that’s another matter.)

Moreover, I would like to point out that unless say what paleo is uninformative about your claim that paleo is not important (at all) seems a bit thin. Climate blog addicts can easily see see that on May 26, 2012 you are chiding Bishop Hill for discussing the Hockey Stick and providing lengthy explanations of its lack of importance while Real Climate’s front page is simultaneously running a post on discussing Hockey Sticks (See
Fresh hockey sticks from the Southern Hemisphere, May 22). It’s quite likely some will suspect that your opinion that the hockey still is uninformative (about something you don’t quite spit out) is maybe not entirely correct.

Third: Returning to “first”. When I watched your talk, I was struck by your tendency toward vagueness. Based on what you write in your defense in comments, I learn that the allusion to “the data” at minute 2:37 likely meant “the thermometer record” and “impact of the whole affair” (i.e. climategate) must have meant “impact of portions of the climategate discussions that relate to the thermometer record”. Your talk is riddled with these types of vague ambiguities. The consequence is that– on the whole– what your talk appears to communicate is false. If the audience comes away thinking you are suggesting that climategate was not about the paleo records, and that you think the only impact of climategate is a small tweak on the thermometer record, then the fault for their misunderstanding you falls on you for communicating rather badly.

Next time you want to make a presentation telling reporters that they shouldn’t focus on the paleo record but rather the thermometer record, you might be wise not to try to turn that into a talk about how the media got climategate wrong. Try to bite off less– stick to just discussing the thermoter record and why you think it tells us that the world has warmed and it’s because of man.

If you want to discuss climategate and how scientists failed to communicate their position, you have a hard row to hoe. Much of the reason scientists communicated the issues in climategate badly is they didn’t want to talk about them. Scientists mistake was to respond to journalists by trying to change the subject; others with plenty of ink keep talking all the whining in the world isn’t going to get people to stop discussing the topic. You can keep trying to do that: it isn’t going to work any better in 2012 than it did from 2009-2011.

McIntyre got the erroneous idea that studies were being done, but were being suppressed if they showed something ‘inconvenient’. This is of course a classic conspiracy theory and one that can’t be easily disproved. Accusation: you did something and then hid it. Response: No I didn’t, take a look. Accusation: You just hid it somewhere else.

One aspect of Schmidt’s response is beyond laughable. I agree that the best way of disarming suspicion is to show data: “take a look”, as Schmidt says. However, if Schmidt thinks that the conduct of the scientists involved in the various data refusals, obstructions and FOI refusals constitutes “take a look”, then he’s seriously in tin foil country. Comical Gav indeed.

Although I find it hard to believe that Schmidt is unfamiliar with the past incidents that gave rise to suspicion that adverse results and data have been withheld or not reported, I’ll review a couple of important ones. These do not, in any sense, constitute an inventory of incidents. They are ones that are either familiar in part to CA readers or which illustrate an important aspect of the problem.

“Dirty Laundry” and Verification r2
In December 2003, despite a number of prior data refusals, I asked Mann for the residual series from the individual steps (termed “experiments”) in MBH98. Unknown to me at the time, Briffa and Osborn had made an almost identical request three months earlier (which Mann had complied with). Residual series permit a reader to carry out standard statistical tests (verification r2, RE, etc) without having to re-do the entire calculation from scratch. I copied David Verardo of NSF on the request. Without waiting for Mann’s refusal (this surprised me), Verardo said that Mann was not required to provide this data to me. Verardo’s letter was later cited in Mann’s evidence to the House Energy and Commerce Committee and in Stephen Schneider’s book.:

His research is published in the peer-reviewed literature which has passed muster with the editors of those journals and other scientists who have reviewed his manuscripts. You are free to your analysis of climate data and he is free to his. The passing of time and evolving new knowledge about Earth’s climate will eventually tell the full story of changing climate. I would expect that you would respect the views of the US NSF on the issue of data access and intellectual property for US investigators as articulated by me to you in my last message under the advisement of the US NSF’s Office of General Counsel.

In response to the identical inquiry from CRU, Mann immediately sent the residual series to Osborn, warning him that the residual series were his “dirty laundry”, provided to Osborn only because he was a “trusted colleague”. Mann asked Osborn to ensure that the “dirty laundry” didn’t fall into the wrong hands, an assurance that Osborn readily gave.

None of the so-called “inquiries” delved into why Mann regarded the residual series as his “dirty laundry” and why he was so anxious to prevent this (apparently “inconvenient”) information from falling into the wrong hands.

One reason might, of course, have been that the residual series would immediately permit the calculation of the verification r2 for each step. (Favorable) verification r2 results for the AD1820 step were illustrated in MBH98 Figure 3; elsewhere MBH98 said that the verification r2 statistic had been considered. But in the SI to MBH98, Mann had archived the RE for each step but not verification r2.

In MM2005, we reported that the verification r2 for the AD1400 step was approximately zero – a very surprising result given the “skill” claimed for MBH98. (Zorita, for one, was surprised by this result and thought less of MBH98 accordingly.) In MM2005 (EE), we expressed our surprise that the results of such a central verification statistic had not either not been calculated or reported.

In Mann’s testimony at the NAS panel in March 2006, Mann was directly asked whether he had calculated the verification r2 for the AD1400 step; Mann flatly denied doing the calculation, saying that such a calculation would have been a “foolish and incorrect thing to do”. However, by that time, Mann had archived part of his source code in response to the House Committee and that code showed conclusively that the verification r2 values had been calculated for all steps.

That Mann had calculated verification r2 results is beyond dispute. That they were “inconvenient” is beyond dispute. That they were not reported is beyond dispute.

Wahl and Ammann announced in May 2005 that all our claims were “unfounded”. Since our codes were very close and I reconciled them almost immediately, I knew that their verification r2 results would be identical to ours. Again, I was asked to review the paper (though my review was disregarded.) As a reviewer, I asked for the verification r2 results. Wahl and Ammann refused. Rather than rejecting the paper, Schneider terminated me as a reviewer. At AGU in December 2005, I asked Ammann what the verification r2 for their AD1400 step was. He refused to answer – a refusal noted by Eduardo Zorita and others.

I asked Ammann out to lunch after the paleo session (I bought). Since our codes reconciled, it should have been possible to clarify the dispute. I offered to jointly (with our coauthors) write a paper stating what we agreed on and what we disagreed on. He refused, saying that this would be “bad for his career”. To this day, I remain dismayed at this answer. I urged him to report the verification r2 results; he refused. I told him that I would not simply stand by while he refused to report the adverse verification r2 results that confirmed ours; he shrugged. I therefore filed an academic misconduct complaint at UCAR; while the complaint was shrugged off without investigation, the verification r2 results appeared in the final article, confirming our point.

The Climategate emails show that Phil Jones, also a reviewer of the paper, was outraged that we had complained about Wahl and Ammann suppressing the inconvenient data, not by them trying suppressing the data.

Jacoby and D’Arrigo
Another equally disquieting incident occurred before Climate Audit and may not be familiar to all readers. This incident also illuminates issues about when data is “used” – an issue that I believe to be relevant to the Yamal incident.

Jacoby and D’Arrigo (Clim Chg 1989), a study of northern North American tree rings, was extremely influential in expanding the application of tree rings to temperature reconstructions (as opposed to precipitation.) (See CA tag Jacoby for prior posts that have been tagged.) The Jacoby-d’Arrigo reconstruction was used in Jones et al 1998 and its components (especially Gaspe) were used in MBH98. It is used to “bodge” of Mann PC1 in MBH99; Mann’s “Milankowitch” argument rests almost entirely on this bodge – ably deconstructed by Jean S here.

Jacoby and D’Arrigo stated that they had selected the 10 most “temperature-influenced” sites from the 36 northern North America (boreal) sites that they had sampled in the previous decade, to which they added Cook’s (very HS) cedar series from Gaspe “because of the scarcity of data in the eastern region”. However, if you pick the 10 most “temperature-sensitive” series from a network of 36 autocorrelated red noise, you will get a HS. This phenomenon has been more or less independently reported by me, Jeff Id, Lucia, Lubos Motl and David Stockwell. We noted this phenomenon in our PNAS Comment on Mann et al 2008, taking some amusement in citing AIG News (Stockwell) since it was unreported in the Peer Reviewed Litchurchur. The phenomenon seems to baffle climate scientists.

Not only did Jacoby and d’Arrigo pick only 10 of 36 sites, they only archived these 10 sites. I asked for the data from D’Arrigo but got nowhere. At the same time, I had also learned that a new Gaspe version had been calculated – one which did not have a HS. (See CA here). I asked D’Arrigo to archive or provide me the data. She refused, saying that the version on file (cana036), which had a huge HS, was a better guide to NH temperature.

In early 2004, Climatic Change did not have a data policy when I was asked to review a submission by Mann et al savaging us. In my capacity of “reviewer”, I asked for the supporting data and code that Mann had previously refused. The late Stephen Schneider, then editor of Climatic Change, said that no one had ever previously asked for supporting data or code in the 28 years that he had edited the journal and that such a request would require a change in editorial policy. The progress of my request is documented in Climategate letters, since Phil Jones and Ben Santer were on the Climatic Change editorial board and both opposed the proposal. (Peter Gleick made his first cameo appearance at this time, also supporting obstruction.) Eventually Schneider adopted a policy requiring supporting data, but not code. Under the new policy, I asked for supporting data for the new submission, which Mann withdrew. (Osborn expresses some annoyance at this in a CG2 email.) This was the first academic paper that I had been asked to review. While I think that Mann knew that I was the reviewer, if I were doing it again, I would only act as an identified reviewer so that any adverse interest was clearly disclosed to the author.

With the benefit of Climatic Change’s newly minted data policy, I asked them to request the missing measurement data for the “other” 26 Jacoby sites. Jacoby refused in a truly remarkable letter, reported on in one of the very first CA posts here. The following is a lengthy excerpt, see the link for the full letter):

The inquiry is not asking for the data used in the paper (which is available), they are asking for the data that we did not use. We have received several requests of this sort and I guess it is time to provide a full explanation of our operating system to try to bring the question to closure…

We strive to develop and use the best data possible. The criteria are good common low and high-frequency variation, absence of evidence of disturbance (either observed at the site or in the data), and correspondence or correlation with local or regional temperature. If a chronology does not satisfy these criteria, we do not use it. The quality can be evaluated at various steps in the development process. As we are mission oriented, we do not waste time on further analyses if it is apparent that the resulting chronology would be of inferior quality.

If we get a good climatic story from a chronology, we write a paper using it. That is our funded mission. It does not make sense to expend efforts on marginal or poor data and it is a waste of funding agency and taxpayer dollars. The rejected data are set aside and not archived.

As we progress through the years from one computer medium to another, the unused data may be neglected. Some [researchers] feel that if you gather enough data and n approaches infinity, all noise will cancel out and a true signal will come through. That is not true. I maintain that one should not add data without signal. It only increases error bars and obscures signal.

As an ex- marine I refer to the concept of a few good men.

…

I was dumbfounded by Jacoby’s response. At the time, I expressed my disbelief as follows:

Imagine this argument in the hands of a drug trial. Let’s suppose that they studied 36 patients and picked the patients with the 10 “best” responses, and then refused to produce data on the other 26 patients on the grounds that they didn’t discuss these other patients in their study. It’s too ridiculous for words.

The incident also sheds light on the question of when data is “used”. I plan to cite this incident in another forthcoming post. No statistician would accept Jacoby’s argument for a minute. By examining 36 series and picking 10, all 36 series were “used”. I find it hard to believe that Jacoby’s position has any traction whatever, but I was unsuccessful in persuading Schneider.

Jacoby’s practices came up unexpectedly in the NAS panel workshop, when Rosanne D’Arrigo told an astonished panel that “you had to pick cherries if you want to make cherry pie” (see here). Although this evidence was highly relevant to the subject of the NAS inquiry and occasioned a flurry CLimategate emails, the NAS panel report avoided the issue entirely.

A related issue arises in respect to the Yamal-Urals regional chronology where CRU examined several versions before reverting back to the very HS-shaped chronology arising from the very small dataset used in Briffa 2000. I’ll discuss this in another post.

Bona-Churchill
In the mining exploration business, investors who trade mining stocks know that “late” results are almost never “good” results. The reason is human nature. In public stocks, you’re legally obligated to report results promptly, but there is some play in timing. If promoters have “bad” results in the first part of the program, there is a great temptation to delay the bad news in the hope that later results will bail out the program. The best and only way to deal with temptations to delay bad results is to establish an announcement schedule ahead of time and stick to it.

In 2006, I noticed that Thompson had swiftly reported Kilimanjaro results, but results from Bona Churchill had not been reported with the same alacrity. (Six years later, Bona Churchill results still have not been published, let alone archived.) My surmise at the time was that the results were “bad” (i.e. did not have elevated O18 values in the 20th century.)

By saying this, I am not saying that climate scientists are less honorable than mining promoters. Only that there are great human temptations to delay reporting “bad” results. And, after a while, delay can turn into neglect, without any explicit decision ever having been made not to report the “bad” results.

In the mining business, promoters are bursting to report good results. I presume that this temptation also affects climate scientists, who, as Schmidt tells us from time to time, are human and subject to human frailties. I remain convinced that climate scientists are more eager to publish “good” results than “uninteresting” ones. Especially when Nature and Science have such an appetite for “worse than we thought” articles to be a mild object of satire even within the “community”.

Six years later, Bona Churchill results remain not only unarchived, but unpublished. At this point, one cannot say that the Bona Churchill results have been “suppressed” for good; but they have clearly been delayed. A graphic in a workshop shows that my surmise was correct: contrary to Thompson’s expectation, 20th century O18 values were not elevated. They are “inconvenient”.

Conclusion
Back to Gavin’s point.

It’s not tin foil country to say that, in respect to Jacoby and D’Arrigo 1989, Jacoby failed to report or archive data that didn’t show what he was expecting. I typically describe this sort of phenomenon in less charged terms than Schmidt: “failed to report” as opposed to “suppressed”. But there’s nothing tin foil about saying that Jacoby was selective in what data he archived – Jacoby said so. One of the many problems in this field is that so many “real” scientists see nothing wrong with this.

Similarly with Mann’s residuals (“dirty laundry”) and verification r2. And Thompon’s Bona-Churchill.

I find it hard to believe that Schmidt seriously believes that climate scientists have reacted to suspicion by saying “take a look”. The pages of Climate Audit are replete with one incident after another, where climate scientists have taken precisely the opposite attitude.

That includes the present case. The sane resolution of the present FOI request would have been for East Anglia to just send me the data, even if they felt that they didn’t “have” to. The best way to remove suspicion would have been to say: “Here, Steve, take a look”. If Schmidt thinks that that’s what CRU has done in this case, then he’s the one in tin foil country.

In the wake of the 2009 Yamal discussion, CRU had contacted Hantemirov for additional data. In CG2-1025 on Oct 5, 2009, Hantemirov wrote Melvin, sending additional data as follows:

Dear Tom,
files with living trees data attached, that I use to update Yamal chronology (these data have been used among many others in Esper et al. 2009: [1]http://www3.interscience.wiley.com/journal/122374111/abstract?CRETRY=1&SRETRY=0).
First letters in ID means river (valley):
TNL – Tanlova-yakha;
HDT, M, X – Khadyta-yakha;
POR – Porza-yakha;
all others – Yadayakhodyyakha
…

This email caught my attention because the CRU reconstruction using “all the data” in October 2009 did not contain any cores with prefix TNL or HDT. (Note that the HDT, M and X prefixes denote cores from Khadyta River – which Mann and Real Climate declared to be “inappropriate” for use in a reconstruction.)

In Table S2 to Esper et al 2009 (mentioned by Hantemirov), site 25 was located in Yamal (70E, 67.5N) and had 120 cores.

I did an RCS-style calculation on the Hantemirov data set and repeated the comparison illustrated in my original Climate Audit post of Sep 27, 2009 that initiated the present controversy, as shown below:

Figure 1. Yamal Chronologies. Green – from Hantemirov _liv.rwl dataset; red- from Briffa et al 2008.

Now here is the corresponding graphic from the Yamal post in September 2009:

Figure 2. Yamal chronologies from original post. Green – Yamal data from Briffa 2000 plus Schweingruber Khadyta River; red- from Briffa et al 2008.

Here is the calculation from Hantemirov data compared to the Sept 2009 green chronology plotted together:

Figure 3 – From Hantemirov data versus Sept 2009.

I think even my severest critic might observe that the chronology using the Hantemirov data set is remarkably similar to the green chronology of September 2009. The closeness of the chronologies is partly coincidence – the September calculation was only intended to show the potential impact of additional data on the very small Briffa data set. But coincidence or not, the likeness really is remarkable. The results through the 1990s and early 2000s are interesting as well: elevated ring widths, but not the multi-sigma of Briffa’s Yamal chronology.

While the form of my RCS calculation differs slightly from CRU’sstyle in Briffa et al 2008 (which is not archived), I am 99.9% sure that the difference is not material to the major point of the result and that an RCS calculation on the Hantemirov data in current CRU style would yield a chronology indistinguishable from the one that I’ve presented (yes, an “insta-chronology”, but they really only take an instant to calculate.)

In contrast, CRU’s 2009 calculation, said to use “all the data”, showed negligible change from the chronology of Briffa et al 2008, as shown below (on the same scale):

Figure 4. Yamal chronologies of Briffa et al 2008 and CRU 2009.

The difference between the two results is the selection of data. The 2009 CRU calculation only included a fraction of the Hantemirov dataset (while increasing populations at the YAD, POR ad JAH sites.)

Postscript:
Esper et al 2009 has almost unintelligible graphic and nothing is archived. (It was received on Jan 28, 2009 and accepted on Jan 30, 2009.)

It discusses West Siberia and includes the Yamal and Urals areas. Here is an excerpt from its Figure 6. The red is the ring width chronology and, if you squint, it has points of similarity to the green curve in Figure 1 – it definitely does not show any support for Briffa-style multi-sigma growth.

Figure 5. Excerpt from Esper et al 2009 Figure 6 (resized to correspond). “Middle panel shows the WSIB and WSIBnew tree growth data scaled over the 1881–1990 (WSIB) and 1881–2000 (WSIBnew) periods to regional JJA temperatures”

Note:
The chronology here was calculated as follows (you can’t replicate this because I haven’t placed the Hantemirov data online) but it shows the form of calculation and shows that the data set is the same as the one used in Esper et al 2009. This should generates all figures turnkey (May 16, 2012).

# REGIONAL CHRONOLOGY ANALYSIS
##
#Utilities  
	source("http://www.climateaudit.info/scripts/utilities.txt")
	source("http://www.climateaudit.info/scripts/tree/utilities.treering.txt")
	trim=function(x) window(x, start=min(time(x)[!is.na(x)]),end=max(time(x)[!is.na(x)]) )

#Hantemirov Living Data
  #living data from Hantemirov sent by email
	#loc="d:/climate/data/tree/measurements/r/hantemirov/liv.rwl.tab"
	#load(loc)
 	#range(tree$year) # [1] 1580 2005
  	#length(unique(tree$id)) # [1] 120
	#chron.hant=RCS.chronology(tree,"nls")
	#chron.hant=chron.hant$series; 
	#X=data.frame(year=c(time(chron.hant)), chron=chron.hant)
	#write.csv(X, file="d:/climate/data/yamal/chron.hant.csv", row.names=FALSE)
	X= read.csv("http://www.climateaudit.info/data/tree/crn/yamal/chron.hant.csv")
	chron.hant=ts(X[,2],start=X[1,1])

#Yamal Chronologies
  #Briffa 2008 Chronology
	loc= "http://www.climateaudit.info/data/tree/crn/cru/yamal_2008.crn.tab"
	  download.file(loc, "temp",mode="wb");  load("temp")
	  yamal08=chron.crn
	tsp(yamal08) #-202 1996
	
#Figure 1: compare Hantemirov Living RCS to CRU 2008
   #png("d:/climate/images/2012/yamal/hantemirov_compare.png",w=480,h=480)
	par(mar=c(3,3,2,1))
  	ts.plot(chron.hant,xlim=c(1850,2005),ylim=c(0,3.5),ylab="",lwd=2,col=3)
 	lines(yamal08,col=2)
	legend("topleft",fill=3:2,legend=c("Hantemirov Live","CRU 2008") )
	title("Yamal Chronologies from 1850")
   #dev.off()

#Figure 2: this is from a September 2009 post http://climateaudit.org/2009/09/27/yamal-a-divergence-problem/ 
    #script is at http://climateaudit.org/2009/09/27/yamal-a-divergence-problem/#comment-195389


#Figure 3: compare Sept 2009 to Present graphic 
	X=read.csv("http://www.climateaudit.info/data/tree/crn/yamal/september2009.csv")
	crn2=ts(X[,"crn2"],start=-202); 
	  tsp(crn2) #  -202 1990 
 
    #png("d:/climate/images/2012/yamal/hantemirov_compare2.png",w=480,h=480)
	ts.plot(chron.hant,xlim=c(1850,2005),ylim=c(0,3.5),ylab="",lwd=2,col=3)
	lines(crn2) 
	legend("topleft",fill=c(3,1),legend=c("Hantmirov Data","Sept 2009") )
 	title("Yamal Chronologies from 1850")
    #dev.off()


#Figure 4: compare CRU 2008 to CRU 2009
load("d:/climate/data/tree/chronologies/r/cru/briffa_2009.crn.tab")
 	loc= "http://www.climateaudit.info/data/tree/crn/yamal/briffa_2009.crn.tab"
	 download.file(loc, "temp",mode="wb");  load("temp")
	yamal09=briffa.crn[,"YAMAL_ALL"]
	  tsp(yamal09) # 202 1996    1

   #png("d:/climate/images/2012/yamal/cru_compare.png",w=480,h=480)
	par(mar=c(3,3,2,1))
	  ts.plot(yamal09,xlim=c(1850,2005),ylim=c(0,3.5),ylab="",lwd=2,col="red4")
	 lines(yamal08,col=2)
	legend("topleft",fill=c("red4","red"),legend=c("CRU 2009","CRU 2008") )
	title("Yamal (CRU) Chronologies from 1850")
   #dev.off()
 	

In yesterday’s post, I focused on points of agreement or points where agreement ought to be possible. In a subsequent RC comment, Schmidt complained that I had failed to respond to his “main point”, which now appears to be his ruminations on the UK Freedom of Information Act.

Schmidt complained inline:

Also of interest is that McIntyre doesn’t even mention the main point I brought up. That this is ongoing and *unpublished work*, and there is a clear FOI exemption for this (for obvious reasons), that can only be trumped by a clear public interest – and despite the endless exaggerations by McIntyre about the importance of these reconstructions, that bar is nowhere close to being met.

The argument in the post itself was:

UK FOI legislation (quite sensibly) specifically exempts unpublished work from release provided the results are being prepared for publication. So McIntyre’s appeals have tried to insinuate that no such publication is in progress (which is false) or that the public interest in knowing about a regional tree ring reconstruction from an obscure part of Siberia trumps the obvious interest that academics have in being able to work on projects exclusively prior to publication. This is a hard sell, unless of course one greatly exaggerates the importance of a single proxy record – but who would do that?

Gavin expanded further on his interpretation of UK legislation in an online response as follows:

RC Reader Jason had written:

UK FOI legislation (quite sensibly) specifically exempts unpublished work from release provided the results are being prepared for publication.” Gavin, this is simply untrue.

Schmidt responded:

Response: You are incorrect, see Section 22. It is not an absolute exemption (it can be overridden by a great enough public interest), but the text is quite clear that: “Information is exempt information if— (a) the information is held by the public authority with a view to its publication, by the authority or any other person, at some future date (whether determined or not)”. Please look stuff up before accusing people of lying (that would be the lesson all around actually). – gavin]

First, let me make a passing editorial comment about all-too-characteristic Real CLimate rhetoric – the inflation of any criticism into a supposed “accusation” against integrity. In this case, Jason had merely asserted that Schmidt’s statement on FOI legislation was “untrue”. Schmidt responded by accusing Jason of “accusing people of lying”. Jason had done nothing of the sort. He merely said that Schmidt’s statement was “untrue”. The express language contained no accusation or even implication that Schmidt had been “lying”.

Jason’s experience here very much accords with my own – a matter that I’ll return to some time.

Schmidt’s argument was based on UK Freedom of Information Act section 22. Now let me quote from the opening paragraph of my Appeal:

On Feb 28, 2011, I submitted a request under the Environmental Information Regulations 2004 (EIR) to the University of East Anglia (UEA) for a regional tree ring chronology referred to an April 28, 2006 Climategate email … On March 28, 2011, the UEA refused all three requests, citing the exemptions set out in sections 6(1)(b) (available elsewhere), 12(4)(d) (incomplete) and 12(5)(c) (intellectual property rights) [my bold].

My request and appeal are under the Environmental Information Regulations, rather than the Freedom of Information Act itself. Speaking colloquially, I’ll sometime (perhaps even often) use the more familiar term “FOI” to describe these proceedings, but the actual applications and appeals are under the Environmental Information Regulations, which under UK practice and precedents have been found to govern this sort of data and information. The EIR is related to the Freedom of Information Act, but differs on some important issues, (which David Holland, in particular, has paid close attention to.) The EIR has been discussed in a number of blog posts, sometimes at exhaustive length.

In short, no matter how convinced Schmidt may be that UEA has a slam-dunk case under FOI section 22, the actual decision will turn on whether they have a case under either EIR section 12(4)(d) or EIR section 12(5)(c).

I submit that the argument is by no means as clearcut as Schmidt believes and urge him to re-read my Appeal in terms of the actual exemptions at issue, rather than the inapplicable one that caught his fancy. In addition, Schmidt should weigh the fact that East Anglia has already lost numerous decisions. In the present case, a number of East Anglia exemption claims have already been discarded; and I’ve already been notified of success on one aspect of the appeal (the list of sites).

After misunderstanding UK law on the matter, Schmidt made the following editorial remark:

So McIntyre’s appeals have tried to insinuate that no such publication is in progress (which is false) or that the public interest in knowing about a regional tree ring reconstruction from an obscure part of Siberia trumps the obvious interest that academics have in being able to work on projects exclusively prior to publication. This is a hard sell, unless of course one greatly exaggerates the importance of a single proxy record – but who would do that?

The first claim is completely untrue.

None of my appeals “insinuated” that no such publication is in progress. The point is irrelevant to exemptions 12(4)(d) or EIR section 12(5)(c), as opposed to Schmidt’s imaginary application of FOI section 22. Further, in my Appeal, not only did I not make the alleged argument, but I quoted at length from David Palmer’s brief describing CRU’s planned publication (see page 13). It’s frustrating that Schmidt makes untrue statement after untrue statement without ever correcting or apologizing,

Schmidt’s argument about trade-offs partly transposes to the EIR “intellectual property right” exemption, but not totally. And while his argument may seem “obvious” to academics, I can say with total certainty that his argument is far from “obvious” to professionals and non-academics, once the data is cited by IPCC and policy decisions are made, even in part, from such data.

Whether policy-makers chat with Schmidt about proxies hardly resolves the tradeoff. The “regional tree ring reconstruction from an obscure part of Siberia” was one of only eight proxies shown in the IPCC AR4 proxy graphic and was used in 5 of 8 AR4 medieval multiproxy reconstructions. If IPCC had not cited Yamal, then the situation would be different. But it did.

East Anglia’s argument under section 12(5)(c) is that disclosure of the data would result in financial loss to the university through reduced grants. Their argument on this point is far-fetched. However, in a tradeoff between relatively remote prospects of reduced grants,as compared to the public interest in a complete record for data used by IPCC, it is hardly “obvious” that the financial interest of the University of East Anglia should be preferred. (In passing, given the usual vehemence with which climate scientists reject suggestions that alarmism is affected or influenced by grant-seeking, it’s ironic that CRU’s primary argument in this case depends on grant-seeking.)

Another important issue that Schmidt more or less sweeps under the carpet is that, in my opinion, many, if not most, professionals would regard the contemplated Yamal-Urals regional chronology as being within the scope of Briffa et al 2008 and, not in the strictest sense, “unpublished”. I realize that reasonable people can disagree on this point and that many academics may not. The difficulty arises because Briffa et al 2008 did not properly disclose that they had considered various Yamal-Urals regional chronologies but had not been able to “finish” one in time for the article. In my opinion, had this been properly disclosed, the editor and reviewers would have demanded that they “finish” the calculation as a condition of acceptance. I’ll discuss this issue some more in a separate post.

Much of the present problem arises from the dissembling Muir Russell “inquiry”, which should have dealt with all of this, but negligently failed to do so.

The Inter Academy Council had strongly endorsed transparency at IPCC:

it is essential that the processes and procedures used to produce assessment reports be as transparent as possible.

Transparency is an important principle for promoting trust by the public, the scientific community, and governments. Interviews and responses to the Committee’s questionnaire revealed a lack of transparency in several stages of the IPCC assessment process, including scoping and the selection of authors and reviewers, as well as in the selection of scientific and technical information considered in the chapters.

This had been strongly opposed by Phil Jones, (see CA post here which reviews Jones’ correspondence with Stocker.) Stocker led a bureaucratic counter-offensive urging more confidentiality:

Confidentiality is part of the basic way in which IPCC goes about its work..

Stocker’s bureaucratic intrigues (see CA post here and David Holland here) led to the adoption of language at the May 2011 IPCC meeting that increased confidentiality (see contemporary Bishop Hill post here).

As a result of David Holland’s further initiatives, the UK Minister of State for Energy and Climate Change wrote to his MP as follows:

We are aware that this new text would mean that reviewers would not have the opportunity to see how their comments had been addressed by IPCC authors before acceptance of the final report. It was not the IPCC’s intention to change the procedures in this way and it is likely a drafting error. Indeed, the intention of the update in the procedures was to increase openness in the way that IPCC reports are prepared. We understand that the IPCC is aware of this issue and intends to address it at the next appropriate opportunity.

The IPCC meets in early June.

I’m betting on Sir Humphrey.

I’ve now looked through his post carefully and, beneath Schmidt’s fulminations, did not find any rebuttal to any points actually made in my post, as I’ll discuss in detail below. Much of Schmidt’s post fulminates against my criticism of inadequate disclosure of adverse results. This is a large topic in itself that provides a context to the Yamal controversy, but the exposition of this context is lengthy and, in my opinion, the Yamal issues are sufficiently discrete that they can be considered on their own, as I shall do in this post.

The “Insta-Reconstruction”
Schmidt criticized my estimate of a Yamal-Urals regional chronology from the FOI list as merely “insta-reconstruction”, saying:

if any actual scientist had produced such a poorly explained, unvalidated, uncalibrated, reconstruction with no error bars or bootstrapping or demonstrations of common signals etc., McIntyre would have been (rightly) scornful.

While my “insta-reconstruction” was only produced for a blog post, I submit that it was better documented than many tree-ring chronologies in common use, including the three chronologies of Briffa 2000 link. that have been very popular in multiproxy reconstructions. Indeed, if “actual” scientists had provided equivalent detail for the tree-ring chronologies in common use, much of the controversy of recent years would have been avoided.

Even though this was merely a blog post, I provided all measurement data as used, together with code that, in a turnkey method, uploads the data as used and produces the chronology and graphics. The code shows the precise calculation for a an interested critic. The blog post included a graph of core counts, together with the computation. In contrast, no measurement data for the three chronologies of Briffa 2000 (Taimyr, Tornetrask, Yamal) was archived. Nor was it archived for Briffa et al 2008 until I convinced the journal to require it. New measurement data for Esper et al 2009, which addresses Siberia, is not archived. (I recently requested that it be archived, but have received no acknowledgement from either Esper or Hantemirov.)

Nor is it common practice for dendros to show “error bars, bootstrapping or demonstration of common signals” in typical publication of tree ring chronologies. I think that such practices would indeed enhance the articles, but it’s not something that I’ve taken issue with in dendro publications. (Merely getting dendro data has been hard enough.) None of Schmidt’s desiderata appear in Briffa 2000 or for other important tree ring chronologies (e.g. the Graybill stripbark chronologies that are such an important part of IPCC multiproxy studies.) Calculation of error bars for tree ring chronologies seems very much to be desired, but, in my opinion, to do so would require a major improvement in the statistical theory of chronology construction, something this is long overdue.

One Point of Agreement
In any controversy, it’s always a good idea to try to isolate any points of agreement and work out from there. There are relatively few points in Schmidt’s post that I agree with, but it is common ground between Schmidt and I that CRU’s claim that the “purpose” of Briffa et al 2008 was merely to reprocess the Hantemirov and Shiyatov dataset was a “mis-statement”.

In their June 17 submission to Muir Russell, CRU had stated:

McKitrick is implying that we considered and deliberately excluded data from our Yamal chronology. The data that he is referring to were never considered at the time because the purpose of the work reported in Briffa (2000) and Briffa et al. (2008) was to reprocess the existing dataset of Hantemirov and Shiyatov (2002).

In my post on Yamal, I had stated that the purpose of Briffa 2000 was to reprocess the Hantemirov data, but this “purpose” no longer applied in 2008.

One of the purposes of Briffa (2000) was clearly to demonstrate the effect of RCS methodology on the Hantemirov and Shiyatov 2002 dataset. I have no objection to CRU claiming this “purpose” for Briffa (2000). But, by 2008, this was no longer their “purpose”. Indeed, one doubts whether the editors of Phil Trans B would have accepted a 2008 paper with such a mundane purpose. The actual “purpose” of Briffa et al 2008 is stated quite clearly and was entirely different: it introduced and discussed “regional” chronologies.

Amidst other hyperventilating editorial comment (on which I’ll not comment right now), Schmidt conceded that CRU had committed a “mis-statement” in saying that the “purpose” of Briffa et al 2008 was to reprocess Hantemirov data – a point on which I can agree.

… using the reference to the 2008 paper is a little contradictory to the paragraphs above which were much more explicit about the background and purpose of the 2008 paper. However, to take a slight mis-statement in a single sentence, when copious other information was being provided in the same submission, and accusing people of deliberate deception is a huge overreach. Were they trying to deceive only the people who hadn’t read the previous page? It makes no sense at all. Instead, McIntyre conflates the situation at the time of the 2000 paper with the very different situation around 2008 in order to paint a imaginary picture of perfidy.

I do not agree that this was merely a “slight mis-statement in a single sentence” as the same argument had been made in their March 1 submission to Muir Russell, a submission which did not include accompanying “copious” information evidencing the error to the Muir Russell panel as a “slight mis-statement”. It also speaks poorly that CRU would make a “mis-statement” to Muir Russell on an issue so central to Climategate.

But in any event, let’s take some comfort that Schmidt and I at least agree on this point: the “purpose” of Briffa et al 2008 was not the reprocessing of the Hantemirov dataset, but the development and presentation of regional chronologies.

An Easily Resolved Point
In addition to the above point of agreement, there is one point that can be objectively resolved: did CRU calculate a Yamal-Urals regional chronology in 2006? I believe that the evidence is unequivocal that such a calculation was made; Schmidt believes otherwise, arguing that my belief rests only on a “very strained” interpretation of a Climategate email.Schmidt argued as follows:

McIntyre then quotes an email from Osborn sent in 2006 in support of his claim that the reconstructions were finished at that point, but that is again a very strained reading. Osborn only lists the areas (and grid boxes) in which regional reconstructions might be attempted since “most of the trees lie within those boxes”. It makes no statement whatsoever about the work having already been done.

Schmidt relied on this exegesis of the email to rationalize CRU’s statement to Muir Russell that they had “intended” to do a Yamal-Urals regional chronology, but decided against it because they felt that it could not be “finished in time”. Schmidt outlines his position as follows:

Of course, regional reconstructions are a definite goal of the dendro-climatology community and Briffa and colleagues have been working on these for years. Some of those results were published in Briffa et al (2008) as part of a special issue on the boreal forest and global change. Special issues come with deadlines, and as explained in a submission to the Muir Russell inquiry, a regional Yamal reconstruction putting together multiple sources of tree ring data was indeed ‘considered’ but wasn’t finished in time

Between these [two other reconstructions] we had intended to explore an integrated Polar Urals/Yamal larch series but it was felt that this work could not be completed in time and Briffa made the decision to reprocess the Yamal ring-width data to hand, using improved standardization techniques, and include this series in the submitted paper [Briffa et al., 2008].

So, Briffa et al did consider a regional reconstruction and are indeed working on it for publication, and it didn’t get into the 2008 paper due to time constraints.

Clear, no?

I think that the last sentence would ought to read: Clear? No!

Schmidt’s argument that the 2006 calculation was never shall-we-say consummated has been picked up by a number of commenters on the RC thread, who’ve thrown down the gauntlet demanding that I prove the existence of the 2006 chronology. An early commenter on the RC thread endorsed Schmidt as follows:

To me, McI’s (as well as anyone who follows him) accusations have to show that Gavin is wrong here. Otherwise, his entire argument crumbles into nothing. … I have to ask anyone defending McIntyre to prove this. Otherwise, he should withdraw his accusations, which amount to scientific malfeasance, if true.

Subsequent RC commenters pile onto the same theme, with Schmidt complaining inline against Jeff Id that it is impossible to prove a negative:

No one can ever prove that they didn’t do a calculation, and ever more insistent demands that they must, are pointless

On the other hand, it is possible in this case to prove that CRU calculated a Yamal-Urals regional chronology in 2006 because the University of East Anglia has admitted it.

They’ve admitted it both implicitly and explicitly. Implicitly: if they didn’t hold the chronology, they would have claimed that exemption in the FOI. However, they made no such claim, instead arguing other exemptions. Explicitly: East Anglia confirmed in response to my FOI request as follows:

In investigating your appeal a further search has been undertaken and it has become clear that the first time series out of the 1001 bootstraps is actually a single nonbootstrap) chronology.

Tim Osborn of CRU showed up as a commenter on the RC thread. In my opinion, he should have set the record straight – that CRU had done the regional chronology questioned by Schmidt. Unfortunately he failed to do so.

Regardless, this is a point that Schmidt should simply concede: CRU did such a calculation in 2006. End of story.

Of course, making this admission has a knock-on effect: CRU’s claim to Muir Russell – that they had “intended” to do a Yamal-Urals regional chronology, but had been unable to finish it – cannot be accepted at face value. This ought to require Schmidt to re-think this aspect of his post, though I doubt that he will do so.

Disclosure
In the (unlikely?) event that Schmidt concedes that the calculation of the Yamal-Urals regional chronology was consummated in 2006, I presume that his next line of defence will be that the consummation was in-some-still-undisclosed-way unsatisfying. (I presume that these grounds will be argued in the forthcoming CRU article responding to this controversy.}

Even if the consummation of the chronology was unsatisfying, the consideration and discarding of the Yamal-Urals regional chronology was an integral part of the methodology of Briffa et al 2008 and should have been to Phil Trans B as part of the methodology.

For amusement, let’s suppose that the editor and reviewers for Phil Trans B were presented with the statement on the regional chronology that CRU had presented to Muir Russell:

Between these [two other reconstructions] we had intended to explore an integrated Polar Urals/Yamal larch series but it was felt that this work could not be completed in time and [we] made the decision to reprocess the Yamal ring-width data [from Briffa 2000], using improved standardization techniques.

If I were an editor or reviewer confronted with such language, I’d undoubtedly ask the authors to “complete” their analysis before expecting it to be accepted.

Once they considered and discarded the Yamal-Urals regional chronology, they should have disclosed the procedure, with at least the following:

Between these [two other reconstructions] we calculated an integrated Polar Urals/Yamal larch series but decided against using it for the following reasons [....whatever they were]. For reference, the Yamal-Urals regional chronology is shown in the Supplementary Information…

What’s so unreasonable about that?

Similarly, CRU’s submission to Muir Russell should have included straightforward disclosure that they had done a regional chronology calculation in 2006 and decided against using it (for whatever reasons.)

CRU’s failure to provide accurate disclosure, first to Phil Trans B and second to Muir Russell, does not mean that the entire field of climate science is pervaded by “perfidy”. But neither is acceptable, let alone praiseworthy.

Postscript
The present dispute arises from East Anglia’s refusal to provide the 2006 Yamal-Urals regional chronology in response to my FOI request. East Anglia has argued that they are entitled to refuse this information under the exemptions for incompleteness and intellectual property.

Even if they are ultimately successful in their arguments (and the UEA has lost almost every decision that has been taken to the ICO or Tribunal), surely both East Anglia and the larger climate community should ponder whether this particular fight is worthwhile, particularly at this time. Even if CRU has the right to withhold the data, they are not prevented from voluntarily disclosing it.

Not every battle is worth fighting. If I were managing the file for East Anglia, I’d tell Jones and Briffa: turn the data over voluntarily. Similarly, Schmidt himself realizes that battles over withholding data play poorly outside the most partisan part of the community: why wouldn’t he encourage CRU to voluntarily disclose the data and save themselves the bad publicity.

In addition, arguing that working results from 2006, especially in a controversial area of climate science covered by IPCC, should continue to attract a form of exemption is about as bad a test case as the academic community could imagine. By litigating this case, there is a risk that the wider academic community won’t like the precedent. By disclosing the data voluntarily, no precedent is set.

In a subsequent post, I’ll try to return to other aspects of Schmidt’s rant, including his offence at criticism of failures to disclose adverse data.

In the present post, I’ll deal with the following spitball:

Similarly, McIntyre recently accused Eric Steig of suppressing ‘inconvenient’ results from an ice core record from Siple Dome (Antarctica). Examination of the record in question actually demonstrates that it has exceptionally high values in the late 20th Century (reflecting the highest temperatures in at least the last 700 years, Mayewski et al.), exactly counter to McIntyre’s theory.

Schmidt’s complaint arose from a CA post earlier this year, in which I criticised Neukom for their failure to archive proxy data in a study prominently cited in IPCC AR5 First Order Draft. Despite self-serving assertions that archiving has improved (e.g. Maibach et al 2012), little has really changed.

In my post on Neukom, I observed that one of the long South Hemisphere series of Neukom et al had been developed by Eric Steig in the mid-1990s and was not “new” since AR4. I also observed that it was unarchived. Based on the fact that this series had not been archived, I speculated (cynically, I’ll admit) that the series would not have a HS-shape.

Steig turned up to comment. He first claimed that the data had been archived and that I had simply not been “adept” enough to locate it. Subsequently, Steig conceded recognized that the data had not been archived and (commendably) placed the data online at his website. The comments ar worth re-reading.

Schmidt claimed that my cynical surmise had been refuted in this case because Steig’s dD record “has exceptionally high values in the late 20th century”:

Examination of the record in question actually demonstrates that it has exceptionally high values in the late 20th Century

Unfortunately, as too often, Schmidt’s statement is not only untrue, but easily seen to be untrue. Below is a plot of the 20th century portion of Steig’s dD record. There isn’t a HS shape in the 20th century – if anything, there is a slightly downward trend. Nor does the record have “exceptionally high values in the late 20th century”, as shown below.

Figure 1. Steig dD from Siple Dome – 20th Century

Update: Schmidt posted up a 1000-year graph in a form similar to the graph shown in comments below, describing the graphic as showing that “the 20th Century is warmer than the long term mean in both dD and d18O”. The archived d18O series is relatively short. If Schmidt had originally said that “the 20th Century is warmer than the long term mean in dD”, I would not have taken exception to his comment. But that’s not what he said. He said that the unarchived record “has exceptionally high values in the late 20th century”. It didn’t. As I originally observed, in my opinion, if this series had had a big HS, I’m convinced that it would have been publicized and archived years ago. As events turned out, it was archived approximately 18 years after being collected, and only after an issue was made about it at Climate Audit. I didn’t use the word “suppress” – that’s Schmidt’s word. In this case, it’s more that the results seemed “uninteresting” and were neglected. One situation is not necessarily the same as another.

If I were in East Anglia’s shoes, I wouldn’t have wasted two seconds fighting this FOI request. I’d have told Wahl and Ammann that we didn’t think that we received the article on a “confidential” basis since it was expected to be used in IPCC, that we had had enough headaches and didn’t want to have one more fight, and therefore we expected them to acquiesce in our decision. If I were the UEA administration, I’d have asked Jones and Briffa to agree with this decision and get Wahl and Ammann to cooperate. That’s what any sane person in the private sector would have done.

Instead, East Anglia has contested every step of the FOI, which has now reached the Tribunal.

As with all legal proceedings, they take on a life of their own after a while and so it is with this case. In today’s post, I’ll discuss an interesting fact-law issue arising out of East Anglia’s submission to the Tribunal on May 9.

In the Information Commissioner decision presently under appeal, the ICO had applied the exemption under EIR section 12(5)(f) as follows:

33. In this case the University has argued that the authors of the requested documents [Wahl and Ammann] provided this information to the University and that those authors were not under or could not be put under any legal obligation to provide the information, the University is not entitled to disclose it and the authors have not consented to disclosure. The Commissioner accepts this position [my bold]…

One of the curiosities in the procedure is that I was never provided copies of the University’s submissions or arguments. A few days after receiving the decision, I requested copies of the University’s submissions, which the ICO treated as an FOI request (as opposed to documents routinely provided to an opposing party in a dispute.)

There is a short appeal period for ICO decisions. As of March 1, 2012, more or less the last day of the appeal period, I had not heard back from the ICO and submitted an appeal to the Tribunal.

A little later in March, I received a partial response from the ICO, but three submissions from the University were exempted on the grounds that they were submitted “in confidence”.

In my Grounds of Appeal, I took at face value the ICO’s assertion that “the University has argued that the authors of the requested documents [Wahl and Ammann] provided this information to the University” and argued that they had no expectation of confidentiality because it was provided for use in an IPCC Assessment Report:

21. I submit that Wahl and Ammann consented to disclosure of this article by their decision to submit the article to an IPC Lead Author for consideration (and eventual use) in the IPCC

In response, UEA argued this week that my case on this point was “misconceived”. Not because of defects in my case as presented (though they would doubtless argue this if pressed), but because Wahl and Ammann had not provided the article to Briffa. Changing the story from the one in the ICO decision, they now aid that the article was sent to Briffa by Jones (then an IPCC Coordinating Lead Author), who, in turn had received it from Mann (a former IPCC Lead Author and a reviewer of the Zero Order Draft). They observed that the terms under which Mann received the article were “entirely unclear”:

32. Mr Mclntyre’s case on this issue is misconceived. Mr Mclntyre’s case proceeds on the basis that document 1 was obtained by Professor Briffa in circumstances where it had been sent to him by Drs Wahl and Ammann in his capacity as IPCC Lead Author (s. 21 GoA). However, document 1 was not sent by Drs Wahl and Ammann to Professor Briffa. In fact, Professor Briffa only obtained a copy of document 1 as a result of it having been forwarded to him by Professor Jones, under cover of an email dated 4 January 2005. Professor Jones obtained his copy of document 1 as a result of its having been sent to him by Professor Mann on 30 December 2004. It is entirely unclear how Professor Mann obtained a copy of the paper and on what terms, save that Professor Mann is clearly of the view that the paper should be treated as a confidential document. In the circumstances, there is no basis for inferring that Drs Wahl and Ammann had consented to the publication of document 1, whether under the IPCC rules or at all.

Watch the pea here.

The provenance described in the present submission is supported by a Climategate email (CG2 – 1737; also in CG1) showing that Mann had sent the article to Jones and that Jones had sent it to Briffa. I was well aware of this thread. However, this email does not in itself preclude the possibility of Wahl and Ammann directly providing the article to Briffa – as, according to the ICO, the University had argued. As East Anglia has pointed out from time to time, the Climategate dossier does not include all emails.

From the perspective of my Appeal, I was not in possession of the University’s submissions to the ICO and proceeded on the basis that the ICO had correctly summarized the University’s argument on this point and that the University had not provided inaccurate information to the ICO.

The only reason why I argued “on the basis that document 1 was obtained by Professor Briffa in circumstances where it had been sent to him by Drs Wahl and Ammann in his capacity as IPCC Lead Author” was that this was the basis of the ICO decision.

If, as the University now claims, the authors of the requested document 1 did not provide the information to Briffa, then the premise of the ICO decision falls apart. In order to invoke an exemption, the University would have had to argue that the 12(5)(f) exemption applied under the entirely different circumstances where IPCC Lead Author received the document from Jones, who in turn had received it from Mann, who had obtained the document under circumstances that were “entirely unclear”. This is a much harder job for them.

This raises the question of the basis of the ICO’s statement that “the University has argued that the authors of the requested documents [Wahl and Ammann] provided this information to the University”. This argument does not appear in the submissions from the University that have been provided to me, but some of the submissions were apparently provided to the ICO “in confidence”.

Did the University make this argument in one of the “confidential” submissions? Or did the ICO simply misunderstand the University on this critical point? I’ve written to the ICO’s solicitor asking him to clarify this point. I’ve also urged the ICO to make a submission acknowledging the error on this point (regardless of whether it was through their misunderstanding or inaccuracy in the information provided to them by the University) to avoid perpetuating the error.

The present appeal arises in part from Acton telling to the Parliamentary Committee that they did not inquire deeply into Jones’ deletion of email request because the “emails do exist”:

Q84 Graham Stringer: Thank you. Sir Muir, on page 92 of your report you say, and I paraphrase, that there is no attempt to delete e-mails after there had been a request made, whereas in actual fact the e-mail of 27 May from Jones actually asked for deletion of e-mails, didn’t it?

Sir Muir Russell: It requested them. I think we said that there was incitement to delete. You have quoted half the sentence. The first bit says: “There seemed clear incitement to delete but we had seen no evidence of any attempt to delete in respect of a request already made.” That is quite a tricky area because they do still exist, apart from anything else, but the question that I think you’re getting at is whether we sought to chase that particular question about deletion of requested e-mails through our review.

On this point, UEA made the following submission:

In 18-19 GoA, Mr Mclntyre makes allegations to the effect that evidence given by UEA’s Vice-Chancellor, Professor Acton, to the STC as part of its inquiry into climategate was untrue. UEA does not accept that these allegations are well-founded. However, the doctrine of Parliamentary privilege in any event means that these are not allegations which can or should be countenanced by the Tribunal.

Lots of interesting information and interpretation. See here.

I subsequently submitted an FOI request for the Yamal-Urals regional chronology and a simple list of sites used in the regional chronology. Both requests were refused by the University of East Anglia. I appealed to the Information Commissioner (ICO).

A week ago, the Information Commissioner notified the University of East Anglia that he would be ruling against them on my longstanding FOI request for the list of sites used in the Yamal-Urals regional chronology referred to in a 2006 Climategate email. East Anglia accordingly sent me a list of the 17 sites used in the Yamal-Urals regional chronology (see here). A decision on the chronology itself is pending. In the absence of the chronology itself, I’ve done an RCS calculation, the results of which do not yield a Hockey Stick.

In today’s post, I’ll also show that important past statements and evidence to Muir Russell by CRU on the topic have been either untruthful or deceptive.

The Relevance of Yamal

The Yamal chronology is relevant both because, since its introduction in 2000, it has been used in virtually all of the supposedly “independent” IPCC multiproxy studies (see an October 2009 discussion here) and because it is particularly influential in contributing an HS-shape to the studies that do not use bristlecones.

IPCC AR4 Box 6.4 showed the eight proxies which have been used the most repetitively (this wasn’t its intent.) Of these eight proxies, Briffa’s Yamal (labelled “NW Russia”) is shown with the biggest HS blade, larger even than Mann’s PC1 (labelled here as “W USA”). See here) and tag yamal.

Figure 1. Yamal Chronology in IPCC AR4 Box 6.4. Labelled as “NW Russia”

In previous posts, I’ve satirized the “addiction” of paleoclimatologists to bristlecones and Yamal as, respectively, heroin and cocaine for climatologists. (In pharmacological terms, upside-down Tiljander would be, I guess, LSD, as the psychedelic Mann et al 2008 is indifferent as to whether proxies are used upside-down or not (cue Jefferson Airplane‘s insightful critique of Mannian statistics.)

Although Yamal and Polar Urals had been long-standing topics at Climate Audit, they first attracted wide attention in late September 2009, when measurement data became available for the three “regional chronologies” of Briffa et al 2008 (Taimyr-Avam, Tornetrask-Finland and Yamal).

The 2008 Taimyr-Avam and Tornetrask-Finland networks were dramatic expansions of the corresponding networks of Briffa (2000), but the Yamal network, which was already much smaller than the other two networks, remained unchanged. Analysis of the previously unavailable Taimyr data showed that Briffa had added measurement data from several Schweingruber sites into the Taimyr-Avam regional chronology (a point not mentioned in the article itself.) Since there were a number of Schweingruber sites (including Polar Urals) in a similarly sized region around Yamal, it seemed almost certain that CRU would have done a corresponding regional chronology calculation at Yamal.

This raised the obvious question of why Briffa et al 2008 had not similarly included Schweingruber data in a Yamal “regional” chronology. Ross posed the question in a contemporary op ed as follows:

Combining data from different samples would not have been an unusual step. Briffa added data from another Schweingruber site to a different composite, from the Taimyr Peninsula. The additional data were gathered more than 400 km away from the primary site. And in that case the primary site had three or four times as many cores to begin with as the Yamal site. Why did he not fill out the Yamal data with the readily-available data from his own coauthor? Why did Briffa seek out additional data for the already well-represented Taimyr site and not for the inadequate Yamal site?

The question applied not just to the Khadyta River site in the original CA post, but to Polar Urals and other nearby sites. These questions resulted in considerable controversy at the time. CRU protested their innocence and posted a lengthy response on October 29, 2009, denying that they had ever even “considered” use of the Schweingruber Khadyta River site, discussed in contemporary Climate Audit posts. In a submission to Muir Russell, they later denied ever re-appraising their Polar Urals chronology.

The Climategate dossier was released in November 2009, a few weeks after the Yamal controversy. As Fred Pearce observed in The Climate Files, the Climategate dossier begins with Yamal and ends with Yamal. Pearce also observed that the word “Yamal” occurs more often than any other “totem” of the disputes, even more than “hockey stick”. Nearly all Climategate documents with unbleached dates were copied after my Yamal posts and Yamal measurement data dominated the earliest documents.

The Climategate dossier revealed that CRU had, after all, calculated a Yamal-Urals regional chronology as early as April 2006. (CG1 – 684. 1146252894.txt). The present FOI request referred to this email.

The Yamal-Urals Regional Chronology
The FOI list consists of 17 larch datasets in western Siberia between 50 and 75E, all but one north of 64N. It includes: (1) Yamal of Briffa (2000); (2) Polar Urals used in Briffa (1995); (3) additional Polar Urals data from 1999 (the “update”); (4) the Schweingruber Khadyta River site; (5) 7 other Schweingruber larch data sets; (6) two Shiyatov data sets collected in the 1960s; (7) four Vaganov data sets collected in 1992. All have been publicly archived except the Vaganov data, which, although used as early as MBH98, was first disclosed in the Climategate documents. (CRU placed a Vaganov subset online at their website in April 2012 as part of their FOI response.)

The Schweingruber Khadya River site in the FOI list (russ035w) is exactly the same site that I had discussed at Climate Audit in September 2009. Site russ176w in the FOI list is the Polar Urals “update” discussed in many Climate Audit posts.

The figure below compares core counts for the withheld regional chronology (salmon) to core counts for Yamal (cyan). Modern core counts for the regional chronology are about 20 times higher than core counts in the reported Yamal chronology, reaching nearly 400 cores in the 1960s. In the 1980s, core counts are still around 300, as compared to 12 in the Yamal chronology.

Figure 2. Core Counts: Withheld Regional Chronology and Reported Yamal “Regional” Chronology

CA readers will doubtless recall CRU’s statements that the “best” indication of regional ring widths requires use of “all the data”).

So what is the “best” indication of relative ring-width changes in this Yamal region? One approach is to judge this by making use of all the data to hand.

The next figure shows a Yamals-Urals regional chronology (script shown in first comment). In this figure, I’ve calculated the chronology after allowing for inter-site differences. If inter-site differences are ignored, a “method” potentially used in other Briffa et al 2008 regions, 20th century levels are a little higher but not exceptional. (I’ll post up an attachment showing the differences.)

Figure 3. Yamal-Urals regional chronology. In this regional calculation, inter-site differences in mean ring width are allowed for by scaling post-1500 mean ring width at each site to equal the overall post-1500 mean ring width.

Although this chronology seems to look a lot different than the canonical Yamal hockey stick, prior to the 20th century, they are extremely similar (800-1900 correlation of 0.91.) Briffa’s Yamal chronology diverges high in the 20th century and especially the late 20th century, as shown in both figures below. In the left figure, both series are scaled on 800-1900 to emphasize the 20th century high divergence of the Briffa Yamal chronology; on the right is a scatter plot of Briffa’s Yamal (800-1996) against the regional chronology, with values after 1850 colored, also showing the high divergence.

Figure 4. Left – Yamal and regional chronology (scaled on 800-1900). right – scatter plot. Click to enlarge.

CRU Statements and Evidence
The question for CRU defenders is to justify their preference of such a small core count, when they had already calculated a regional chronology with an order of magnitude more cores. Since the original criticism in September 2009, CRU has given a variety of different responses, but none, in my opinion, answer the question. Indeed, none of their responses to date have even admitted or disclosed their prior calculation of a regional chronology, let alone explain why they didn’t report it, preferring instead to attack their critics.

Derision
CRU’s first reaction to Climate Audit questioning of their inconsistent handling of the Taimyr and Yamal regional chronologies was ridicule – a reaction quickly endorsed by realclimate and other advocacy blogs.

In The Climate Files, Fred Pearce described this response as follows:

For Briffa, as a tree ring technician, what McIntyre was doing was farce. McIntyre simply didn’t know what he was dealing with, he argued. The new data, Briffa said with evident disdain for the ignoramus on his trail, was estimating temperature from “maximum late wood density” measurements. These were quite different from the tree ring width measurements used in the Yamal series. Scientifically they were not comparable. And the data McIntyre had chosen only went back 100 years, which was useless for the longer time series Briffa was interested in.

Realclimate, known from Climategate emails to have been coordinating with CRU on this matter, immediately accused me of “randomly” adding in data found on the internet:

McIntyre has based his ‘critique’ on a test conducted by randomly adding in one set of data from another location in Yamal that he found on the internet. People have written theses about how to construct tree ring chronologies in order to avoid end-member effects and preserve as much of the climate signal as possible. Curiously no-one has ever suggested simply grabbing one set of data, deleting the trees you have a political objection to and replacing them with another set that you found lying around on the web.

These criticisms were widely accepted in the advocacy “community”. However, the incorrectness of the derision is eloquently rebutted by the inclusion of the Khadyta River dataset in the FOI list. Not that I expect any of them to withdraw these criticisms.

Khadyta “Not Considered”
A few weeks later (October 29, 2009), CRU took a different and inconsistent line in an article on the CRU website. This time, instead of ridiculing the inclusion of Khadyta River, they conceded that the site met their criteria, but claimed that they had “simply not considered” it at the time:

Our current practice when selecting data to incorporate in a regional chronology, is to include data exhibiting high levels of common high-frequency variability (i.e. on the basis of high inter-site correlations, where these are calculated using high-pass filtered data). Judged according to this criterion it is entirely appropriate to include the data from the KHAD site (used in McIntyre’s sensitivity test) when constructing a regional chronology for the area. However, we simply did not consider these data at the time, focussing only on the data used in the companion study by Hantemirov and Shiyatov and supplied to us by them.

They re-iterated this claim in both their March 1 and June 17 submissions to Muir Russell, stating, for example, in the latter (p. 8):

“McKitrick is implying that we considered and deliberately excluded data from our Yamal chronology. The data that he is referring to were never considered at the time because the purpose of the work reported in Briffa (2000) and Briffa et al. (2008) was to reprocess the existing dataset of Hantemirov and Shiyatov (2002).

Again, CRU’s claims not to have “considered” inclusion of Khadyta River (and other similar Schweingruber sites) is refuted by the FOI list. The untruthfulness of this evidence was not commented on by Muir Russell.

The “Purpose” of Briffa et al 2008
According to CRU’s website statement and submissions to Muir Russell, the reason why they had failed to “consider” the Khadyta River data was that the “purpose” of Briffa 2000 and Briffa et al 2008 was simply to “reprocess” Hantemirov and Shiyatov 2002. (See quotations above.)

As always with CRU, one has to watch the pea.

One of the purposes of Briffa (2000) was clearly to demonstrate the effect of RCS methodology on the Hantemirov and Shiyatov 2002 dataset. I have no objection to CRU claiming this “purpose” for Briffa (2000).

But, by 2008, this was no longer their “purpose”. Indeed, one doubts whether the editors of Phil Trans B would have accepted a 2008 paper with such a mundane purpose. The actual “purpose” of Briffa et al 2008 is stated quite clearly and was entirely different: it introduced and discussed “regional” chronologies. In accordance with that purpose, the datasets for the Taimyr and Tornetrask regions were dramatically expanded from 2000.

CRU’s evidence to Muir Russell on the “purpose” of Briffa et al (2008) was deceptive and/or untruthful. [Update May 11 - Gavin Schmidt agreed that CRU's evidence here is a "mis-statement" but characterizes the "mis-statement" as only "slight" and rejects the idea that CRU would intentionally mislead Muir Russell. Be that as it may, the evidence as it stands was untrue on this point and has not been corrected.]

Insufficient Time
Nor was CRU consistent with their excuses, even within individual submissions to Muir Russell.

In their June 17 submission to Muir Russell, after previously making the implausible and untrue claim about reprocessing, CRU admitted that the “purpose” of Briffa et al 2008 was to present regional chronologies and that they had, indeed, planned to do one for the Yamal-Urals region. However, they claimed that they didn’t have enough time to complete it and therefore reverted to the small Yamal dataset:

When we later received a request to submit a paper to a planned themed issue of the Philosophical Transactions of the Royal Society about ‘The boreal forest and global change’, Briffa and colleagues decided to use some of the material to hand in preparing a draft. It was intended that this should describe 3 continuous 2000-year ring-width series, each originally planned to represent the integration of a large-regional data set of subfossil and living tree data. The focus was to be on representing large-regional growth signals and initial comparisons with equivalent regional temperature data. The western, ‘Fennoscandia’, series would incorporate near tree-line pine data from northern Sweden and Finland; the Avam-Taimyr series would integrate larch data from near the Taimyr peninsula tree-line region. Between these we had intended to explore an integrated Polar Urals/Yamal larch series but it was felt that this work could not be completed in time and Briffa made the decision to reprocess the Yamal ring-width data to hand, using improved standardization techniques, and include this series in the submitted paper.

The “explanation” of insufficient time is completely implausible. They had already calculated a Yamal-Urals regional chronology: they could have used the one at hand? And why did they tell Muir Russell that couldn’t “complete” the chronology in time, when they had already done the calculation. If they felt that there were technical issues that disqualified the regional chronology calculated in 2006, why didn’t they report these problems (along with the regional chronology itself) in the article itself and/or to Muir Russell.

Ironically, in East Anglia’s FOI refusal, they argued that release and use of “incomplete” data could result in “incorrect or misapplied conclusions” (apparently disregarding the implications for the admittedly incomplete Yamal chronology applied in Briffa et al 2008):

There is little public interest in the release of unfinished or incomplete data – i.e. does not contain a description of how it was created or why the “selected” methods were chosen – and so does not reflect the full breadth of academic rigour and thought applied to it. The information may well be incorrect, untested, unreviewed and may not accurately reflect the proper outcome of the research. Incorrect or misapplied conclusions could be drawn from the publication of unfinished data and any assessment of the merit of the work should be based upon a final, approved version of the data.

In any event CRU’s evidence that they couldn’t complete the regional chronology in time is not credible on its face. The question of the regional chronology was raised in my submission and Muir Russell’s negligent failure to examine this topic means that questions that should have resolved continue to linger.

Polar Urals
Instead of disclosing to Muir Russell that they had already calculated a Yamal-Polar Urals regional chronology (and explaining their reasons for withholding it), CRU spent much of their energy in their submissions attacking my supposed endorsement of an update to the Polar Urals chronology.

The issue in older Climate Audit posts was the failure to report the updated Polar Urals chronology, given both the lack of 1000-year chronologies and the discrepancy between the two chronologies in the modern period.

However, after the Climategate dossier proved the existence of a Yamal-Urals regional chronology, the principal issue was the failure to use the Yamal-Urals regional chronology in Briffa et al 2008 or to otherwise report it (not a beauty contest between Yamal and Polar Urals). As noted above, CRU had noted their use of “common high-frequency variability” as their test whether to include data in a regional chronology. Polar Urals and Yamal have strong common high-frequency variability, expressed by Hantemirov and Shiyatov (2002) as follows:

Initially, a 1250-year chronologyfrom the Polar Ural mountains (Shiyatov, 1995) was used as a ‘master’ dating series as well. There is a high degree of similarity in the annual variability of radial tree growth between the Yamal and Polar Urals areas because of their proximity (about 200 km).

CRU was well aware of the common high-frequency variability between the two sites: this was presupposed in their development and analysis of the “Yamal-Urals” regional chronology. Astonishingly, instead of reporting on these results in their October 2009 website article or in their submissions to Muir Russell, CRU claimed that they had “never undertaken any reanalysis” of Polar Urals: (June 17, p.7):

We had never undertaken any reanalysis of the Polar Urals temperature reconstruction subsequent to its publication in 1995.”

Obviously this assertion is refuted by the presence of both the original Polar Urals data and the “update” in the FOI list.

“All the Data”
As shown above, there are obvious and visible difference between the regional and Yamal chronologies. However, in their website response to Climate Audit in October 2009 and to their submissions to Muir Russell, CRU claimed that there was little difference between their Yamal chronology and a chronology using “all the data”. How can these contradictory claims be reconciled?

The graphic below shows compares core counts of CRU’s “all the data” (violet) to the core counts of the Yamal-Urals regional chronology that CRU had calculated in 2006. Evidently, contrary to their representations, CRU did not use “all the data” after all.

Figure 5. Core Counts: Yamal (cyan); additional cores in CRU 2009 (violet); withheld 2006 regional chronology (salmon).

Conclusion
In their original statement on Yamal, CRU stated:

We would never select or manipulate data in order to arrive at some preconceived or regionally unrepresentative result.

Undoubtedly this is how they think of themselves. But their history shows that they have had a strong sense of what their results “should” look like and have, on other occasions, selected and manipulated data so that their results accord with “preconceived” results.

The “Briffa bodge” was a completely arbitrary “adjustment” of Tornetrask MXD data so that the answer made ‘sense’. The “Briffa bodge” was the predecessor to the “very artificial adjustments” described in Climategate source code documents.

Likewise, CRU’s decision to “hide the decline” by deleting MXD data after 1960 was evidently done so that the MXD temperature reconstruction accorded with preconceived ideas. This was done by CRU themselves and was a different manipulation of data than “Mike’s Nature trick (as described in more length in previous CA posts.)

Had their Yamal-Urals regional chronology had been in accordance with their previous results, I am completely convinced that they would have used it in Briffa et al 2008 and/or their October 2009 online article without a second thought. My surmise is that the apparent failure of the (still withheld) Yamal-Urals regional chronology to accord with their expectations caused CRU not to use it in Briffa et al 2008. I realize that this is a harsh statement, but it’s what I think. Muir Russell had an opportunity and obligation to investigate, but unfortunately failed to do so.

Postscript
The response of IPCC defenders is always that none of this “matters”, that they can “get” a Stick using other data and that CRU’s withholding of the Yamal-Urals chronology is no more than punctuation errors.

Climategate 2 emails contain an interesting vignette between Mann and Andy Revkin on this point. Revkin relied on the “without tree rings” reconstruction of Mann et al 2008 to re-assure himself that the picture was “solid”

needless to say, seems the 2008 pnas paper showing that without tree rings still solid picture of unusual recent warmth, but McIntyre is getting wide play for his statements about Yamal data-set selectivity.

As CA readers know, Mann’s “solid” reconstruction without tree rings was a mirage: it depended on Mann’s use of contaminated Tiljander data, used upside down. In the SI to Mann et al 2009, Mann conceded that his no-dendro reconstructions did not validate without contaminated data (but, unfortunately, did not notify PNAS or retract the earlier paper). Nor did Mann notify Revkin of the then pending admissions, instead allowing Revkin to continue to believe that the Mann et al 2008 no-dendro reconstruction was “solid”. In his recent book, Mann made no reference to the apparent concessions on the invalidity of Mann et al no-dendro reconstructions, instead claiming that Mann et al 2008 had used “objective” methods to validate the contaminated data.