For the very first time, the Climategate Letters “archived” the deleted portion of the Briffa MXD reconstruction of “Hide the Decline” fame – see here.
Gavin Schmidt claimed that the decline had been “hidden in plain sight” (see here. ). This isn’t true. The post-1960 data was deleted from the archived version of this reconstruction at NOAA here and not shown in the corresponding figure in Briffa et al 2001, though pre-calibration values were archived in a different NCDC file here. While the decline was shown in Briffa et al 1998 and Briffa 2000, it was not shown in the IPCC 2001 graph, one that Mann, Jones, Briffa, Folland and Karl were working in the two weeks prior to the “trick” email (or for that matter in the IPCC 2007 graph, an issue that I’ll return to.) For now, here is a graphic showing the deleted data in red. A retrieval script follows.
Figure 1. Two versions of Briffa MXD reconstruction, showing archived and climategate versions.
The relevant IPCC 2001 graph, shown below, clearly does not show the decline in the Briffa MXD reconstruction. Contrary to Gavin Schmidt’s claim that the decline is “hidden in plain sight”, the inconvenient data has simply been deleted. The reason, as explained on Sep 22, 1999 by Michael Mann to coauthors in 938018124.txt, was to avoid giving “fodder to the skeptics”. Reasonable people might well disagree with Gavin Schmidt as to whether this is a “a good way to deal with a problem” or simply a trick.
Figure 2. IPCC 2001 Fig 2.21 showing Briffa, Jones and Mann reconstructions together with HadCRU temperature.
Climate Audit 
91 Comments
##COMPARE ARCHIVED BRIFFA VERSION TO CLIMATEGATE VERSION
#1. LOAD ARcHIVED DATA
url<-"ftp://ftp.ncdc.noaa.gov/pub/data/paleo/treering/reconstructions/n_hem_temp/briffa2001jgr3.txt"
#readLines(url)[1:50]
Briffa<-read.table(url,skip=24,fill=TRUE)
Briffa[Briffa< -900]=NA
dimnames(Briffa)[[2]]<-c("year","Jones98","MBH99","Briffa01","Briffa00","Overpeck97","Crowley00","CRU99")
sapply(Briffa, function(x) range( Briffa$year[!is.na(x)]) )
# year Jones98 MBH99 Briffa01 Briffa00 Overpeck97 Crowley00 CRU99
#[1,] 1000 1000 1000 1402 1000 1600 1000 1871
#[2,] 1999 1991 1980 1960 1993 1990 1987 1997
Briffa= ts(Briffa,start=1000)
#2. LOAD CLIMATEGATE VERSION
loc="http://www.eastangliaemails.com/emails.php?eid=146&filename=939154709.txt"
working=readLines(loc,n=1994-1401+104)
working=working[105:length(working)]
x=substr(working,1,14)
writeLines(x,"temp.dat")
gate=read.table("temp.dat")
gate=ts(gate[,2],start=gate[1,1])
#Comparison
briffa=ts.union(archive= Briffa[,"Briffa01"],gate )
briffa=window(briffa,start=1402,end=1994) #
plot.ts(briffa)
X=briffa
par(mar=c(2.5,3,2,1))
plot( c(time(X)),X[,1],col=col.ipcc,lwd=2,ylim=c(-1.2,.5),yaxs="i",type="n",axes=FALSE,xlab="",ylab="")
for( i in 2:1) lines( c(time(X)),X[,i],col=i,lwd=1)
axis(side=1,tck=.025)
labels0=seq(-1,1,.1);labels0[is.na(match(seq(-1,1,.1),seq(-1,1,.5)))]=""
axis(side=2,at=seq(-1,1,.1),labels=labels0,tck=.025,las=1)
axis(side=4,at=seq(-1,1,.1),labels=labels0,tck=.025)
box()
abline(h=0)
title("Hide the Decline")
legend("topleft",fill=2:1,legend=c("Deleted","Archived"))
OT but exciting
Looks like Turnbull wont last the day and KRudds ETS is scuttled
http://www.twawki.wordpress.com
Nice work to find the data that quickly.
At this point I’d be grateful for a reminder – which part of the temperature record (or so-called record) did Briffa use to calibrate his purported tree thermometers?
Steve: 1881-1960
Steve I’m a bit confused by this post. Are you saying that the tree ring proxy data is a more accurate reflection of the true temperature record then the instrumental data? Otherwise what exactly is the point of plotting data that is known to be erroneous and that does not match the instrumental record?
As to your challenge, if I’m reading the graph correctly then Briffa, 2000 (the green line) stops in about 1960. So what is the issue exactly?
-ps- I’m just an interested layman so could have completely misunderstood the point of your post.
Is there any significant trend in he Briffa MXD reconstruction (Figure 1) if the deletion isn’t performed?
http://www.americanthinker.com/2009/11/crus_source_code_climategate_r.html
briffa_Sep98 code appears to be applying a heavy self-described “artificial correction” to data points that are used, but this code may not have been used by Briffa in his 2000 reconstruction shown in Figure 2.
Forgive my ignorance, but where does the zero on the y axis derive from? If, in the past 1000 years, temperatures have been below “average”, and in the last few years have been “above average”, even using their data, WTF?
Mr. McIntyre, hats off! Maybe there is hope for real climate science to emerge sometime in the future after all.
Thanks for answering that Mr. McIntyre. I’d asked Ace in the Harry ReadMe File thread for a translation of “hidden in plain view” but got zilch. I’m starting to get the feel for climate-speak. I now know that “plain view” means deleted. My personal favorite climate-speak is still “prestige” which means “gullible” (1254259645.txt).
As a skeptic I now realise why I’ve felt undernourished all these years. How good that makes the offering taste this Thanksgiving. So yes, fervent thanks for Steve, for the whistleblower of UAE and for anyone else that’s played their part. Mann’s inhumanity to man can be left for another day. For now, I’m simply enjoying the fodder.
Steve: 1881-1960
So, do I understand correctly — they have to truncate the output at 1960, because it diverges — doesn’t that mean that they could not use any data at all following the calibration period?
Huh?!!
Can;t see Fig. 1. My browser or ?
So just to make clear. Is this how the hockey stick graph should have looked in the IPCC report?
The deleted material in the upper figure doesn’t seem to hide a decline, because there is a steep decline before 1960 which was not deleted and “temperatures” for 1900, 1960 and 1990 are all about the same.
However the IPCC 2001 graph fails to show the steep decline before 1960. Is this just an artifact of the smoothing method they used? It seems you could get whatever 20th century results you want by selecting the appropriate method for smoothing (and deleting) the data at the edge of the graph.
Was the full dataset (including the deleted data) published, in full, anywhere else?
Steve: I’ve never previously seen a complete digital version of the Briffa reconstruction until the Climategate Letters. There is a related series that does go to 1994 but it’s not precisely what was used.
As a newcomer to this debate, the thing that most bugs me is: How can the tree ring data have ever been considered a good temperature proxy in the first place? It’s so obviously different from the instrumental record since 1960. I’m sure this has already been argued back and forth ad infinitum on all the blogs… anyone got a link to Briffa’s justification for using the data this way?
From the band “Toto”, 1979, I give you the satirized version of the song “Hold the Line”:
Hold the decline
Warming isn’t always on time
Oh, oh, oh
Hold the decline
Warming isn’t always on time
Oh, oh, oh
Somebody please make a YouTube music video of this! 😀
jorgekafkazar:
Cheers, Steve.
Thank you for your diligence.
Other than “whistling past the graveyard”, does anyone know why they seem confident that the “divergence problem” doesn’t exist anywhere else in the pre-instrumental record? For instance, if reaching a certain threshold of temperature is what causes it, couldn’t it also be impacting why they don’t see the MWP as being as warm as all the contemporary reports indicate? I wonder what it would look like to figure out the amplitude of the post-1960 divergence and then apply that same correction to the MWP?
There are so many layers of adjustment, homogenization and other fiddling in CRUTS (and GISTEMP) that I have to wonder….
What if the divergence problem is in CRUTS, not in MXD?
What if MXD is correct?
I don’t think anyone has the answer for certain, but that doesn’t mean the question shouldn’t be asked.
Tim S., something like it has already been done:
Although they seem to have interpreted “hide the decline” to refer to real temperatures rather than the downward trend in tree-ring temperature reconstructions.
well that’s some honesty and integrity in Climate Science.
No wonder the Team like to “Self Peer Review’ so much.
Wouldn’t want just ordinary people to share the experience.
Jorge:
I got the code to work nicely in R. be sure to uncomment this line:
#readLines(url)[1:50]
ie, if you change the above to
readLines(url)[1:50]
it all works
I have to say I think taking all this seriously is a waste of everyone’s time. I do not believe that you can extrapolate the world’s temperature from a bunch of tree rings in Siberia. This is pseudo science. It’s on a level with Astrology and Tarot readings. In fact Astrology is better, it has a perfectly sound mythological basis.
Why are we even thinking about wrecking the world economy on the say so of people who read tree rings?
It is jaw dropping that people take this seriously.
What I find interesting is they truncated the data in the middle of linear decline. i.e. the density steadily decreased from 1940 until almost 1990 yet they assume that the trees stopped responding to temp around 1960 and started responding to something else in a way that happened to exactly match the response to temperatures prior to 1960?
Looks implausible to me.
The government has to get some independent statisticians looking into this. All data must be released and scrutinized by an independent group of statisticians and scientists. The Urban Island effect must be closely examined, and everything must be audited back to the very, very start by teams of smart people.
If Mann and Jones want to save themselves they must open all the books and everything must be looked at. With some money,people and TRANSPARENCY all this can be done in not that long a time period. It could have been done already without the STALLING and OBFUSCATING of Jones,Mann and Gavin. I’m sorry but I have never heard of such a bunch of selfish people in my life, willing to literally risk billions of yours and mine hard earned money to keep their fantasy and egos alive. Unbelievable. They share the same mindset as power hungry dictators.
Hello Gavin:
Since I cannot post at RC, I have a question, “is (since it is ‘hidden’ in plain sight)”, the same as, “I see the invisible gold clothes the King is wearing”?
Do I have to have a clear or CRUked conscience to see it?
Steve, thanks for the links, I’ll have something to play with later tonight. Right now I’m looking for one of my standard analysis sheets (excel based – I typically run this before I use R, unless the data set is too large). I would have looked into all this before, but tracing back from the IPCC report a while back led me to a dead end – no data to play with.
Thanks again.
Shona-
I wouldn’t expect a strong correlation between temperature and tree growth rings, but there’s no doubt a correlation can be made. I’d expect large error bars associated with this conversion.
OT / Offtopic?
Hmmm…
Michael Mann @ bbc
http://news.bbc.co.uk/2/hi/science/nature/8381317.stm
“Most climate models used by the Intergovernmental Panel on Climate Change (IPCC) show that the Earth will respond in an El Nino-like way to global warming.
But a few of the models do recreate this dynamic “La Nina effect”, and suggest that that when you heat the Earth’s surface, the climate system tries to offset and cool. “
proposed terminology adjustments:-
Peer review -> crony review
CRU databases -> CRU datamidden
divergence -> oh bugger, it doesn’t work
the science is settled -> travesty.
Raven,
In the middle of the early code, the suggestion was made to truncate in 1940 also. Not sure what difference it makes but thought you’d like to know.
“In the middle of the early code, the suggestion was made to truncate in 1940 also. Not sure what difference it makes but thought you’d like to know.”
Any idea why that would have been suggested?
Steve: The MXD data peaks around 1940. I guess 1960 was a sort of compromise.
Nice work, Steve!
There is also something funny about how the Briffa data was smoothed. The unsmoothed data in Figure 1 is uniformly warmish from 1400-1550, with the first few years a little cooler.
Yet the smoothed green series in Figure 2 shows 1400 as distinctly warmer than the rest of 1400-1550. I don’t see how this can happen, not even with Mann’s double flip endpadding!
Jeff,
Thanks. Just more evidence that the ‘divergance’ is something they pulled out of hat.
I’m having trouble replicating the IPCC Figure even with the digital data.
There’s an interesting endpoint issue : the IPCC figure has been smoothed and has been smoothed to the endpoints. It looks like they substituted “projected” data for actual data in smoothing the Briffa reconstruction.
Alan Shore, the real question is whether the proxies actually represent temperature at all. If it can’t accurately characterize 1960-current, why should we believe that it can handle pre 1880 temperatures either?
Hiding data that call into question the proxy isn’t just a “trick” for climate denalists, it prevents other scientists from making an independent judgment of the quality of the data and the plausibility of the explanation for divergences.
In the HARRY readme file, there is a complaint about the station COATZACOALCOS (the place where the snake hides) in Mexico, where the mean monthly temperature in november 1995 was 78 degrees (Celsius one presumes). Almost hot enough to boil your blood, yet it was not reported in the news at the time. Had Harry been a scientist, he would have wondered about this. Anyway, in the master.dat.com file in the subdirectory douments/cru-code/f77/mnew, which seems to be a collection of temperature data for 3526 stations, the november 1995 data reads as 25.7 degrees Celsius (which converts to 78.3 degrees Fahrenheit), but the publicly available GISS record reads 999.9 – meaning no valid data available.
From what I understand, there is overlap between HadCRU data and GISS data, as far as stations is concerned. Obviously, in that case, the data should be the same. But this is not the case! OK, this may be an isolated occurrence, but in the 3 lines of data for this station in HARRY’s file, there are 3 discrepant numbers between the three sources of data I mentioned, for only 36 data points. I do not have the time to check how frequent such discrepancies are, but it may be worth looking into.
I therefore welcome Steven’s effort to get the raw data in the public domain, so that everybody can check the database on its integrity. There is no reason for the CRU people not to release the publicly available time series (the tiny fraction of the data which they can apparently not release because of collaborative agreements should not matter that much).
This is an important question about this proxy. This is why it is important that they truncated the display. If they had treated it honestly, they would have had to admit that this is a very poor proxy…and removed it completely from the graphic.
For others who are new to all this… remember: the Hard Question is not whether it is warming today.
The Hard Question is: is today’s climate unusual? To answer that, we need histories and data proxies for past climate, to tell us what happened 400, 1000, 2000 years ago.
What the embarassing deleted data shows is that this tree ring proxy can’t predict modern climate, so it can’t be trusted for past climate.
MrPete: v
Very clearly put. Thanks on behalf of all newbies in these parts – and this one deserves an influx to put past server-stopping spikes in the shade.
Dave:
One of the many things that made me rejoice when I first saw this a couple of hours ago is that it made clear that ‘hide the decline’ is indeed a worthy catchphrase for the whole climategate car-crash. Most ‘laymen’ do I’m sure take it to refer to trying to hide the downward line (if not trend) in temperatures since 95, 98, whenever. The slightly more sophisticated perhaps see it as the attempt to hide the fact that the models didn’t predict such a disappointing postscript to the hockey stick. A not unimportant point. In fact we see here that it had to do with hiding the fact that trees are not good thermometers, surprise surprise. Either of three ways, it’s unethical and it’s stupid, because in the end you get found out. But hats off to the man who once again did the finding.
Consensus -> collusion
Robust -> flimsy
Model -> attempt to prove the infinite monkey theorem
”
What the embarassing deleted data shows is that this tree ring proxy can’t predict modern climate, so it can’t be trusted for past climate.
”
It apparently becomes uncorrelated (or perhaps negatively correlated) for the recent past. Predicting is a very different question. From a strict statistical point of view, empirical models cannot validly be used to extrapolate outside of the data space – therefore none of this could be used for projecting future trends in the first place. On the other hand, if there were fundamental physical laws translated into a model, it becomes a matter of fitting parameters to the fundamental physical model, and projections would be valid within the validity of the physical laws. In the case of climatic events, the fundamental physical laws are so far removed from the macro environmental variables which are measured that it is a mathematical / computational impossibility to make the connection.
All these models that these scientists are coming up with are fundamentally empirical models with the accompanying limitations. Even if the correlation between tree ring growth and temperature is perfect and the values are known to an arbitrary number of decimal places, the models cannot be used in the fashion they are being used.
In the interest of full disclosure, I’m a Physicist, not a Statistician, but I learned some decades ago to listen to PhD Statisticians.
Just to chip in on the skepticism on using tree rings as thermometers. I’m not a climate scientist, but the system should contain at least the following interdependencies:
Tree growth = f (temperature, sunshine, rain, humidity, CO2 )
temperature = g (sunshine, CO2, rain, humidity)
humidity = h (temperature, rain, sunshine)
CO2 = j ( temperature )
The only thing that they might assume as known (with limited precision) would be CO2 (from ice drilling, with limited precision). That leaves rain, humidity and sunshine as unknowns – how can you determine temperature without also determining these? Statistical “validating” alone isn’t good enough here, and with the “divergence problem” (i.e. empirical failure) already at their hands it is quite breathtaking that their crap ever got published anywhere.
It is probably worth reminding people that there is peer reviewed literature that disputes the teams claims about the divergance:
http://www.springerlink.com/content/45u6287u37x5566n/
Of course it probably did not make the cut when the team redefined what peer reviewed literature is.
Worse than we thought!
Re: Steve McIntyre
There is something very odd about the smoothed trace from IPCC report. The start from 1402-1410 has a pronounced downward slope despite the absence of any such trend in the email data. In addition, the decline from 1940 to 1960 drops less than 0.1 deg C. If you use the full set from the email to smooth to the 1960 endpoint (using a 25 year average, for instance), the drop is greater than 0.2 deg C. It also drops to a level comparable to that seen in 1900. The IPCC trace is well above the 1900 level.
They padded the 1402 and 1960 endpoints with something to smooth the data, but what?
Steve: I’m trying to figure it out.
I’m sorry Steve, but this graph from Nature clearly shows what Gavin means when he says “hidden in plain sight” – the green line stops at 1960, therefore it is clear to anyone (who knows what they are doing), simply by looking at the graph, that some data has been ‘hidden’.
Challenge:
Rather than posting misleading and duplicitous posts such as this one, that simply display your own ignorance and unethical behaviour, why don’t you tackle the divergence problem itself? Either tell us why there is no divergence problem, in which case data deserves to be included, or explain why the divergence problem brings into disrepute all the tree-ring data.
— Layman
Steve: In AR4, I asked them to show the decline. They refused. Have you read the correspondence on their decision to hide the decline?
layman,
Look at my post above. It references a peer reviewed paper that demonstrates that the divergence problem makes all past data suspect.
Why don’t you start there and see if you can explain what that paper’s conclusions are wrong.
If you pad the ends with zeroes and run a 20 year moving average, the ends look awfully similar to the IPCC trace. I’m sure it’s not that simple, but it makes one wonder.
Steve: Maybe they used a Butterworth filter instead of a Hamming filter. Mann uses Butterworth filters in other publications.
dearieme,
At this point I’d be grateful for a reminder – which part of the temperature record (or so-called record) did Briffa use to calibrate his purported tree thermometers?
Steve: 1881-1960
====================
If Briffa calibrated against 1881-1960 then is it against the raw temperature data prior to ‘corrections’ by HadCRU and GISS or afterwards?
Well, maybe it is that simple except they use a more sophisticated filter than a moving average. Since this is an anomaly plot, they may have rationalized putting zeroes at the end since that is the reference value.
Please do not lose site of the fact that the data is only part of this. We also need to continue the presure on getting ALL the data – both in and out and most importantly a repeatable METHOD. Presure must be maintained on all those who do not play by the rules of normal and acceptable science processes. Only the Method will unveil the secrets held within. Until the method is passed into public domain any reverse engineering is questionable and easily shot down with simple motherhood statements. Whilst code comments are revealing – they still revel nothing of substance regarding the method. Plain english assessments of the methods used must be made and published. See the NZ temp data issues. They are refusing to reveal the method to ALL the adjustments. NZ is a small number of sites. Start there and move out……to my mind it is the observational temp readings and the mass manipulation and coersion of data to form a premeditated picture that is the smoking gun. KISS in the hunt.
Jeff C., It would appear rationalization is their specialty.
layman:
It sounds like your problem, not Steve’s. Sure, he’s finally unveiled it, after a mere eleven years, because you wouldn’t let him have the data and he had to wait until the accumulated stench caused one of your own to finally break ranks and leak it. You call the man unethical. I’m sure in your terms he is, just as the Wall Street Journal is being unethical to read the last rites for cap and trade today, on the back of all this. They also happen to be right. You’re losing. And soon you will have to face the wrath of the duped masses, whose view of the ethics has already changed beyond recognition this week. You haven’t seen anything yet.
Layman, I’m not sure if you’re being deliberately obtuse, so I’ll keep it reasonably simple. If dendro MXD (in this case that’s what we are referring to) proxies do not accurately reflect measured temperatures (which you assert is correct), what is the source of your confidence that they have not done so in the past ? I strongly suggest that neither you nor anyone else can provide any proof either for or against such a historical divergence. Therefore historical reconstructions based on such proxies must be inherently unreliable.
This is a simple “trick” unworthy of serious consideration. The claim is that since 1960 MXD measurements do not reflect true (local, and there’s another can of works in that as well) temperatures, however they always have in the past 1 – 2,000 years. There is a proposed anthropogenic mechanism that causes the recent disconnect which although currently unknown has never applied before. Now, to rely on MXD proxies one must prove that the above supposition is at least most probably true, why don’t you indicate how you might provide proof of that ?
Layman-
”
Either tell us why there is no divergence problem, in which case data deserves to be included, or explain why the divergence problem brings into disrepute all the tree-ring data.
”
This is an easy one. There has never been a correlation established between tree ring growth data and temperature data for the period 1000AD – 1600AD or so. Any validation of the technique necessarily must rely on periods for which there are actual temperature readings, ie, relatively modern times. Excluding data because they don’t fit should be done if and only if an actual cause has been established with some certainty, not if a cause is proposed or supposed or postulated. In this context, the recent divergence invalidates the tree ring growth data as a useful proxy for temperature.
In other words, the data should be handled with generally accepted good practices.
Layman,
You’ve got to move the thimbles faster; we know where the pea is.
I believe that Layman also doesn’t understand the scientific method. It is not the reviewers job to fix the mistakes of the person, that made them in their original Hypthothesis, that is their problem. The original proponents need to go back and rework their Hypthothesis and start over.
[p]Carrick
Alan Shore, the real question is whether the proxies actually represent temperature at all. If it can’t accurately characterize 1960-current, why should we believe that it can handle pre 1880 temperatures either?
Hiding data that call into question the proxy isn’t just a “trick” for climate denalists, it prevents other scientists from making an independent judgment of the quality of the data and the plausibility of the explanation for divergences.[/p]
[p]Is this another way to say GIGO?[/p]
Robust – I remember Microsoft referring to Windows NT as “robust” which we interpreted as complicated, bulky, and buggy as we all waited for Service Pack 7 to no avail.
Hello Mr. McIntyre, I was curious if you had tried to see what happens when you apply the corrections applied to the instrumental record to the proxy record. Am I just being paranoid here? Does it bear any resemblance to the instrumental record?
@Andy: proposed terminology adjustments (continued):-
complete transparency -> hide the decline.
Well, the complete graph tells a very different story and it is a more true story, too. This is the story what the trees – players that should actually be affected by the hypothetical catastrophic changes – have been doing. They were recently doing nothing special and they even show behavior that would be expected from a cooling.
Trees are bad temperature proxies but in some sense it’s true that the behavior of trees is more important for us than the behavior of thermometers. And what the trees were recently doing is kind of more important for the true environmentalists.
I’m a sceptical layman trying to understand the post 1960 divergence and its significance for the wider argument, am I right in thinking that the post-’60 divergence OUGHT to have come like the thirteenth stroke of a clock – which casts doubt on all its predecessors?
In a temperature reconstruction of millennial scale we have, in chronological order, three cases:
AD1 – c1720 – no instrumental record, proxy record only;
1720 – 1960 instrumental record AND proxy record available and CONCUR;
1960 – present – instrumental record AND proxy record available and DISAGREE
Appropriate conclusion – until post ’60 divergence understood agreement between proxy and actual cannot be assumed to invariable for the pre 1720 period.
These guys’ conclusion – hide the post ’60 divergence and use the pre 1720 proxy data as though agreement between it and reality were invariable.
Sorry if this is a bit clunky (or just wrong!) but I suspect that you are about to encounter the sceptical laiety in increasing numbers, so searching for ways of making this stuff digestible at no cost to accuracy may be of some value!
The real temperatures probably aren’t real either. I think it’s a spiel. proxies aren’t perceived to be as accurate as the ‘real’ thing (ie. measured temperature). Once the temperatures were fixed/adjusted/fiddled and accepted they had to find a way to deal with the divergence. This has to be done in the data that is perceived as least accurate, thus in the proxies. Could it be that there wasn’t enough leeway in the treering data to get enough of an up-turn in the 1960-to-present time? And that that’s the reason they grafted on the ‘not so real’ temperature data at the end?
Looking through the code and other documents, the most charitable conclusion I come to is there is a lack of rigor in all this.
The most disturbing here is that if they tried to hide the decline because there was a mismatch with the temperatures, then the temperature record they used matches the local climate of these tree sites very poorly. Several of the long-running stations in Siberia in the vicinity of Yamal and the Polar Urals site have temperature records that are not that far from Steve’s “resurrected” proxy graph at the beginning of this article. Here’s one example: I downloaded the June-July-August temperatures from GISS for the station Ostrov Dikson (quite close to both Yamal and the Polar Ural sites) and averaged with a 3-year period to remove some of the noise: http://i45.tinypic.com/2ns6jk6.jpg
There’s absolutely no doubt about the decline here. Please note that the Briffa series ends in 1994 (correct me if I’m wrong), I’ve included all years up to 2008 (i.e. the average for 2007-08-09) just for the record (and so you can see that the current period in the Arctic is indeed warm, but not unusual even in modern times).
Luboš Motl wrote:
Well, the complete graph tells a very different story and it is a more true story, too. This is the story what the trees – players that should actually be affected by the hypothetical catastrophic changes – have been doing. They were recently doing nothing special and they even show behavior that would be expected from a cooling.
Trees are bad temperature proxies but in some sense it’s true that the behavior of trees is more important for us than the behavior of thermometers. And what the trees were recently doing is kind of more important for the true environmentalists.
_________________
I like this line of thought!
Why are we reducing trees to temperature proxies and then using highly suspect formulas (GCMs) to reconstruct past climates and model the present climate? The trees have already integrated all the variables! When and where they thrived, so did we. We’re actually interested in climate, not just temperature, right?
Many issues unresolved in this thread. I am a desultory lurker, so don’t know the responses to all issues, but here are a few quick tips:
– Badger: Zero reference period is 1961-90 (see y-axis legend).
– Shona and others : Tree rings are in fact poor temperature proxies. There are two tree-ring measures: density and thickness. Density is held to be more responsive to temperature, but still not great. Thickness responds to temperature, precipitation, disturbances (fire, glacier scrapes…), shadow from neighboring trees etc. etc. etc. Very high temperatures reduce thickness instead of increasing it. So the correlation between tree-ring thickness and temperature is poor, and “non-linear”, i.e. warm does not necessarily mean thick, because very warm starts getting thinner again. The “non-linear” response means that it is impossible to use “principal components analysis” (PCA), a form of statistical selection that prioritizes certain proxies because of their observed relation to temperature. Even so, Mann does use PCA to prioritize tree ring series in his temperature reconstructions.
– Several posters: Mann’s methods (but I think not, or not always, Briffa’s) pick out proxies according to whether they mimic hemispheric temperatures, not local temperatures. So a tree that has wider and wider rings when northern hemisphere temperature is going up gets selected as a“good proxy” by Mann, even if local temperatures around that tree are going down. This is Mann’s “teleconnection” fallacy, on which Climate Audit has a lot of material.
The combined effects of the two last points mean that Mann’s reconstructions are nonsense. What they do is select tree ring series that match recent rises in hemispheric average thermometer-measured temperatures. These prioritized series are then used to “reconstruct” past temperatures. Since the series selected are practically 100% noise, they simply revert to the mean as one goes back in time. The mean is the midpoint in the (recent) period over which they were matched to thermometer readings. Since there was a rise over this period, the endpoint is usually at a historical high in the proxy series.
The correlation period usually finishes some time in the last 50 years. But if the proxy series extends beyond the end of the correlation period, or even if the correlation period is just a long time, the proxies tend to be reverting to the mean near and after the end of the correlation period. This gives rise to the famous “divergence problem” where the proxy series appear to be cooling in recent years, whereas the thermometer-based readings are going up. Warmist scientists ponder this problem deeply, but it does not, in reality, exist, since the selected proxies (especially Mann’s) never measured temperatures in the first place. All his proxies show is 1000 years of noise disturbed by aligning the 20th century to rising 20th century thermometer readings.
Steve: If journalists are really accessing this site in droves, you need to put up a moron’s guide fast. Maybe Ross could help?
BTW if the Briffa data here are real, then Jones has actually diddled himself slightly in his explanation of “hide the decline” here: http://www.uea.ac.uk/mac/comm/media/press/2009/nov/homepagenews/CRUupdate. Briffa’s green line actually went back up again after 1980, though only towards zero, nowhere near up to the thermometer readings.
It appears that in the beginning, the tree ring guys were trying to work up some hypotheses about the correlation between tree rings and site conditions. In forestry this is usually referred to as the site index. A site index is usually very dependent on moisture. More moisture better growth. Soil types, slope and other factors also come into play.
Along the way, someone thought about trying to correlate all of this to temperature and then thought about connecting this to the historical tree ring record. It may be somewhat unfortunate that Briffa and Mann ever got connected to this. Clearly, in my mind, the science of dendrochronolgy as related to temperature, is still very immature.
re: Layman ( http://camirror.wordpress.com/2009/11/26/new-the-deleted-data/#comment-1613)
“Challenge:
…. explain why the divergence problem brings into disrepute all the tree-ring data.
– Layman”
Layman, taking your username at face value but assuming you can follow simple logic (or you wouldn’t be here):
They have a bunch of data from the rate trees have grown at in the past, which they can measure from the annual growth rings.
It seems logical that temperature might affect the rate of growth (but other things will as well). So they compare the rate of growth with known historical temperature measurements to see if they can find a fairly simple relationship between temperature and growth rate over a “calibration period”.
For the 79 years of the calibration period (in this case 1881 to 1960) they find that tree growth seems to relate fairly accurately to recorded temperature.
But they also find that, for the 30 years immediately after the calibration period (up to 1990), that relationship breaks down badly.
Despite the fact that the relationship only holds for 79 out of 109 years they decide that the trees have somehow “become faulty” since 1960 and that the relationship they’ve found can be safely extrapolated for over 1000 years into the past.
In other words, never mind the sceptics being wrong, by simply ignoring the later data, they’re actually arrogant enough to be saying that the TREES are wrong!
Just a laymen here.. can’t interpret any of this data or make sense of half of the comments but I have question. The temperatures in the place that I live have been recorded for decades. If I were to cut down the big tree in my back yard (probably 60years old) could I guess the temperature each year going by looking at the rings? Would I need a microscope or something? Should be pretty easy to verify if this ring reading really works since the temperature data is available at the local library.
I still think that an outside group of scientists who dropped in on this debate with no prior knowledge of climatology would, after some time perusing the dendrochronology data and the statistical manipulations used to “interpret” (for lack of a better word) it, would back away from what are obviously compounded series of fudges and kludges and say, “Come back in a few years when you’ve grown up, and maybe there will also be some better data to do real science with.”
And they might say (Hey speculation is what most of the AGW team seem to be allowed to do, so why not me?), “And, about this business of using your findings to tamper with the world’s economy and ecology: I think we’d all be no less well off if you didn’t try to do that just yet.”
And while in Copenhagen, don’t sit around the table and say anything. Take some time out. Go see the sights. That ‘Little Mermaid’ statue, for example, it’s been restored (again), hasn’t it. Go think about her sitting in a slightly warmer harbor. Don’t worry. Be happy.
Ok, so we are to take it that 1000 years of history is a precursor to the next 10-20 years?
Where is the data for the past 10 to 20,000 years? I was under the impression that our planet was 3 Billion years old.
What about the ice cap temperature data? Isn’t that relevant?
Deleting data because it is inconvenient, isn’t science. Neither is ignoring all sources of information available.
Perhaps you could help the Team find the instrumental data for 2001-2009 that is missing from their latest offering? (See Jean S comment at old CA.) It’s probably “somewhere on the internet” “hidden in plain sight”. Maybe Gavin knows? Or one of his fine gurus?
Mr. McIntyre, is it fair to say that you believe that the hockey stick as pictured in the IPCC report is actually in good shape, and that your dispute concerns the only the tree-ring proxy, which matches all the other proxies for the past, but diverges for unexplained reasons after 1960, at which point the instrumental record is more reliable?
Steve: Of course not.
I’ve been playing around with the Briffa series attempting to replicate the smoothed trace shown in the IPCC plot. It hasn’t matched exactly (discrepancies in the high frequency wiggle damping), but it’s getting close. I’ve only used filters that I have readily available (moving average, triangle, Gaussian, Hodrick-Prescott), but the Gaussian seems to be the closest match (no Butterworth available).
One thing that has matched almost exactly is the smoothed endpoints when using zeroes to pad the ends. Note the “reverse hockey stick” at the start of the trace (1402 to 1420) and the downward slope reversal at the end of the trace (1950 to 1960). There is nothing in the raw data at either end that suggests any sort of upturn. I believe they used zeroes to pad the ends of the series (pre 1402 and post 1960) when calculating the smoothed trace. They did this despite the fact that the mean of the 1402 to 1960 series is -0.33 degrees (an easier to justify selection if you must pad, IMO). Yes, zero is the anomaly reference, but that appears to be calculated using only a small portion of 20th century values.
Not only did they “hide the decline” by truncating post 1960 values, they used very questionable smoothing methods to suggest that the drop that started around 1940 was reversing itself by 1960. This is despite the fact the raw data to 1960 shows a continuous downward decline. Perhaps this was unintentional, but the shortcomings all seem to go in the same direction.
snip – off topic
Oops – I used common email marks that I should have recognized are HTML tags, the missing quote from Mike is “The only thing that they might assume as known (with limited precision) would be CO2 (from ice drilling, with limited precision).”
Here is a link to a plot of the smoothed and raw data. Note the inflection at the endpoints when compared to the raw data. By happy coincidence, the Briffa recon now has a reversal around 1960 similar to the other recons even though the raw data drops another 0.15 degrees between 1950 and 1960. What nonsense.
The last ten days have provided fascinating reading on this and other “climate sites”. Arguments for and against the ability of tree ring based proxies, MXD and TRW, to act as indicators of the climate local to the trees have been put forward. A group of climate investigators (I hesitate to use the term scientists) apparently have a (faith based?) belief that these observations mimic in some sort of quantitative fashion the actual temperatures observed for the globe as a whole. I understand that such behaviour is called a Teleconnection, but which seem to me to be roughly on a par with the teletransporters of science fiction, but no matter, they seem to be hard currency in the world of AGW proponents, so have to be treated seriously.
I have understood quite readily that the tree ring records are believed (known?) to respond to the conditions prevailing in the growing season, which appears to be May to September (give or take a month at either end) in the Northern hemisphere. Let’s say four warmer months. Thus any “calibration” attempts presumably use the appropriate instrumental record for these months as the gold standard – that is numbers that are known very precisely relative to those obtained from ring data. I believe that valid linear calibration normally explicitly presupposes no error in the gold standard.
I wonder about the treatment of instrumental records for those months that are deemed to lie outside the growing season. As a somewhat naive scientist I have always thought that the annual thermometric record for any site would also make use of the remaining, non-growing months’ data rather than being confined to those associated with summer. What if November to March happened to be very cold, or comfortably warm? Is their information to be ignored in our assessment of what the climate was like? These neglected data items might well contain some very important information. Incidentally, have any of you who download and analyse data from arctic regions noticed that many sites frequently have missing values for the months of deep winter? Typically, there are few if any data below -20C, whilst -19.x etc are well represented. Histograms illustrate this readily. This may be a health and safety issue I suppose, but it does not help in assessing yearly averages, many of which would strictly have to be regarded as “missing” because of some absent components.
Robin
“Steve: Of course not.”
Okay well then my question is, do you have a posting on this site (or anywhere else) which gives a general overview of your argument? I am not technically inclined, but reading just this post makes it appear that it is only one data set that is under question, not all of them. A link which gives your overall argument would be most helpful.
Steve: Go to the main site climateaudit.org. There are many discussions of various datasets.
Thank you. I couldn’t get through to it before.
I’ll apologize for using the same name as another commenter… or was it the author?
In any case I have been following this article since it seems to have data and science that makes actual sense to me.
http://www.globalwarminghoax.com/comment.php?comment.news.113
Global warming should be based on data that goes beyond that last 1000 years. Never mind the inconvenient data for the past decade or so.
If anyone can refute the argument on this link I’d be happy to hear it. If your only response is that the argument is settled – I’ll refer you to the “earth is flat” society.
OK, all the arguments about which data sets are used, whether the data is smoothed, hidden, or otherwise, is all very revealing – and does not help their argument for AGW.
As a non-scientist where do I find the basis, the explanation for the AGW theory that all of this is caused by Man and his CO2? Where in the hell is THAT data? What repeatable, reproducible experiments confirm the role of CO2, and does correlation equal causation in this case? What about all the CO2 emitted by Nature, doesn’t that play ANY part? Aren’t Man’s CO2 emissions only about 3% of what Nature contributes?
Aren’t real scientists by nature supposed to be skeptical and trying to find data that DISPROVES their pet hypotheses and not just looking for data that fits and throwing away data that doesn’t (I guess that question has now been answered)?
Can I suggest that you take a 1000 year “temperature record” that is random beteen -0.5 and +0.5 and filter it by convolution with a Gaussian profile (sigma=15 years)? Then calculate the the probability of lengths of above or below average temperature. (As a Monte Carlo problem)
You will find the reults surprising if you don’t know the answer already.
I have concluded that no matter how good the CRU crowd were at climate science, they have have a lot to learn about signal processing.
Richard (Saumarez) – I like your suggestion because it impacts on how I view climate data.
I would be really interested in your answer to this problem.
Are you suggesting doing a Runs test on the pseudo-random sample, or perhaps calculating a realisation of the occurrence of lengths 1,2,3,,,n (or something of that type) for one or (many) more trials.?
Robin
“Climategate” started out when there appeared on the Internet a collection of e-mails of a group of climatologists who work in the University of East Anglia in England. These documents reveal that some climatologists of international preeminence have manipulated the data of their investigations and have strongly tried to discredit climatologists who are not convinced that the increasing quantities of carbon dioxide in our atmosphere are the cause of global warming.
It is true that a majority of the scientists who study climatic tendencies in our atmosphere have arrived at the conclusion that the world’s climate is changing, and they have convinced a group of politicians, some of whom are politically powerful, of the truth of their conclusions.
A minority, however, is skeptical. Some believe that recent data that suggest that the average temperature of the atmosphere is going up can be explained by natural variations in solar radiation and that global warming is a temporary phenomenon. Others believe that the historical evidence indicating that the temperature of the atmosphere is going up at a dangerous rate is simply not reliable.
snip – too generalized
14 Trackbacks
[…] the whole story here! Comments (0) Trackbacks (0) Leave a comment […]
[…] New!! Data from the Decline […]
[…] warming hypothesis (and I am being kind still calling it a hypothesis as to a failed theory). Steve McIntyre has discovered the missing data which shows current temperatures are not any different from the first half of the […]
[…] this article, I spotted a related one over at Steve MacIntyre’s Climate Audit site, entitled New!! Data from the Decline, which demonstrates exactly what hiding the decline is all about. You should read that with this […]
[…] See the rest of this article here: New!! Data from the Decline […]
[…] Decline – Exposed! Posted on November 27, 2009 by rightwingliberal Steve McIntyre manages to undo “Mike’s Nature Trick” and reveals the decline that Phil Jones et […]
[…] Decline – Exposed! Steve McIntyre manages to undo “Mike’s Nature Trick” and reveals the decline that Phil Jones et […]
[…] There are some new postings from Steve McIntyre (and here) that show us the hidden decline in Briffa’s tree ring data – basically it […]
[…] Email Deletion http://camirror.wordpress.com/2009/1…mail-deletion/ Data from the Decline http://camirror.wordpress.com/2009/1…-deleted-data/ US Congress investigates Climategate e-mails: this could be the beginning of the end for AGW […]
[…] Climate Audit Mirror site has reconstructed the data from Briffa before the math tricks and outright […]
[…] temperature history. See the discussion by Jean S here, and follow-up posts by Steve McIntyre here and here. The emails contain many similar examples whose cumulative impact will take time to sort […]
[…] […]
[…] […]
[…] referee report contains some awful language that reminds us of the let’s not give fodder to the skeptics remarks in some climategate […]