Climate Audit

ali baba

Two things are missing in this post — the code and an explanation.

As far as I can tell, the comments document (1) a hack around a bug in IDL5.4 p_correlate and (2) the “trick”.

Apparently your correspondent Neal knows better but forgot to say.

Eric

UEA may be careless with archiving – from README_GRIDDING.TXT:
Bear in mind that there is no working synthetic method for cloud, because Mark New
lost the coefficients file and never found it again (despite searching on tape
archives at UEA) and never recreated it. This hasn’t mattered too much, because
the synthetic cloud grids had not been discarded for 1901-95, and after 1995
sunshine data is used instead of cloud data anyway.

I am also reading the various Fortran source files – would have been nice if they had heard of the concept of descriptive comments. I wonder if this code has ever been peer reviewed, verified or validated (I am a software engineer)?

• Uninformed Luddite
  Posted Nov 24, 2009 at 9:40 PM | Permalink

  Eric,

  You are a software engineer and you have to ask? I don’t even use the language but a quick glance at the code’s structure and perusing a few manuals makes it pretty clear that this looks more like the work of a script kiddie(with a pre-conceived result in mind) than the work of a real scientist. Although, the code may have been written by someone who hasn’t had any exposure to Comp. Sci. (although in that case you would expect the code to have been written by a professional).

  • Steve
    Posted Nov 25, 2009 at 7:33 AM | Permalink

    Obviously he has a preconceived result in mind. I am a computational physicist, and this is how all modern science works.

    It might not seem right to you, but this method of approaching problems actually works. If it didn’t, no scientific progress would have been made since about the time of Newton and we’d all still be living in the dark ages.

    Before you approach a problem esp. a computational one, you have a predicted outcome in mind based on theory. You then apply known rules to your program, and hope it’s results agree with your predicted outcome. If they don’t, you try to adjust your theory slightly, write these adjustments into the program, and so the cycle continues until you have agreement.

    This is certainly the way my field of physics is done, and I know from studying philosophy of science that this is the way all modern science is done.

  • UninformedLuddite
    Posted Nov 26, 2009 at 5:53 AM | Permalink

    As I cannot reply to your post Steve I will reply here

    Maybe I didn;t phrase what I meant that well. Some of the code(two verified with this behaviour so far) will output certain ‘preconceived’ results with a very wide range of inputs and appear to have been designed that way (with some very basic obfuscation(hence my perjorative ‘script-kiddie’ reference). Sorry I was not more clear.

  • UninformedLuddite
    Posted Nov 26, 2009 at 6:07 AM | Permalink

    Actually mate upon re-reading your response to my initial post just delete anything I have said on this blog(just a little to condescending for me). I promise not to return. You have been given several smoking guns and they aren;t in the emails. find them yourself.

Eric

From documents\harris-tree\recon_esper.pro:

; Computes regressions on full, high and low pass Esper et al. (2002) series,
; anomalies against full NH temperatures and other series.
; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N
;
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline
;

Note the wording here “avoid the decline” versus “hide the decline” in the famous email.

Eric

Another “avoid the decline” in documents\harris-tree\recon1.pro – only this time to avoid the decline after 1940 – there could be perfectly reasonable explanations for these so I am not drawing any conclusions:

;
; Computes regressions on full, high and low pass MEAN timeseries of MXD
; anomalies against full NH temperatures.
; THIS IS FOR THE AGE-BANDED (ALL BANDS) STUFF OF HARRY’S
;
; Specify period over which to compute the regressions (stop in 1940 to avoid
; the decline
;

Also appears in recon_jones.pro and many other files in this directory.

The file calibrate_nhrecon.pro gets more specific:
; Specify period over which to compute the regressions (stop in 1960 to avoid
; the decline that affects tree-ring density records)

The file plotagehuger.pro seems to be creating correlations at different time scales – including 20 and 40 year time scales.

Interesting – plotregions_instr2.pro:
; Early N. American ones seem biased to being too warm early on (due to
; using north facing walls rather than Stephenson screens; the walls get
; sunny during high latitude summer!). so cut the first two regions off:

recon_mann.pro:
; Computes regressions on full, high and low pass MEAN timeseries of MXD
; anomalies against full NH temperatures.
; THIS IS FOR THE Mann et al. reconstruction
; CALIBRATES IT AGAINST THE LAND-ONLY TEMPERATURES NORTH OF 20 N
; IN FACT, I NOW HAVE AN ANNUAL LAND-ONLY NORTH OF 20N VERSION OF MANN,
; SO I CAN CALIBRATE THIS TOO – WHICH MEANS I’m ONLY ALTERING THE SEASON

briff_sep98_e.pro:
;
; PLOTS ‘ALL’ REGION MXD timeseries from age banded and from hugershoff
; standardised datasets.
; Reads Harry’s regional timeseries and outputs the 1600-1992 portion
; with missing values set appropriately. Uses mxd, and just the
; “all band” timeseries
;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********

After doing that, this code then looks at both 20-year and 50-year components.

The code of all these files is very similar – mostly changing the data sets, it seems. From a s/w engineering perspective, having so much duplicated code can lead to maintenance, modification – and test and validation problems.

The complete 2001 Briffa tree ring data set is located in
documents\harris-tree\banding\b01abd_site.txt

Table of tree-ring sites used in

Briffa KR et al. (2001) Low-frequency temperature variations from a northern tree-ring-density network. J. Geophysical Research 106, 2929-2941.

Number of sites: 389

The code in this directory is also many files containing roughly the same code with minor changes.

• Calvin Ball
  Posted Nov 22, 2009 at 9:57 PM | Permalink

  If someone has an innocent interpretation of “;****** APPLIES A VERY ARTIFICIAL CORRECTION FOR DECLINE*********”, I’m all ears. Seriously.

  • Joe
    Posted Nov 24, 2009 at 2:54 PM | Permalink

    Calvin,

    I can. When I put a comment like that in my code amongst several other lines of non-capitalized / asterixed sentences it means “WARNING TO ANYONE READING THIS CODE, TAKE THE FOLLOWING WITH A GRAIN OF SALT, PLEASE DON’T MIS-INTERPRET THESE RESULTS AS REAL.” As in the code is meant simply to demonstrate something. Has anyone even figured out what (if any) end-product this code truely was used for?

• John Murphy
  Posted Nov 23, 2009 at 8:58 PM | Permalink

  Does anyone know whether this is the standard of coding used for the many GCMs floating about?

  If it is, we can have no confidence at all in them either, much less in the data they are “supposed” to be using.

Eric

documents/osborn-tree3/declineseries.pdf is a chart of numerous locations apparently showing ring density and widths.

For the density category, there is a divergence post 1940. For the width category (as I understand the chart labels), there is a divergence post-1960.

Some one familiar with tree rings may wish to help explain what these charts show and what they mean, and how the declining values may related to the name “declineseries” and perhaps the phrase “hide the decline”.

DRE

Keep looking at the code comments. I’ve found some interesting stuff there as well.

Holy &*%$*@ Batman I’d hadn’t seen this before!

; plot past 1960 because these will be artificially adjusted to look closer to
; the real temperatures.

Smoking howitzer?

“trick to hide the decline” perhaps?

Jim

*********************
crosspatch permalink
What the world needs someplace is a common standard temperature record database that they can all draw from where the data is standardized. It would be a very difficult job but once done, could provide some continuity and make future studies much easier and less expensive. You could have better assurance that people are comparing apples to apples and not constructing databases that differ because of how they handled data input inconsistencies.
****************
The truth is that probably there are not enough surface instrumental records from instruments that are well sited and well distributed around the globe. So there probably never will be a good global record of the past. Probably the satellite record is the best, but we only have 30 years worth.

Nick Stokes

Calvin – there is a “community project” developing a framework for GCM’s etc – CCSM.

• John Baltutis
  Posted Nov 23, 2009 at 5:55 PM | Permalink

  there is a “community project” developing a framework for GCM’s etc – CCSM.

  Hmmm! Started in 1983, redone in 1996, and the last version in June 2006, with not much else going on. Seems more like a propaganda exercise, mimicking the IPPC’s stance.

richard

Here’s an interesting May 2007 e-mail from Phil Jones that Ben Santer is replying to in 1178107838.txt. Jones says:

As a side issue , the disappearance of sea ice in the Arctic is going
to cause loads of problems monitoring temps there as when SST data
have come in from the areas that have been mostly sea ice, it is always
warm as the 61-90 means are close to -1.8C. Been talking to Nick
Rayner about this. It isn’t serious yet, but it’s getting to be a problem.
In the AR4 chapter, we had to exclude the SST from the Arctic plot
as the Arctic (north of 65N) from 1950 was above the 61-90 average
for most of the years that had enough data to estimate a value.

Now, to me, this sounds like he’s talking about a real issue in the data: i.e. the land temps on ice are colder than the sea temps when the ice melts and the 61-90 anomaly is based mostly on land temps, therefore anytime the ice melts, the temperature anomaly jumps up more than it should. Sounds like a legitimate data issue to be concerned about.

But then, he quickly moves on to how they had to exclude SST from the Arctic plot in AR4 because 1950 was too high, higher than the 61-90 average. Interesting…. Are they doing the same in plots of Arctic temperatures now? Or is it okay when it happens now (increases the trend) but needs to be suppressed in the past (decreases the trend)?

It’s all of these little gatekeeping choices that concern me the most.

wattsupwiththat

Steve, confirmed.

The source files with the comments that are the topic of this thread are in this folder of the zip file

/documents/osborn-tree6/mann/oldprog

in the files
maps12.pro
maps15.pro
maps24.pro

These first two files are dated 1/18/2000, and the map24 file on 11/10/1999 so it fits with Jones 1999 email where he mentions “Mike’s Nature trick” which was dated 11/16/1999.

• stevemcintyre
  Posted Nov 23, 2009 at 12:59 AM | Permalink

  Thanks, Anthony. I’ve looked through the predecessor emails to Mike’s trick – they refer to the IPCC spaghetti diagram which I’ve discussed on a few occasions at CA. I’ll have a detailed post on this.

aha

I like this one from README_GRIDDING.TXT..

“Use dist to specify the correlation decay distance for the climate
variable being interpolated – necessary information to determine where
to add dummy or synthetic data.”

Keen to find out whether dummy and synthetic data was used during the runs that produced the graph data. Moreover, what was the methodology used to produce the synthetic data. Oh, and whether dummy data was just used for testing.

Not sure if there’s anything untoward going on here, but you know what modelers tend to say… Garbage In Garbage Out (GIGO).

FINN

How should this be:

\f77\mnew\master.dat.com
29740 6030 2500 58 HELSINKI/SEUTULA—- FINLAND—— 1829 1999 297400 2974001
1829 -129 -150 -97 -18 76 139 173 139 112 34 -47 -77

This suspects me… was there in Seutula (Helsinki/Vantaa’s airport) a measuring point since 1829 – long before the opening of the airport in 1952? Or what?

I suspect the coordinate data is in the header 2. and 3. field, 60° 30’N ja 25° 00’E?

According to map, that is 5km southwest from the centre of Järvenpää.. in the middle of the forest. Pretty interesting? In the middle of the forest of Järvempää there is a sation called “Helsinki/Seutula” (which is the airport)????

Ace

I think this proves my point.

http://www.realclimate.org/index.php/archives/2009/11/the-cru-hack/

No doubt, instances of cherry-picked and poorly-worded “gotcha” phrases will be pulled out of context. One example is worth mentioning quickly. Phil Jones in discussing the presentation of temperature reconstructions stated that “I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) and from 1961 for Keith’s to hide the decline.” The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear. Scientists often use the term “trick” to refer to a “a good way to deal with a problem”, rather than something that is “secret”, and so there is nothing problematic in this at all. As for the ‘decline’, it is well known that Keith Briffa’s maximum latewood tree ring density proxy diverges from the temperature records after 1960 (this is more commonly known as the “divergence problem”–see e.g. the recent discussion in this paper) and has been discussed in the literature since Briffa et al in Nature in 1998 (Nature, 391, 678-682). Those authors have always recommend not using the post 1960 part of their reconstruction, and so while ‘hiding’ is probably a poor choice of words (since it is ‘hidden’ in plain sight), not using the data in the plot is completely appropriate, as is further research to understand why this happens.

• Sean
  Posted Nov 23, 2009 at 2:50 PM | Permalink

  They have not been clear abuot this, and even today they are not clear about it in the mainstream media.

  This is from Andy Revkin’s article in the New York Times:

  “Through the last century, tree rings and thermometers show a consistent rise in temperature until 1960, when some tree rings, for unknown reasons, no longer show that rise, while the thermometers continue to do so until the present.”

  So far so good, but here comes the obfuscation:

  “Dr. Mann explained that the reliability of the tree-ring data was called into question, so they were no longer used to track temperature fluctuations.”

  To any layman this would sound like the reliability of tree-rings was called into question, so tree-ring proxies were dropped from use due to their unreliability. Sounds very scientific to me.

  But that’s not what happened at all,. No, the tree ring data was truncated at 1940, so the inconvenient bits were snipped away and the convenient (but equally questionable) pre-1960 data was not dropped. Note that truncated is kind of like “no longer used” so Mann is choosing his words very carefully.

  And in case you think that I am just misunderstanding Mann even though his meaning is clear, here’s the next sentence of the article:

  “But he said dropping the use of the tree rings was never something that was hidden, and had been in the scientific literature for more than a decade.”

  So even Andy Revkin (who knows this area very well) interpreted Mann’s statement to mean that the unreliable data set was dropped, not selectively used.

  So, contrary to the claims of Real Climate, “hiding” is indeed an excellent choice of words to describe Mann’s ongoing obfuscation.

  By the way, I tried to post this same point to Andy Revkin’s blog on dotearth twice and it never made it through moderation, as near as I can tell.

  • Jon
    Posted Nov 23, 2009 at 7:24 PM | Permalink

    snip – don’t leverage this thread to introduce this topic

Gerry

What programming language is this?

TurkeyLurkey

Mr. Mc;
So very glad you had gotten through that esperesque treatment of the Yamal trees before this all hit the fan.
I’m thinking that will be worthy of revisitation after this noise dies down.
Perhaps the ninth word of the prior sentence is better as ‘if’ …
Highest regards,
TL

Syl

ACE

This part of Gavin’s response you quoted is the actual trick:

“The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear.”

The reconstruction flattened centuries past to emphasize the recent warming.

Up until this paper was published, it was thought there had been much more variation in past temperatures.

W W

For the problem of divergence ClimateAudit refers to this article:
http://www.nature.com/nature/journal/v391/n6668/full/391678a0.html

The abstract says: “The cause of this increasing insensitivity of wood density to temperature changes is not known, but if it is not taken into account in dendroclimatic reconstructions, past temperatures could be overestimated.”

Obviously the code did the exact opposite: past temperatures are underestimated and the proxy must be corrected to compensate.

Am I missing something?

• W W
  Posted Nov 25, 2009 at 9:10 PM | Permalink

  I meant “RealClimate refers to this article” (not ClimateAudit)

Asimov

Steve said: “Obviously he has a preconceived result in mind. I am a computational physicist, and this is how all modern science works.”

That isn’t the issue. Does modern science also go back and change old data PERMANENTLY and IRREVOCABLY because it doesn’t fit in with their preconceived result? Oh, and throw the real data away after the changes?

Sorry, but that’s NOT how modern science works. Having a preconceived result in mind is totally acceptable, changing the facts because they don’t fit in with your particular view is NOT acceptable.