I’m finally stating to come up for air after dealing with the fetid grubs and maggots of Hansen’s code. Needless to say, key steps are not mentioned in the underlying publications, Hansen et al 1999, 2001. I’m not going to discuss these issues today. Instead, I want to show 3 case studies where I’ve been successful in replicating the Hansen adjustment. In the more than 20 years since Hansen and Lebedeff 1987 and the nearly 10 years since Hansen et al 1999, to my knowledge, no third party has ever examined Hansen’s adjustments to see if they make any sense in individual cases. Some of the adjustments are breathtakingly bizarre. Hansen says that he doesn’t “joust with jesters”. I guess he wants center stage all to himself for his own juggling act.
I’m going to examine 3 stations today, which are the only 3 stations that I’ve examined so far. So these have not been chosen as particularly grotesque examples. The first two stations, Wellington NZ and Batticoala were chosen because they only had a few neighbors, a tactic that I used in order to try to figure out what the zips, bops and whirrs of GISTEMP actually did. The third station, Changsha, is the 1000th station in the GISTEMP station inventory.
After I do this, I’ll show how you can easily do similar plots for any station that you fancy, using materials and scripts that I’ve placed online. I have no idea how the emulation will hold up in other cases – I expect that there are more zips, bops and whirrs and adaptations will be needed – so let me know. Nonetheless, I’ve obviously broken the back of this particularly step. Going from here to gridded temperatures and hemispheric averages looks like a breeze in comparison, as Hansen uses a similar reference style in gridding and I can probably make simple adaptations of the hansenref program to do this.
We’ve talked a lot about this station as this has been my template for emulating Step 2. I’m going to show two plots for each station. The first plot shown below is a simple plot showing the difference between the actual Hansen dset 2 adjustment (monthly dset2 minus monthly dset1) and my emulation. Zeros all the way through here. So this one’s been nailed.
Next, as before, is a more detailed diagnostic plot, showing a variety or relevant adjustment information:
1) the actual rural reference minus the dset1 target (black solid) that gives rise to the observed adjustment (black dashed).
2) in green, the two-legged and one-legged fits over the M0 period (3-stations).
3) cyan – interpreted fit using values extracted from Hansen’s PAPARS.list file. All the values in Hansen’s log were divided by 10 to yield deg C. In this case, Hansen shows two positive slopes, but the results here require that the 2nd slope be negative. There are some remaining puzzles in the PAPARS.list file.
4) red – one of the two fits (two-legged or one-legged) is selected according to an interpretation of Hansen’s flagging method and rounded to 0.1 deg C, yielding steps. The graphic also shows a displacement of the selected fit zeroed to the end of the M0 period, again rounded to 0.1 deg C. This is the series that matches the observed Hansen adjustment in this and other cases.
In this case, Hansen has displaced the two-legged fit (said in the articles to have been used) with a one-legged fit, nowhere mentioned in the original articles.
Stepping back a little from the esotrica of decoding the Hansen codex, one know ponders whether the adjustment makes any sense and here I urge you to look particularly at the extrapolated periods. The adjustment, such as it is, is fitted to the M0 period (3 stations), but is extended by 50% to periods with less than 3 stations. The extrapolated adjustment does not consider the behavior of the reference series/target delta (the underlying black plot), but instead simply projects the closing values at either end of the M0 period. So in this case, there is a two fold whammy. The actual difference declines sharply in the 1980s, but the adjustment remains at high levels. But before 1951, the actual difference increases sharply, but the adjustment remains at low levels. Viewed as a statistical methodology, it’s hard to imagine a stranger procedure. I wonder what Wegman would say.
Batticaloa (Sri Lanka) is another station with only a few comparanda (only 3 rural stations). Again, the first graphic demonstrates the emulation of the Hansen adjustment has been successful with only a 0.1 deg C adjustment in one year unresolved.
Now here is the more detailed plot of adjustments, corresponding to the Wellington NZ plot. Again, the accuracy of the emulation is evident. Again, I urge you to look at the behavior in the extensions. You have to eyeball the fit a little since Hansen has displaced the fit (and I’ve only shown the extrapolation matching the displaced Hansen version, but the eyeballing is easy.) Once again, the extrapolation in the early portion yields a very poor fit. In this case, the actual delta declines, while the fit is left at high levels. In both cases, the fits don’t make any sense. Perhaps the nonsense cancels out overall, but maybe it doesn’t. But why have this sort of crap in software that supposedly meets NASA standards.
Wellington NZ had only 4 comparanda, Battocaloa only 3 comparanda, but Changsha had 20 “rural” comparanda and tests a few more components. Here is the comparison of the emulated and observed adjustments. Right on for recent periods, but 0.1 deg C off in the early portion, for reasons as yet unknown.
Next, here is the comparison graphic using the same routines as for the above two cases. While the emulated and final adjustments match, the plot from the PAPARS log parameters (cyan) doesn’t match.
This discrepancy can be patched by changing the sign of the slopes preserved in the PAPARS log. Why? Who knows? Another GISTEMP zip, bop and whirr. Aside from this, once again, we have a bewildering extrapolation. In this case, there is a disconnected piece of the Changsha record, which ends in 1938, re-starting only after World War 2. The two pieces of the Changsha record come from one version and therefore Hansen doesn’t bother with quality control details (he requires a 10 year separation and different versions before it catches his attention.) Although the pre-1938 piece seems disconnected, nonetheless Hansen extrapolates the 1946 adjustment for the extension of the record back to 1926.
There’s another interesting feature to this example. We’ve already seen reprehensible handling of Chinese data by Jones et al , an often cited study, where they claimed to have inspected station metadata, but there is convincing evidence that the station metadata did not exist. (Their defence against Doug Keenan’s complaint appears to be that the station metadata, having survived the Japanese invasion, World War 2, the Great Leap Forward, the Cultural Revolution etc, was “lost” by modern climate scientists in the IPCC period.)
Anyway one of Hansen’s “rural” comparanda for Changsha was Yueyang. According to Wikipedia, it has a population of 5.1 million.
Do It Yourself
If you want to inspect Hansen adjustments for a station of your choice, here is a script. Watch out for quotation marks if you paste from WordPress, which screws them up.
First load collated versions of Hansen data, laboriously prepared by yours truly. This gives you giss.dset1, giss.dset2, giss.anom (annual anomaly version), stations (giss information), station_versions ( a separate info file of versions in use) and two PAPARS logs (one has been re-zipped in gz from a Hansen format, both to read easier, and, as it turned out, using about 25% of the space of the Hansen zip) plus a file of functions.
#1. Benchmark Station Data
download.file( file.path(loc.giss,”giss.dset1.tab”),”temp.dat”,mode=”wb”); load(“temp.dat”)
download.file( file.path(loc.giss,”giss.dset2.tab”),”temp.dat”,mode=”wb”); load(“temp.dat”)
download.file( file.path(loc.giss,”giss.anom.tab”),”temp.dat”,mode=”wb”); load(“temp.dat”)
#. Station Infor
download.file( file.path(loc.giss,”station_versions.tab”),”temp.dat”,mode=”wb”); load(“temp.dat”)
pi180=pi/180;R= 6372.795 #earth’s radius in km
#3. PAPARS info
download.file(“http://data.climateaudit.org/data/giss/step2/PApars.statn.log.GHCN.CL.1000.gz”, “temp.gz”,mode = “wb”)
length(paparsghcn.log ) # 197043
The required data can be calculated for a given station by using the id number (look it up in the stations.tab directory) as follows:
That’s all you need to do. Do check the first plot, simply go:
To do the second plot, go:
To check out the selections in the rural network, go:
I might add a little more to this. There’s lots of underlying info in the W object. Get back to me if you run this and come up with anything odd. This is very far from being final or production quality, but at least it’s beginning to clear away the underbrush of the grotesque Hansen code.