Back in September when I was busy trying to figure out how Hansen combined station data, I was bothered by the fact that he used annual averages as the basis for combining scribal records (via the “bias method”) rather than monthly averages, which are readily available in the records that he uses. In my thinking, the use of monthly averages would provide twelve times the number of data points to use for combining records. I thought this particularly important when MCDW records were combined with older records, because the period of overlap tended to only be about four years. Forty-eight data points must be better than four, correct?
Even worse, we learned at the time that the first annual average in every scribal record is estimated. This is because the annual average is calculated from the seasonal averages, and the winter season (DJF) uses the December temperature from the previous year. Unfortunately, the previous year’s December temperature is not included in the first year of a scribal record, so it must be estimated. And because December must be estimated, the resulting DJF is an estimate, as is the resulting annual temperature. In the end, MCDW records tend to be combined with older records using three actual annual averages and one estimated average, instead of using forty-eight actual monthly averages.
As I worked through the puzzle there seemed to be a lot of estimating going on, more than just the beginning of a scribal record. There are a lot of “999.9″ monthly values in Hansen’s data (this equates to the “-9999″ entries in the raw GHCN data), but he still manages to calculate a lot of annual averages. As we later learned, Hansen’s estimation algorithm enables him to estimate an annual average when up to six monthly averages are missing. Following are three examples of his estimation algorithm at work. I had downloaded the data for the stations below on August 31, and at that time each station already had an estimate for 2007. Compare the estimate with the actual value calculated at the end of 2007:
Bagdarin: estimated -4.88 (May, Aug-Nov missing), recent estimate -4.39 (May data still missing)
Erbogacin: estimated -4.05 (Feb, Aug-Nov missing), actual -4.71 (all months available)
Budapest-Lori: estimated 13.57 (Aug-Nov missing), actual 12.66 (all months available)
Recently, I began wondering just how much estimation is going on. On February 7 I downloaded the raw GHCN data (v2.mean.Z) from the NOAA FTP site to see if I could get a handle on how much estimation Hansen does by examining the frequency of missing monthly data. Hansen does not use every single record from this dataset, but he does use almost all of them. Thus, an analysis of the GHCN data should provide a close approximation of how much estimation Hansen does. Yes, I am estimating the amount of estimation. It was either that or scrape the data off GISS, and frankly I don’t have the patience for that.
I think you will find the results of this analysis interesting.
In the first figure we can see the completeness of station data worldwide on an annual basis. The green section represents the percentage of station records with valid data for all twelve months as needed to calculate an annual average. The yellow section represents the percentage of station records with fewer than twelve months of valid data, but enough data for Hansen’s algorithm to calculate an estimated average. The red section represents the percentage of station records missing enough data to preclude even an estimation of the annual average temperature.
The thing that struck me was just how much more estimating had taken place in recent years versus earlier years. The next figure provides a close-up of the past 30 years, from 1978 to 2007:
[Edit] For fun, I decided to compare the GISS ranking of annual global temperature anomaly with my ranking of the annual temperature estimation that is done. I’ve sorted by the 25 warmest years. I find it interesting that all but three of those years (highlighted in red) rank in the top 25 of the amount of estimating that is done.
As the table shows, while GISS says 2007 was the hottest year on record, it was also the second highest year with estimated and/or unavailable temperature data.
To compound the problem, the last thirty years have seen a significant station record die-off. Most are familiar with the graphic on the GISS website showing the number of stations used in Hansen’s analysis:
I always found it interesting that this graphic ends with the year 2000, and seems to have a rather precipitous drop in stations during that year. Thus, I decided to count the number of GHCN records on an annual basis, and the results tracked rather well with the GISS graphic. Note that my count is of records, whereas Hansen counts stations. Prior to 1992 multiple records might consolidate to a single station, which explains why my absolute numbers are higher than Hansen’s. The first chart shows the number of records on an annual basis since 1880:
The following image zooms in on the last 30 years (1978 to 2007):
The above graphic shows that, while GISS says 2007 was the hottest year on record and GHCN indicates it had the second highest level of temperature estimation, GHCN also indicates that the number of data points for 2007 were the fewest since before 1900.
To summarize what I am seeing from the GHCN data: (1) the number of stations / records has been dropping dramatically in recent years and (2) with that drop the quality of the record-keeping has also dropped dramatically because we are seeing a corresponding rise in estimated annual temperatures and/or insufficient data to calculate an annual temperature. Using this data, GISS is showing that the temperature anomaly in recent years is the highest recorded in the historical record.