Back in September when I was busy trying to figure out how Hansen combined station data, I was bothered by the fact that he used annual averages as the basis for combining scribal records (via the “bias method”) rather than monthly averages, which are readily available in the records that he uses. In my thinking, the use of monthly averages would provide twelve times the number of data points to use for combining records. I thought this particularly important when MCDW records were combined with older records, because the period of overlap tended to only be about four years. Forty-eight data points must be better than four, correct?
Even worse, we learned at the time that the first annual average in every scribal record is estimated. This is because the annual average is calculated from the seasonal averages, and the winter season (DJF) uses the December temperature from the previous year. Unfortunately, the previous year’s December temperature is not included in the first year of a scribal record, so it must be estimated. And because December must be estimated, the resulting DJF is an estimate, as is the resulting annual temperature. In the end, MCDW records tend to be combined with older records using three actual annual averages and one estimated average, instead of using forty-eight actual monthly averages.
As I worked through the puzzle there seemed to be a lot of estimating going on, more than just the beginning of a scribal record. There are a lot of “999.9” monthly values in Hansen’s data (this equates to the “-9999” entries in the raw GHCN data), but he still manages to calculate a lot of annual averages. As we later learned, Hansen’s estimation algorithm enables him to estimate an annual average when up to six monthly averages are missing. Following are three examples of his estimation algorithm at work. I had downloaded the data for the stations below on August 31, and at that time each station already had an estimate for 2007. Compare the estimate with the actual value calculated at the end of 2007:
Bagdarin: estimated -4.88 (May, Aug-Nov missing), recent estimate -4.39 (May data still missing)
Erbogacin: estimated -4.05 (Feb, Aug-Nov missing), actual -4.71 (all months available)
Budapest-Lori: estimated 13.57 (Aug-Nov missing), actual 12.66 (all months available)
Recently, I began wondering just how much estimation is going on. On February 7 I downloaded the raw GHCN data (v2.mean.Z) from the NOAA FTP site to see if I could get a handle on how much estimation Hansen does by examining the frequency of missing monthly data. Hansen does not use every single record from this dataset, but he does use almost all of them. Thus, an analysis of the GHCN data should provide a close approximation of how much estimation Hansen does. Yes, I am estimating the amount of estimation. It was either that or scrape the data off GISS, and frankly I don’t have the patience for that.
I think you will find the results of this analysis interesting.
In the first figure we can see the completeness of station data worldwide on an annual basis. The green section represents the percentage of station records with valid data for all twelve months as needed to calculate an annual average. The yellow section represents the percentage of station records with fewer than twelve months of valid data, but enough data for Hansen’s algorithm to calculate an estimated average. The red section represents the percentage of station records missing enough data to preclude even an estimation of the annual average temperature.
The thing that struck me was just how much more estimating had taken place in recent years versus earlier years. The next figure provides a close-up of the past 30 years, from 1978 to 2007:
[Edit] For fun, I decided to compare the GISS ranking of annual global temperature anomaly with my ranking of the annual temperature estimation that is done. I’ve sorted by the 25 warmest years. I find it interesting that all but three of those years (highlighted in red) rank in the top 25 of the amount of estimating that is done.
As the table shows, while GISS says 2007 was the hottest year on record, it was also the second highest year with estimated and/or unavailable temperature data.
To compound the problem, the last thirty years have seen a significant station record die-off. Most are familiar with the graphic on the GISS website showing the number of stations used in Hansen’s analysis:
I always found it interesting that this graphic ends with the year 2000, and seems to have a rather precipitous drop in stations during that year. Thus, I decided to count the number of GHCN records on an annual basis, and the results tracked rather well with the GISS graphic. Note that my count is of records, whereas Hansen counts stations. Prior to 1992 multiple records might consolidate to a single station, which explains why my absolute numbers are higher than Hansen’s. The first chart shows the number of records on an annual basis since 1880:
The following image zooms in on the last 30 years (1978 to 2007):
The above graphic shows that, while GISS says 2007 was the hottest year on record and GHCN indicates it had the second highest level of temperature estimation, GHCN also indicates that the number of data points for 2007 were the fewest since before 1900.
To summarize what I am seeing from the GHCN data: (1) the number of stations / records has been dropping dramatically in recent years and (2) with that drop the quality of the record-keeping has also dropped dramatically because we are seeing a corresponding rise in estimated annual temperatures and/or insufficient data to calculate an annual temperature. Using this data, GISS is showing that the temperature anomaly in recent years is the highest recorded in the historical record.
Climate Audit 
95 Comments
Very interesting… It would be intriguing to see your table expanded for all years and than used as a basis to see if the ranks correlate. I also do not get how coverage can go up while # of stations goes down…
Yes-very interesting. It might be clearer to change the column header in the table to “Rank-% Est” and add another column with the % Actual Data?
John Goetz,
Here is a link to a graphic you might be interested in.
http://www.climateaudit.org/?p=1956#comment-195582
I posted up a link to a spreadsheet of the years of coverage for the GISS stations a couple weeks ago. Not sure what thread. So here it is again.
I scraped the information in January from station data page 2. The one with the link to the graphic for the station. It’s for dataset 1. The one leading you to the “after combining sources at the same location” graphic.
I just downloaded GHCN v2.mean yesterday and they now have data for 900+ stations for January this year. Data is still trickling in I think. I expect GISS to update their January data in the next few days. They haven’t done it yet.
One more thing. Most of the stations where the data ended in 2006 actually don’t have any data past March 2006. It’s not random during the year. Almost all ended with March.
I don’t understand this at all. We live in the modern age. Today we could easily place climate sensors around the country and around the world and run them with solar panels/batteries and a satellite internet connection and tie the world together with accurate data quite nicely. In the period of 1993-2001 when the Clinton administration was in power the Earth sciences community had gobs of money to spend on things like this. Steve do you understand what is going on here?
Excellent work John. The effect of the cold war coming to an end is clear in the drop starting around 1989 from 10,000 to 4000 stations by 1993. Russia closed a lot of stations, and for some reason, so did Canada during that time.
If you can, what would really be useful would be a scatter plot showing station distribution with latitude/longitude. Getting a handle on how well sampled the surface is is equally important. There’s always the criticism tossed at the surfacestations.org project saying that the USA has just 6% of the worlds landmass. With 1200 or so USHCN stations against what looks like about 2000 GHCN stations remaining, it sure seems like the USA has the lions share.
I’d love to see the distribution of remaining stations plotted this way if you feel so inclined. It should be an easy task.
RE#5 It makes perfect sense to me, its all about the funding trend. NOAA”s COOP network is going through a similar thinning…about 1990, NOAA started closing NWS offices Nationwide, with lots of job elimination and consolidations. COOP managers (responsible for Stevenson Screens, MMTS, site management, and observer training) are down to about 1/3 of what they used to be, yet the geographic area of the USHCN is unchanged. No wonder sloppy siting issues are creeping in.
The NOAA funding shift had to do with moving from a labor intensive organization with traditional methods to one that relied most heavily on new technology and automation. ASOS, AWOS, NEXRAD, satellite systems, etc all got much more funding at the expensive of offices and personnel. It was all part of NOAA’s grand modernization program, Great for forecasting, bad for climate monitoring.
No wonder Hansen and Karl declared in a meeting around 1997/98 that the surface network was falling apart. To Karl’s credit, he did something about it and got the Climate Reference Network (CRN) up and running. Hansen just keeps working with less and less data but doesn’t appear to have done anything to help stem the loss.
For a historical view I think it would be vital to know the profile of the stations that have dropping out.
– Were more “rural” stations dropping out than “urban” stations?
– Did they have a higher or lower average station quality rating? (I know this is unknowable – these stations were never surveyed!)
– Did their dropping out cause a bias in the remaining average?
How could one possibly have an accurate historical comparison – with a massively shrinking pool of stations, and 70% of the remaining stations rated CRN=4,5 (error >= 2 to >=5 degrees C). I agree – it is odd that this big die-off occurred during such a critical period. Don’t they care – we’re all gonna die and the polar bears are gonna melt or something… Heh
I’ve been watching this blog for a few months now and very much appreciate all the work that goes into it.
Heh, Anthony beat me to it. 🙂 Thanks for your hard work Anthony.
I’m confused as to why the number of stations is dropping. Less stations means more variance and a greater likelihood of an exceptional value.
Anthony Watts says:
The Canadian government deficit was out of control in the early 90s which forced the government to slash spending on just about everything. Programs that voters did not care about would have been the first to go.
I suspect that proximity to population centers would have been a primary concern because I believe these stations exist primarily for weather forcasting – not climate analysis. This would lead to a larger percentage of rural stations being closed which would introduce a significant urban bias into temps in northern canada since 1990.
One also should consider that a lot development has been going on in northern canada with diamond mines, oil sands and similar ventures. This will have led to significant land use changes in the last 20 years – most of which would likely bias towards heating this up. I suspect a similar trend has occurred in Russia.
I don’t think it is coincidence that the red spots on northern canada on this map:
http://wattsupwiththat.wordpress.com/2007/06/22/a-note-from-a-nasa-climate-researcher/
also happen to be the areas with the most development going on.
They stopped getting data from 1062 surface stations world-wide in 2006. Of these 707 were classified by GISS as rural. 700 of those were in the US. Another 106 stations were 15,000 or less population. 104 of those US. Both GISS and GHCN had about the same drop at the same time. For 2007 GISS temperatures were based on 1051 stations.
I’m sure if you asked, they’d say the result is an improvement or no change in data quality. I have my doubts about either possibility.
And what difference does it make to the historical record if only actual data is used?
#12 are you asking what difference data quality makes? Do we really need to respond to your question?
Only a guess, but has there been a removal of suspect UHI and other poor quality stations?
I don’t have the data nor the means, but would it be interesting to take the remaining year 2007 stations and, using only their records, do a new surface temperature plot back in time? I have a suspicion based on belief science and not on evidence that the larger part of the hump of the decade 2000 would reduce, at least for the USA-48.
A bit off thread, but on the theme of estimation, the WG4 IPCC report has a chapter on sea level rise from ocean heating, mainly to 700m depth. I have forgetten the physics, but does the land expand a little as it gets hotter too? Relative magnitudes unimportant? Circulating groundwaters don’t have the heat capacity to warm the land suface much? Too small a coefficient of thermal expansion in solid land? I don’t know. I’m forever worried about a stable datum for estimation of sea level change. Looks like the centre of the earth is popular now that satellites are also popular.
It’s probably been discussed before, but the situation with Australian stations is interesting. Over 380 Australian stations were dropped out of the GISS network in 1992 alone. Over 340 (90%) of these were listed at the time as “rural”.
I believe that the majority of these stations are still operational. At least 44 of the dropped stations are currently included in the Australian Bureau of Meteorology’s “High Quality” network that has data up to 2007. However, while the Bureau publishes combined historical trends for this network, the historical data for individual stations is no longer freely available, as far as I know; you can buy a CD of the data, though.
I found just 39 Australian mainland stations in GISS with data up to 2007. Only 13 (33%) were listed as “rural”, and of these 13, at least eight were actually airports at country towns of populations up to 15,000 (such as Broome popn 13,000 + many tourists).
Out of almost 600 Australian stations in GISS, I have found only one mainland station that is completely rural (as far as I know) and has continuous data from 1930 to the present (Cape Leeuwin).
somewhere on ushcn#3 ClaytonB did a chart showing the warming trend ( using opentemp)
for CRN1,2,3,4,5 Using 3 different data sets. Raw, TOBs, and FILNET (after fillin in missing data)
I think I recall that the file that shows the biggest trends is filnet.
in anycase it might be interesting to take stations with complete records from GISS,
then randomly remove months, then apply hansens method and see how well it does.
Then try other methods
#3 Bob
Yes, I saw the same thing in the US – a dramatic drop in valid records from 1132 in March 2006 to 134 in April 2006. In the link you pointed me to, Steve comments that it is probably a reporting lag and at some point in time a large update will be made to the record. Given that we are now approaching two years in this lag – and that 2007 and 2006 are being hailed / reviled as two of the warmest on record – it seems important that the record be kept up to date. At least before such pronouncements are made.
Here is a graphic I created of the number of valid monthly temperatures (non -9999 values) reported from the US into GHCN since 1990. Ignoring the April 2006 die-off, the trend clearly remains downward:
I think all this averaging is masking what is really going on.
Here is GISS’s monthly average global anomalies from 1997 to 2007.
As you can see, a large drop in temps during 2007 and lots of variability over the 1997 to 2007 period leading one to conclude there is not much trend in the data over this 10 year swath.
However, when you go to the annual averages (from 1880 to 2007 this time) and chart a five-year running mean of the 12 month annual averages (60 months in each data point in the line) …
It looks like the 1997 to 2007 temps are skyrocketing (when they are clearly not if you look at the monthly data.)
Two points come to mind in reading this thread. First, if Russian and Canadian stations closed en masse, most of these stations would have been producing colder than average numbers, so wouldn’t their loss automatically produce a new average with a higher answer?
Second point is similar. It seems to be generally agreed that the minimum temperatures recorded seem to be the ones going up due to gorebull warming, whereas the maximums are remaining the same. Surely, this means that there is no change in, what one might call, maximum discomfort temperature, while there is a reported increase in “global average temperature” which, as reported by the MSM make the general public think that all temperatures have gone up. In fact all that has happened is that we will need fewer blankets at night.
Thoughts anyone?
re” #18 John Lang,
Looking closely at your figure C what strikes me is that there seems to be a peak in the anomalies DJF quarter practically every year. I can think of two reasons for this:
1. Something to do with the NH, SH differences.
2. The fact (as mentioned in #19) that it’s rising lows rather than rising highs that we’re seeing.
Can the graph be split into HN and SH versions to see a little better what’s happening?
Tony,
GW theory says that the effect of CO2 warming will be strongest in the higher latitudes in the winter and at night (e.g. -20 vs -30). This is because cold dry air has less water vapor to mask the contribution of co2.
So yes, it is a fair question ask what the problem is with a little co2 induced warming since it will be largely beneficial. A fact largely ignored by the MSM.
Anomaly map after station closures
The Former Soviet Union and Canada play pretty significant roles in “global” warming from the looks of this plot.
It may just be a coincidence, but it is certainly worth investigating. Nice work John.
Mr McIntyre,
Concerning “Off to Georgia” you wrote:
“I think that I’m detecting a bit of a change in attitude among some climate scientists, especially younger ones.”
Still feel that way? Were the younger scientists at GA Tech scientifically open, or were they
ideologues? I’m just looking and hoping for confirmation of shifting attitudes among the younger scientists.
I’d like to see you come to Germany. But I must warn you, it’s a very close-minded and hostile AGW climate here.
Dave Dardinger #21 – I haven’t seen GISS’ monthly data split into southern and northern hemispheres.
The annual data is split in this chart 1880-2007. Quite a difference between north and south.
Monthly anomalies (north and south but no charts, data only) from the NCDC is linked here.
http://www.ncdc.noaa.gov/oa/climate/research/anomalies/anomalies.html#anomalies
I’d be willing to bet that there is now a much higher percentage of stations at airports, since airports HAVE to have this data. We probably have a significant “tarmack factor” in these data.
re 10
One should consider the enormous area contained in northern Canada when compared to the development of diamond mines etc. Unless there is some specific data, my assumption would be that the developed area would be negligible when compared to the vast regions that will remain undeveloped for the foreseeable future. Flying from Winnipeg to Toronto, one is immediately struck by the vast size of Northern Ontario. Flying for hours over forest and lake with not a light visible on the ground. This doesn’t support a contention that there have been significant land use changes in the north. – back to deep lurking
Might want to look at this for comparison with GISS.
http://wmbriggs.com/blog/2008/02/09/how-to-look-at-the-rss-satellite-derived-temperature-data/#comments.
Am I really supposed to believe that we have surface temperature measurements from 80 to occasionally above 90% of the Southern Hemisphere?
There can be hardly any coverage below 50 S.
Last week on the Samizdata blog I posted the following comments. I think they might be on topic for this thread. (And recycling is good. Right?)
And in a later comment.
Population within 100NM of a station back in 1975, versus today.
Any GIS Gurus around
27. briggs needs to be on the blogroll here
Re: 22
It depends on what you are charting. Typically, it is not absolute temperatures, but rather it is the difference from average. If you were charting absolute temperatures, then you are correct that removing stations from ‘cold’ places would cause values to rise. But, if all you were doing was charting ‘delta from mean’, then it wouldn’t matter where you removed the stations from.
Caveat, I am not making any statements about what this would do to uncertainties/coverage/etc. Only addressing the original point about removing stations from places that were ‘cold’.
Tom Gray says:
“One should consider the enormous area contained in northern Canada when compared to the development of diamond mines etc. Unless there is some specific data, my assumption would be that the developed area would be negligible when compared to the vast regions that will remain undeveloped for the foreseeable future.”
One should also assume that any temperature data has been collected from a very small number of thermometers in the developed areas and then used to estimate the average temperatures across large swathes of wilderness. The lack of coverage in northern Russia and Canada could result in a huge UHI effect even if the population centers are small.
How is the estimation done?
#34 John
See http://www.climateaudit.org/?p=2019, starting with comment #122.
#35, That’s interesting. Now I wonder how one decides how many of these estimated points to use?
John Goetz: this info and the stuff on how GISS does estimation for missing recs is very important and needs to be made public. If there is any way to publish this (besides at CA), it would be great.
#37 The is a climate audit wiki but I presume you want it in something like a textbook or journal.
I was always taught that averages are tricky things – easy to compute, easy to understand but don’t tell you much about the data (variability, distribution, etc.). I was also taught that an average of an average is a pretty meaningless number. Maybe I had bad teachers but those two things have stuck with me for a long time.
It would be interesting to conduct an evaluation of the trends for those weather stations in use throughout the entire data record – or some meaningful portion of it. This might give an indication of the effect that the dropped data has on the record.
Has anyone compared the trend lines of the closed stations to those of the remaining ones, in the years when they overlapped? If these stations are being closed randomly, those trend lines should be almost identical. But the closing of so many stations opens the door to the possibility of creating a trend simply by removing the contradictory data. Given some of the other creative statistics used in this debate, (the debate that’s already over) I wouldn’t be at all surprised to find that the closed stations showed a significantly different trend.
Even without malicious intentions, basic economics could cause such a difference. It seems clear that the most appropriately placed and maintained stations would also be the most expensive ones, and therefore the most likely to be closed.
This dovetails in perfectly with my contention that the anomaly is probably nothing more than a consequence of modern sampling and combination methods. Sampling is what it is, grabbing air temp 5 feet over the ground in scattered locations that were picked for some other reason (rather than the location’s appropriateness to monitor climate change), mostly with mercury thermometers as the mean of daily min/max, as indicative of the land temps, and sampling the surface of the water from space now but with ships and buoys mostly, as indicative of sea temps, and averaging them all together to smooth out a highly variable signal of a large range measured in whole degrees to some number that many call the global temperature, but is at best a generalized proxy that may have no real bearing on anything..
#17 John,
2005 is the year with a drop in valid monthly data and coincidentally the warmest year “ever” recorded by the GISS 😉
Anyway, considering the huge non covered surface (in grey on the GISS map), it’s increasingly nonsensical for the GISS to announce a “global” temperature.
Re NH-SH question, I not sure if it’s widely known but what really puts me off in the global anomaly is it’s exactly the arithmetic mean of the NH and SH anomalies. And it’s true not only for GISS but also for RSS, UAH, CRU, NOAA ! (see graphs here).
I’ve raised the question of the meaning of lumping together 2 diverging temperatures (NH is heating, SH is cooling) several times to the AGW people but as usual, I’m served with their usual sermoning crap and no scientific explanation. Maybe somebody here has an idea ?
See how curious this is?
2007 Tied as Earth’s Second-Warmest Year
NASA scientists have determined that 2007 tied with 1998 for Earth’s second warmest year in a century. (Jan ’08)
+ Read News Release and Data Update
That’s on the main page.
1. Go here: http://data.giss.nasa.gov/gistemp/
2. Get this: Global-mean monthly, annual and seasonal land-ocean temperature index, 1880-present, updated through most recent month http://data.giss.nasa.gov/gistemp/tabledata/GLB.Ts+dSST.txt
3. Tell me what the year with the highest anomaly is.
While we’re at it, what’s the climate science degree? http://www.giss.nasa.gov/staff/jhansen.html
The centers of excellence are Iowa and East Anglia.
Not Cambridge Mass. or Cambridge UK.
Makes you think.
John Goetz
Thank you for your contribution. The usual maxim is that more data are better than fewer.
Is the following a reasonable suggeston? My lack of resources prevents an answer, so please pardon the question.
The suggestion is a re-plot the 2,000 or so GHCN station records on your graph at year 2007, back until each started. Annual basis would be good enough, then a summary graph. Will the 0.8 degrees C per century (or so) that is often quoted, remain in this set of selected data?
(This is not an endorsement that the ‘adjustments’ made to the data so far are appropriate or reflect the weather conditions at the times and places.)
#44 Sam:
I don’t understand your point. The dataset in your link shows 2005 as having the highest anomaly, followed by 1998 and 2007, as they report. Not that I put any special credence in their numbers.
In January I downloaded the following dataset as having been combined from sources at the same location. They now have the January GISS temperatures online. So I downloaded them again for those stations the inventory says had data in 2007.
217444540001 KATHMANDU AIR 27.7N 85.4E 354000 1951 2007
This station is no longer a combined record. The last inventory date for this dataset has changed to 1991. The three datasets available are now separated.
A close look at my previous download shows the combined record at that time was for dset1 and dset2. Dset0 was not part of it even though it had 20 years of temperature data. Seems like they’re making changes to how they combine data or they’re having coding problems.
Here’s a link to a 3-page spreadsheet of the current dset1 data compared with the previous copy I made last month.
Re: #14
Heat transfers through the ground only by diffusion. That’s really slow compared to convection in water. Also, a significant fraction of incident solar radiation penetrates on the order of 100m. So the effective heat capacity of the surface of the ocean is orders of magnitude higher than the land. What that gets you is diurnal temperature variation of 10’s of degrees for dry land surface temperature while ocean near surface temperature varies by fractions of a degree over the course of a day. The thermal coefficient of expansion for rock is also orders of magnitude smaller than for water at temperatures above 4 C. So I think that means thermal expansion of rock is insignificant compared to other geologic processes (which also make it difficult to define sea level).
Thank you DeWitt Payne #49
You confirmed my suspicions on expansion but the hoary old topics of isostasy etc still leave me wondering. I also wonder how much circular logic (appropriate for an orbit) there is in satellite measurement of sea level changes. Atomic clocks are accurate, but the speed of light in other than a vacuum must call for some modelling presumptions. You also raise inferences about the choice of lighthouses for land surface temperature records, when the sea would have an influence greater than inland stations get.
#48 Bob
I am trying to understand your comparison, and the Jan 08 tab has me confused. Does the data in that tab represent the combination of the KATHMANDU records available in January (dset1 and dset2) or does it represent dset1 only?
#51 John,
I probably confused you with my use of dset1, dset2 etc. in my last post. I should have said source files 0, 1, and 2. Dset0 is actually the pre-combined, dset1 is the combined sources, and dset2 is the homogeneity adjusted. I wasn’t precise about what I was describing.
The Jan08 tab has the downloaded dset1 data that I originally got back in January. It came as one file combining two of the three source files into dset1. The Feb08 tab is how I got it this last time. I never looked at the station prior to last night and was surprised to see they didn’t include one of the source files in the combined one I downloaded in January. It has 20 years of data and should have been included.
As of when I downloaded it yesterday, you can no longer get Kathmandu as a combined record. It is only available in the three individual files even though they are the same location. All the source data included in the combined dset1, whatever the number, should be listed in the upper left corner of the graph.
When I auto-download I do it by sending them the full 12 digit ID by combining their station_list ID with the country-code. Padding the 4th position with a zero if it isn’t 12 digits long. Never use the input box with the name unless I want only one or two stations. I can’t retrieve some 240 sites by name since they contain a foreign language apostrophe.(Diacritic maybe?) The full station ID always works and I get to skip the input box page. I can get quite a few datasets an hour(at night anyway) with the program I wrote. It uses my browser as it’s slave to make the requests. 🙂
I do a couple data checks before I save the downloaded data. That’s the only reason I noticed the station at all. I expected data up to at least 2007, but it only returned data up to 1991. So the station was saved off as a download error.
I compared the 833 stations I got last night to the same station data I got back in January. Evidently some stations are still to report for January as there were 1051 stations at the end of 2007 and those were the ones I was trying to get.
802 of the stations had some sort of change. I found 306 stations had data changes in 2007 most having an effect on the annual temperature. 157 stations adjusted the annual temperature down for a total of 31.14c. Maximum change of 0.95c. Going the other way 95 stations adjusted up for a total of 10.43c. Maximum change of 0.48c. The net change was down 20.71c on the 252 stations. About a 0.08 reduction per change. Of the remaining 54 stations, 26 still had no annual, and 28 had no change. Both groups had changes elsewhere.
There were changes to 4262 pre-2007 station records going back as far as 1881. Much of it trivial, but cumulative. Looking only at annual, there were 2,008 temperatures adjusted down for a total of 36.47c. Maximum change of 0.11c. 2,025 records were adjusted up for a total of 42.52c. Maximum change 0.1c.
I think they should consider changing the way they estimate missing data. Using the surrounding data seems like it would work much better. Tough to maintain a close correlation when you do it using data that becomes less correlated as time passes.
#51, 52. I checked some of the stations that we considered in analysing the dset=1 step – Gassim, Bagdarin. Their dset=1 in the present version is a combined record. As I recall, there is a minimum overlap benchmark in order to combine records and Kathmandu probably falls short of that. As to why there should be a difference between Jan 2008 and Feb 2008 versions, a mystery indeed. I’ve got a version from last summer’s scraping and will check against that some time.
It is also inconsistent. A change was made that affected Kathmandu, but not Bratsk (as far as I can tell anyway).
Much machination behind the green curtain.
==========================
#54 Steve
It looks like the only true overlap for Kathmandu is 1979, between the first two records.
#53 Bob
Hansen’s method of estimating an annual average has the potential of changing previously-estimated annual averages as new data is added to the record. I created a graphic example of that for Jakutsk record #7 last night and was going to post it in a day or two after I re-examined the equations I derived last fall. The long and short is the historical record is not static. New data affects old estimates.
Steve,
I think the problem may have to do with the idea that source0 has twenty successive years of annual data but isn’t the longest record by number of years. Source1 has the most years, but has only 17 broken years of annual. Source2 is the current record, but has only 13 years of annual.
Maybe they just decided they can’t properly combine them.
#58 Bob,
I looked through the Kathmandu data and saw that 1961 – 1978 have no recorded temperatures (the 999.9 I mentioned in the post). So the date range reported on the GISS website is misleading, and a 20-year overlap does not actually exist. This is why, when I created my graphics of station coverage around the world, I did not include a station in a given year if no annual average could be estimated.
John,
#57
I’ve noticed that. As long as there is a trend over the length of the record, their estimate moves in the same direction as the trend and propagates back in time. Doesn’t make much sense.
I would estimate a missing month from the surrounding months. Get a series mean for each month up to the time of missing data. Never past it. If you need to estimate a quarter with a missing month, ignore the quarterly values and get the mean of each of the three months centered on the missing one and total them. Each monthly mean now has a fixed climatic portion of the three month longterm. Maybe the monthly portions work out to be 30%, 37%, 33%. It would correlate the current missing month with it’s average share of the three month longterm figure. In this scenario you’d get 63% of the 3-month mean from the current surrounding monthly temperatures, estimating the balance from the missing month’s 37% by comparing with what the surrounding months 63% amounted to for the current year.
You’d never find the constant adjustments back in time, because once you make the estimate it won’t be changed by future temperature trends. It’s also a closer correlation in time. Can’t be worse than what they do now.
I think they do it the way they do so they can fill in more quarters and therefore more years. The way I described wouldn’t allow them to do an estimation if two consecutive months were missing.
John,
#59
I posted up a link to an inventory of their stations, but figured I’d use as gospel what they claim to have in their inventory. I quickly realized it was well-padded and probably should have mentioned it. I’ve seen several stations with 20-30 consecutive years of 999.9. One station I saw, GISS claimed data since 1881 and not a legitimate value in sight for the first 50 years. Why do they do that kind of thing? Probably trying to impress funding agencies with how much data they’ve accumulated, I guess.
GISS monthly data for jan 08 is in: +.12C anomaly.
Estimate the yearly anomaly for 2008.
HADLEY has estimated 2008 anomaly at .37C ( adjusted to GISS.. means about .47C)
Can you beat a GCM at climate prediction? Deep Green versus Climate Audit.
#60 Bob,
It doesn’t make sense to me either, which is why I am fascinated with it.
If I were to estimate a temperature I would largely do it the way you describe. However, if I were to estimate the local January 2008 from December 2007 and what we are seeing thus far in February 2008, I would never get the warmer-than-average January we had…it would be colder than average instead! So no method is perfect.
As for your question “Why do they do that kind of thing?” in #61, I think it is nothing more than dumb software, rather than ulterior motives. There is a lot of data and I don’t think many people took a critical look at it – line by line, entry by entry – until folks on this site did.
Ahhh. Great! Comments are working again. They were down for a short period.
Looked up that station I mentioned. It’s Skikda ID: 101603550002.
Not only is there no data the first 50 years, but from 1942-1965 there is another similar stretch.
Looking at the graphic for the station makes me think they keep it around as a talking point. The station has a 128 year record and never in the history of the station have temperatures risen as dramatically as they have in the past 25 years. 🙂
The same problem exists in the AHCCD data in Canada, many of the stations are not reporting anymore and they cover large regions in Canada. The other problem with the Canadian data is that many of the systems were installed at the start of the cold war so the data is very truncated. in Saskatchewan the Cree Lake historical data is from 1969 and ends in 1994. Uranium City data is from 1953 and ends in 2006, but the data from 2001 to 2003 and 2005 is not available. These stations cover almost one third of Saskatchewan.
The many instances and evidences presented here on problems of collecting the surface land temperatures makes it apparent that “adjustment and correction” procedures used by those responsible for these data sets are asked to accomplish what might appear to be an impossible task. Unless we are willing to go through the agaonizing analyses of those corrections procedures to either verify or discredit them we can only conjecture on the end results and/or accept the assurance of the responsible parties.
That is why I agree with Steve M that we should concentrate on the more straight forward (and yet not uncomplicated) historical temperature measurements of the oceans or SST. I would complement that with analyses of the accuracy and precision of the satellite temperature measurements as in the end those measurements would appear to be free of many of the known problems and potential biases of the land surface records.
I’ve got a question hopefully someone is interested in answering. How well does an average temperature represent the actual temperature data measured? Any numbers that can be quickly calculated to measure how well the average fits the 60-ish data points for any given station?
Oops, forgot to limit to a single monthly average!
#67
Earle these questions pop up periodically, here is a link to a prior thread (that then links to other threads) that discusses some of the issues that you’re getting at.
Curt: The point is you get different answers from supposedly the same data.
mosh: Given that the GLB.Ts+dSST file has Jan last year at .87 and Feb at .63, if this month came in at .12 or about 14% of last year’s. I’ll go ahead and estimate Feb at about 20% of last year, or .12 just to see. BTW, .12 is the lowest month since it was .08 in May 1995 and you have to go to Jan 1989 to get a Jan under .12 (.03 – but the curious thing is that May 1989 was only .04; what’s going on with Jan and May? And do they like April and June?)
Anyway, if all the months end up around 20% then the year next year would be something like
.12 .12 .12 .12 .11 .11 .11 .11 .10 .11 .10 .08
Or a yearly anomaly of about .11 It just depends on what the anomaly does; keep going down, go up or what? Do the rest of the months look like most of last year around .5? For those of you keeping count, Jan is the 3rd straight month where the anomaly is moving towards 0 rather than away from it. (Feb 1994 is the last time there was a month with a negative anomaly, but that year Jan was twice as high as this year’s)
In addtion, with some variation, the anomaly’s been basically flat since 2001, with only 2005 over .6 (Or in other words, it was .13 higher than 2004, where as 2006 was only .05 above 2004)
Then my next comment would be; with 2005 is the highest, but it’s only .05 above the 98/07 tie of .57 , and the anomaly was the same last year as it was 11 years ago, what exactly is the thing telling us in the first place, when last year’s value was the same as the trend over 125 years?
Anyway, next years I will pad and put at +.18
I’ve been trying to understand how Giss calculates a quarter when a month of data is missing. These are the basic rules I thought I came up with as being pretty much on the mark. After trying to do an extremely cold station I’m wondering.
To calculate a quarter with a missing month.
Calculate the long-term series mean for each month.
Round those values to one decimal. Add 273.15 Kelvin and round again one decimal.
Sum the individual monthly series mean for the quarter.
Calculate each months percentage share of that sum.
Sum the real data months given in Celsius and add 546.3 Kelvin.
Divide that figure by by the sum of the two real data monthly percentages.
Then divide by three and round to one decimal.
Subtract 273.15 Kelvin and round again to one decimal.
I estimated missing month data from a station at 30N in Algeria and it came out perfect to the decimal with Giss quarterly figures. Sample was about 10 different years where only one month was missing in a quarter. The kicker is that no negative temperatures were involved. It turns out there still seems to be something amiss. Either the way I’m doing it isn’t correct, or perhaps Giss has some sort of coding error.
By chance my second site was a very cold one and it is way off for the quarterly estimations. There is quite a gap in some of the estimates. Mine resulted in an average of three degrees cooler per estimate.
Anyone have any idea why?
The line above the month names contains the real data for the months in the quarter that are not missing. Months in the same quarter will have the same data up to the final two columns.
I’ll check in probably late tonight sometime in case anyone has any ideas.
Station ID:222246880006 OJMJAKON 63.2N 143.2E
Jan 1933 Giss estimate DJF -49.7
-49.9 NA -47.6
Dec Jan Feb Sum Dec% Jan% Feb% DJF
227.3 225.9 230.1 683.3 0.33 0.33 0.34 223.5 -49.7
Oct 1952 Giss estimate SON -22.5
-0.3 NA -45.8
Sep Oct Nov Sum Sep% Oct% Nov% SON
275.1 257.8 237.1 770 0.36 0.33 0.31 241.8 -31.4
Dec 1987 Giss estimate DJF -45.7
NA -48.9 -44.3
Dec Jan Feb Sum Dec% Jan% Feb% DJF
227.3 225.9 230.1 683.3 0.33 0.33 0.34 226.3 -46.9
Nov 1993 Giss estimate SON -21.2
-2.5 -20.4 NA
Sep Oct Nov Sum Sep% Oct% Nov% SON
275.1 257.8 237.1 770 0.36 0.33 0.31 252.1 -21.1
Apr 2000 Giss estimate MAM -11.8
-30.7 NA 3.7
Mar Apr May Sum Mar% Apr% May% MAM
241 258.6 275.1 774.7 0.31 0.33 0.36 256.5 -16.7
Lets see if a code block makes it look any better.
Station ID:222246880006 OJMJAKON 63.2N 143.2E
Jan 1933 Giss estimate DJF -49.7
-49.9 NA -47.6
Dec Jan Feb Sum Dec% Jan% Feb% DJF
227.3 225.9 230.1 683.3 0.33 0.33 0.34 223.5 -49.7
Oct 1952 Giss estimate SON -22.5
-0.3 NA -45.8
Sep Oct Nov Sum Sep% Oct% Nov% SON
275.1 257.8 237.1 770 0.36 0.33 0.31 241.8 -31.4
Dec 1987 Giss estimate DJF -45.7
NA -48.9 -44.3
Dec Jan Feb Sum Dec% Jan% Feb% DJF
227.3 225.9 230.1 683.3 0.33 0.33 0.34 226.3 -46.9
Nov 1993 Giss estimate SON -21.2
-2.5 -20.4 NA
Sep Oct Nov Sum Sep% Oct% Nov% SON
275.1 257.8 237.1 770 0.36 0.33 0.31 252.1 -21.1
Apr 2000 Giss estimate MAM -11.8
-30.7 NA 3.7
Mar Apr May Sum Mar% Apr% May% MAM
241 258.6 275.1 774.7 0.31 0.33 0.36 256.5 -16.7
I’m going to try this one more time. The code block looked good in the preview. Even had a scrollbar. I suspect I should have hit enter and moved to a new line after closing the tags. That may be why it didn’t show up properly. Gonna find out right now. Last shot in any case.
Station ID:222246880006 OJMJAKON 63.2N 143.2E
Jan 1933 Giss estimate DJF -49.7
-49.9 NA -47.6
Dec Jan Feb Sum Dec% Jan% Feb% DJF
227.3 225.9 230.1 683.3 0.33 0.33 0.34 223.5 -49.7
Oct 1952 Giss estimate SON -22.5
-0.3 NA -45.8
Sep Oct Nov Sum Sep% Oct% Nov% SON
275.1 257.8 237.1 770 0.36 0.33 0.31 241.8 -31.4
Dec 1987 Giss estimate DJF -45.7
NA -48.9 -44.3
Dec Jan Feb Sum Dec% Jan% Feb% DJF
227.3 225.9 230.1 683.3 0.33 0.33 0.34 226.3 -46.9
Nov 1993 Giss estimate SON -21.2
-2.5 -20.4 NA
Sep Oct Nov Sum Sep% Oct% Nov% SON
275.1 257.8 237.1 770 0.36 0.33 0.31 252.1 -21.1
Apr 2000 Giss estimate MAM -11.8
-30.7 NA 3.7
Mar Apr May Sum Mar% Apr% May% MAM
241 258.6 275.1 774.7 0.31 0.33 0.36 256.5 -16.7
RE #70 SamU.
Guessing the year from one month is an excercise in how people think.
Jan ANOMALY ( difference from a mean) is .12C. Assuming no trend in yearly temps
wouldnt your best guess for the Year be .12C? In fact the average of 120+ years
is Jan+.01C.. or about 1C/century. Simply, If I tell you jan is .12C your best
guess, based only on Jan ( and the record), is .13C. Weird that everyone guess higher. I think they may
guess higher because they always forget what ANOMALY means, until they are reminded.
I played around a bit with Guessing the year from DJ, Perhaps it will be fun to Show
DJF versus the year.
The other issues here is PROXIES, which no one has picked up on. So I’ll Say only
that a thermometer is a proxy. A tree is a proxy that MAY measure temps for a season
Suppose we only used thermometers during a “growing” season. How well would they predict
the year.
@Steven–
Since the climate system does have heat capacity, and surface temperature can be affected by what I would think of as sudden bursts, it’s actually reasonable to consider a very sudden drop or dip a burst. In that case, it makes sense to expect the temperature will more or less revert to last years temperature when the cold packet essentially mixes in.
So, if I were to use “Lumpy”, and make a conditional prediction, I’d assume Jan and Feb stay cold, and then do something to revert to the “Lumpy’s” prediction form March-December.
Of course, that still might not work. But it’s has some sort of rationality if you are wagering.
Hansen predicts a quarter when a month of data is missing as follows:
Tq = (1/3)TA + (1/2)(Tb + Tc) – (1/6)(TB + TC) where:
– Tq is the temperature of the quarter/season with the missing month
– TA, TB, TC are the average temperatures for each month A,B,C in the season, where the average is calculated using all existing temperatures for the specified month. For example, if a record has 10 January temperatures but only seven are valid, the average is calculated across only the seven valid temperatures.
– Tb, Tc are the actuals for the months in the season for which we have data.
All temperatures are in C, as they are recorded in the GHCN data.
In my implementation I do not round intermediate results. The seasonal results are rounded to one decimal point. I have yet to find a record that this approach does not give me the same result as I see in GISS.
For completeness, Hansen predicts an annual average when a single seasonal data point is missing as follows:
Tann = (1/4)TA + (1/3)(Tb + Tc + Td) – (1/12)(TB + TC + TD) where:
– Tann is the estimated annual temperature
– TA, TB, TC, TD are the average temperatures for each season A,B,C,D in the year, where the average is calculated using all existing temperatures for the specified season. For example, if a record has 10 MAM temperatures but only seven are valid, the average is calculated across only the seven valid temperatures. It is important to note that seasonal averages estimated in earlier years are a part of this average. Seasonal averages estimated in later years are not a part of this average. However, actual seasonal averages in later years are a part of this average.
– Tb, Tc, Td are the actuals for the seasons in the year for which we have data.
All temperatures are in C, as they are recorded in the GHCN data.
In my implementation I make two passes through the data. In the first pass I calculate estimated seasonal averages for the seasons where a single monthly temperature is missing. I do not round any intermediate results nor do I round the seasonal result in this pass. I then calculate the annual temperatures (or estimate them as described above, as necessary) using the unrounded seasonal values. The annual results are rounded to two decimal points. Then, I make a second pass through the data and round the seasonal averages to one decimal place. I have yet to find a record that this approach does not give me the same result as I see in GISS.
A final tidbit: Hansen will predict an annual temperature if up to six months of data are missing, but this requires the following pattern: all three months in a single season are missing, and exactly one month in each of the remaining three seasons are missing.
Example: if January, April, August, September, October, and November are missing, Hansen will estimate an annual average.
Someone may ask: while Hansen may theoretically predict an annual temperature if up to six months of data are missing, does it ever happen in practice. The answer is yes! More frequently than I might have thought. An example can be seen with Hoseda-Hard (ID 22223219000) – look at the record for 1995.
Thanks for that, John. I’ll try it out.
John, thanks for all your interesting work uncovering what Hansen has done to compute temperature. It bothers me that, in essence, he is substituting a model for data by all the manipulations to fill in missing data. The linear interpolation for a missing month has obvious problems. How often have you noticed a month with unseasonable weather? It’s quite common but this washes it out. What’s the justification for the missing month computation, missing season computation?
#81. Look, there’s missing data in the record. That doesn’t mean that the scientists involved should just throw up their hands. Hansen doesn’t substitute model data for missing data, he does simple interpolations. That’s not per se unreasonable. The issue that we’ve raised is a very narrow and technical one – that the method that he used for splicing records is quite peculiar and can lead to some pointless biases, especially given the synchronous inclusion of so much MCDW data. It’s not that these particular biases overthrow any particular theory, but they’re the sort of thing that really make you scratch your head. There’s no reason either to get particularly excited about the peculiar splicing method, but neither is there any reason why a more sensible method shouldn’t be used.
John,
I used the formula you kindly provided to do a small test of the estimation accuracy using the Giss style series mean and a mean only up to the year of the missing month. In this test there didn’t seem to be much difference though I suspect the Giss style probably has a slight edge.
I identified all US stations that had perfect data for the J-J-A season for every year between 1951 and 2005.(300+) Estimated each July temperature for 25 of the stations and subtracted the observation figure to find the amount of error. Did the same for January for the same stations as there were only three missing data points between them in the station sample I used.
So far it appears the estimation method is somewhat closer during the warm season than the cold. Estimates are high in the cold season and low in the warm. Now I’m wondering if this is a seasonal difference, a cold climate versus a warm one. Sample is too small to tell.
Here is the spreadsheet.
I’ll have to automate it properly and do a larger sample.
That is if real life doesn’t get in the way of my virtual life too much. 🙂
Paul: What’s the justification? I could say ‘hey, it’s climate science’. Or ‘because he wants to’ But instead I’ll answer you that it’s justified because anything that supports AGW is justified, of course.
John; You said
Have you tried the method on a year that has all the monthly temperatures there but you randomly remove months and see how well the estimation process works to predict the number you already have?
mosh: You do things that react directly to temperature in real-time in predictable verifiable ways a diservice refering to them as proxies; comparing them to tree rings and ice cores, how cruel! You should be ashamed of yourself; appologize right now to mercury expanding and contracting, bandgap voltage comparisons between different currents, and coiled bimetalic strips. Now, look what you’ve done, you’ve made Delta V-be cry; it gave the answer that let the circuit say 100C and the water just started boiling. Great, now you’ve given it a complex.
Seriously, the thermometers themselves aren’t proxies; as long as we’re talking about the temperature they give, at the time they give it. (This assumes the thermometers are actually measuring the normal ambiant temperature and not some human, placement or instrument bias). What is a proxy (the first one in a string of them) is using the Tmin and Tmax out of the temperature samples of the day (or more likely part of the day) to get Tmean. Tmean is a proxy for the “day’s temperature”, which is a proxy of the variably sized area, which is then averaged into a proxy for the month to calculate the anomaly, which is a proxy for the temperature delta for the month for that area compared to other multiple proxies such as itself; the base reading being just like it is, but also proxied from being averaged over multiple months).
Sorry, OT. How anyone can estimate that month X will be something, I have no idea. Logically, if December is 0.5 would I estimate Jan would be 0.1 or 1.0 But it could be. More guessing… By quarter, how about .2 .4 .6 .5 which would put the year at .425 Oh, hey, we’re missing Mar; Jan was .4, Feb was .4, so I’ll estimate Mar at .4
I would guess John could answer that better…
Steve,
When only a few records are missing, it is reasonable to use some kind of extrapolation to guestimate the missing number. Personally I feel that a more sophisticated method should have been used. My prefered method would have been to use adjacent years to determine what the trend for that month was, then apply that same trend to the year in question. Possibly also refering to the same three months in nearby stations to see if something unusual was happening that year.
As I said, when only a few records are missing, it is reasonable to try to estimate the missing data, however as the amount of missing data increases, there has to be a point at which throwing up your hands and throwing out the station is the only reasonable response.
John Goetz #76. If you express Ta=(1/2)(Tb+Tc) +1/6(TA-TB) +1/6(TA-TC) it looks more reasonable. As you say Ta is the estimated temperature for month A. So Hansen estimates the temperature for month A starting with an average of the 2 months (B&C) that have data and then adjusts by the amount that the historical record of A differs from the historical records of B&C. I presume that he uses a weighting of 1/6 for the historical adjustment based on empirical studies that he did. Otherwise I have no idea where the 1/6 comes from.
Same holds true for the estimate of seasonal temperature in #77.
In high lattitudes a station within a distance of 500 km works as a good predictor for it’s neigbour. You could use that to fill the gaps. That’s what was done with CET by Manley: he used Dutch data.
#82
I stand by what I wrote earlier, interpolation is a model. What’s the justification for the formula, where do the coefficients come from? Unseasonable weather is very common but interpolation smooths that away. Missing data is by definition unknowable.
Certainly not, but they have to use methods that don’t introduce false data. It would be much better if they simply averaged available data and expanded the error bars accordingly. Which brings up another irritating problem with climate science analysis, what are the errors?
#86 wkkruse
Actually, the equations are mine. I have never seen the equations as I have described in Hansen’s code, nor have I seen them in his written material. We had discovered that Hansen estimates a missing monthly temperature by finding the average temperature for that month and adding to it a delta temperature. The delta temperature used is the average of the delta temperatures for the other two months in the season. I derived the final equations as shown above so that I would have one simple expression to program. Through a few trials I also figured out when he does his rounding. See here for how the equations were derived.
#84 Sam
Yes, I have played around a lot randomly removing months from the record and seeing how close I get. I can’t say I see any particular pattern or trend in the results, other than I have yet to see it produce the actual temperature. I’ve toyed with doing a Monte-Carlo style simulation where I make hundreds or thousands of runs against the same set of stations, randomly removing records and seeing what kind of distribution I get out of it. Right now it is just a thought.
#81 Paul
Yes, a month with unseasonable weather is generally washed from the record using this method. While no estimation method is perfect, I do have have two problems with what is being done here.
First, the method used has the potential to change old estimates every time new data is added to the record. I’ve run a number of experiments and have seen that happen very, very frequently. The changes may not be huge (0.1 to 0.8 degrees perhaps) and the estimates seem to converge on a single value after a while (although no one knows how accurate that value is), but it bothers me to know that from one year to the next the old data is changing.
Second, I see no reason to estimate monthly or seasonal values at all. Hansen chooses to run his gridcell analysis using annual temperatures, but why he can’t he just do it with the available monthly temperatures at any given point in time? Then, at the end of the year take the twelve monthly averages and calculate an annual average. We could argue that different stations contribute to the monthly average, but that happens with the annual temperatures anyway.
John Goetz #89
In # 76 above you said that Ta was the temperature of the missing month. I looked at your earlier derivation cited in your reply in #89, and it seems that your equation for Ta in # 76 is supposed to be the estimate for the quarter that is missing the temperature in month a. However, in reviewing your earlier derivation for the quarterly estimate, I can’t get your result. I get (using the notation in #76 above) that
Tq=TA/3+(Tb+Tc)/6+(TB+TC)/6, which I think makes more sense since it properly weighs the 3 months.
#90 wkkruse
Thank you for pointing out the error in comment #76. It should read Tq= rather than Ta=.
I do not get your resulting equation, however. I would need to see your step-by-step derivation to comment on it. While I now see that I had an error in the intermediate result in the post I referred you to, the resulting equation is still correct (as far as I can tell). I’ve rerun it a bunch of times on GISS data and I get the same results as Hansen.
One trap to watch out for is that the December value you must use is from the previous year, not the current year. That is a programming error I recall stumbling into.
John Goetz #91
In your post of Sept 6 to which you referred me, the next to last line in your derivation of Tq is
Ta=TA+(TB+TC)/2 -(Tb+Tc)/2 where Ta is the estimated temperature for mont a that is missing data. (This comes about from deltaTn=TN-Tn) Since Tq= (Ta+Tb+Tc)/3 then by substitution
Tq=[TA+(TB+TC)/2 -(Tb+Tc)/2 +Tb+Tc]/3 = [TA+(TB+TC)/2+ (Tb+Tc)/2]/3=TA/3+(Tb+TC)/6+(Tb+Tc)/6.
On the other hand if deltaTn=Tn-TN then I get your result for Tq.
There is a typo in #92. It should say that the final result Tq=TA/3+(TB+TC)/6+(Tb+Tc)/6
#93 wkkruse
It should be deltaTn = Tn – TN. When I was solving the problem back in September, I figured Hansen was adding a bias to the month’s average in order to get the month’s estimate. So my second step in that derivation is correct. I got sloppy with the Latex in the fourth step, which should read deltaTn = Tn – TN instead of deltaTn = TN – Tn. The signs are backwards in the fifth step as well. Fortunately, I had worked it out in a notebook and was trying to copy from my notebook into the Latex, and I did copy the sixth and final step correctly.
When I get a chance tonight I will post a complete and corrected derivation – hopefully without the typos. What I posted in the past is incomplete in that it does not specify when rounding of results occurs, nor does it specify when the seasonal averages are calculated.
John Goetz #95
My take on this now ( and I assume it’s the same as your take) is that the GISS process can be thought of as estimating the deviation of the temperature in month a from its long term average. And it does this as as the average of the deviation of months b and c from their respective long term averages. In other words,
DEVa=[(Tb-TB) +(Tc-TC)]/2. Then the estimated temperature for month a is TA+DEVa which leads to your result for Tq.
Thinking about it like this makes it sound more reasonable. I would hope, however, that GISS tested this method against other alternatives.
I’m going back to lurking.
4 Trackbacks
[…] this year I did a post on the amount of estimation done to the GHCN temperature record by GISS before generating zonal and […]
[…] seen is written by Steve McIntyre over at Climate Audit in two separate discussions. From February 9th: Unfortunately, the previous year’s December temperature is not included in the first year of […]
[…] I viewed the data using both a 1200km and a 250km smoothing radius. Doing so helped me gauge the station coverage and the extent the 1200km smoothing algorithm estimates temperatures over the vast unsampled swaths […]
[…] How much Estimation is too much Estimation? […]