Released today on the promised schedule are CRUTEM3 and HadCRUT3 for October 2008. October 2008 was in the top 8 crutem3 (0.517 deg C)and in the top 6 hadcru3 (0.440 deg C) Octobers.
Because our collective eyes right now are fairly attuned to the colors of these grids and how changes in individual stations affect the contours, it’s interesting to take a look at these new contoured maps and compare them both to each other and to the GISS contour map.
First the CRUTEM3 (top), HadCRUT3 (middle), GISS (Nov 13 version – 250 km smoothing rather than 1200 km of the frontpage GISS image).
First a small point. Living in Toronto, I often look first at how the maps represent Toronto since I know what the weather’s been like here. For the most part the land portion of the HadCRUT3 map is identical to the CRUTEM3 map, but not where I live. In CRUTEM world, we experienced a colder than average October (which is how it felt on the ground here), while in HadCRU world we experienced a warmer than average October. One possible and even likely explanation is that HadCRU includes temperatures from the Great Lakes (which weren’t warm for swimming this year.) I didn’t go swimming at our place on Lake Ontario once this year. But maybe October water was less chilly than usual.
More points after you look at the graphics.
A second small point in the light of our watching the ball as stations got added to the GISS map of the Canadian Arctic. The land station contributions to HadCRU3 look like a slightly later GHCN version than CRUTEM3. HadCRU3 has a gridcell a bit to the southwest of Ellesmere Island (presumably Resolute) that got added after Nov 7, according to Gavin Schmidt. This gridcell is absent from the CRUTEM3 version. The Nov 13 GISS version lacks both Resolute and Alert for reasons, presumably because of an oversight as they put patches on patches.
A third point – the global average in the GISS 250 km version is 0.78 deg C, while it’s 0.61 deg C in the 1200 km version.
The CRUTEM3 version looks a lot like the GISS version. Although these compilations are often described as “independent”, recent events have clarified (if clarification were needed) that these compilations are not “independent”. Both rely almost exclusively on GHCN – GISS adds a few series around the edges.
The reverse engineering of CRUTEM3 looks almost pathetically easy given that we’ve already waded through step 0 of GISS, where they collate different GHCN versions (dset0) into a single station history (dset1.) CRU doesn’t have the bewildering sequence of smoothing operations that Hansen uses at multiple stages (though Hansen, mercifully, doesn’t use Mannian butterworth smoothing).
To my knowledge, unlike GISS, CRU does not make the slightest attempt to adjust for UHI, relying instead of articles like Jones et al 1990 purporting to show that UHI doesn’t “matter”.
We can already emulate GISS step0 – not that it makes any sense, but it provides a benchmark. Here’s all that seems to be necessary to produce a gridded CRUTEM3 series given a dset1 data set. First, create an anomaly-version of the series. I have a simple function anom on hand and this could be done as follows:
dset1.anom=apply(dset1,2,anom)
Then one could make an average of dset1 series within gridcell i as follows, where info is an information dataset in my usual style containing for each station, inter alia, its lat, long and gridcell number (called “cell” here):
for (i in 1:2592) grid[,i]=apply(dset1.anom[,info$cell==i],1,mean,na.rm=T)
This would yield the CRUTEM3 series. My guess as to why they don’t want to show their work is because they probably use hundreds of line of bad Fortran code to do something that you can do in a couple of lines in a modern language. Anyway, I’ll experiment with this at some point, but this is my hypothesis on all that’s required to emulate CRUTEM3. CRU has been funded by the US DOE; if, like GISS, they are doing nothing other than trivial sums on GHCN data, one feels that the money would be better spent on beefing up QC and data collection at GHCN.
I downloaded CRUTEM3.nc (today’s version) and checked for gridcells with October anomalies of 5 deg C or higher and then checked to see what stations were in those gridcells. I obtained teh following list, all but one in Siberia, the other one being Barrow, Alaska (where there is an extraordinary contrast between nearby stations that deserves comment.)
1710 22223724000 NJAKSIMVOL’ 62.43 60.87
1716 22223921000 IVDEL’ 60.68 60.45
1729 22224817000 ERBOGACEN 61.27 108.02
1720 22224125000 OLENEK 68.50 112.43
1723 22224329000 SELAGONCY 66.25 114.28
1721 22224143000 DZARDZAN 68.73 124.0
1724 22224343000 ZHIGANSK 66.77 123.4
1686 22220069000 OSTROV VIZE 79.5 76.98
1688 22220292000 GMO IM.E.K. F 77.72 104.3
1693 22221432000 OSTROV KOTEL’ 76 137.87
1685 22220046000 GMO IM.E.T. 80.62 58.05
3361 42570026000 BARROW/W. POS 71.3 -156.78
It looks like CRU has been paying attention to the GHCN commotion and has avoided using one of the problem versions. It is however interesting that some of the above stations (Olenek, Erbogacen etc) were problem stations and one would hope that one of the data distributors has actually checked these stations against original daily versions.
Climate Audit 
59 Comments
Nice post. I predict very little heat.
As to your last sentence is there a specific results oriented reason you feel this is bad procedure? ( I asked Nicholas the same thing in an earlier thread. ) Unless the values are discrete I will usually allow multiple bins or less often treat the boundary condition like the proverbial donkey and hay piles.
Changing the title was a nice gesture. Thanks.
Steve: I moved the comment about filters to an earlier spot in the post. I’m wary of Mannian-padded butterworth filters because they are designed for a completely different purpose – things like audio receivers, and focus on recovering frequencies rather than end points. In climate studies, you have negligible interest in recovering specific frequencies and considerable interest in not screwing up the end points. So they seem about as poor a choice of filtering method as one could make. Plus I instinctively resist the idea that an applied study should concurrently be developing smoothing functions. If Mann wants to advocate Mannian smoothing, publish in a statistics journal not in a climate journal where none of the reviewers or readers really have any idea whether the concept makes any sense or not.
Steve, in my layman’s way, I’ve gone through the stations you listed above. Basically, what I’ve tried to demonstrate is that EVERY single one of them with data from pre-1951 (i.e. the start of the 1951-1980 GISS baseline), had at least one October pre-1951 that was milder than October 2008. This has obvious implications in terms of how appropriate the 1951-1980 baseline may be in judging what is “normal”. (I’m doubt you’ll be surprised by what I’ve come up with, but I hope it’s useful work anyway).
Also, the following links provide an interesting graphical representation of a couple of the mildest pre-1951 Siberian Octobers, to compare against October 2008:
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2008&month_last=10&sat=4&sst=0&type=anoms&mean_gen=10&year1=1943&year2=1943&base1=1951&base2=1980&radius=250&pol=reg
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2008&month_last=10&sat=4&sst=0&type=anoms&mean_gen=10&year1=1947&year2=1947&base1=1951&base2=1980&radius=250&pol=reg
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2008&month_last=10&sat=4&sst=0&type=anoms&mean_gen=10&year1=2008&year2=2008&base1=1951&base2=1980&radius=250&pol=reg
1710 22223724000 NJAKSIMVOL’ 62.43 60.87
Oct 2008 +2.9C
Oct 1944 +3.0C
Oct 1955 +3.3C
1716 22223921000 IVDEL’ 60.68 60.45
Only goes up to 1990?
1729 22224817000 ERBOGACEN 61.27 108.02
Oct 2008 -1.1C
Oct 1947 -0.7C
1720 22224125000 OLENEK 68.50 112.43
Oct 2008 -6.9C
Oct 1947 -3.3C
Oct 1948 -6.2C
1723 22224329000 SELAGONCY 66.25 114.28
Only goes up to 1990?
1721 22224143000 DZARDZAN 68.73 124.0
Oct 2008 -6.6C
Oct 1943 -6.5C
Oct 1947 -3.6C
Oct 1949 -6.4C
Oct 1951 -6.0C
1724 22224343000 ZHIGANSK 66.77 123.4
Only goes up to 1990?
1686 22220069000 OSTROV VIZE 79.5 76.98
Note: only goes back to 1951
Oct 2008 -5.4C
Oct 1985 -4.5C
(Oct 1983-1985 average -5.3C)
1688 22220292000 GMO IM.E.K. F 77.72 104.3
Oct 2008 -8.1C
Oct 1938 -6.9C
Oct 1943 -5.2C
Oct 1945 -7.1C
Oct 1948 -7.5C
Oct 1954 -7.9C
Oct 1947 -3.9C
1693 22221432000 OSTROV KOTEL’ 76 137.87
Oct 2008 -7.2C
Oct 1943 -6.4C
Oct 1947 -4.6C
1685 22220046000 GMO IM.E.T. 80.62 58.05
This is weird. Only appears to go up to ~2000, but then looking at the data it has anomalies for a few months since then. The only month with data in the last 3 years is…… Oct 2008!
(Note: only goes back to 1957)
Oct 2008: -6.7C
Oct 1959: -5.2C
3361 42570026000 BARROW/W. POS 71.3 -156.78
Oct 2008: -5.1C
Oct 1902: -3.8C
Oct 1911: -2.7C
Oct 1938: -3.8C
Oct 1940: -4.8C
Oct 1949: -4.5C
Re: Chris (#2),
Nice work, Chris! The raw station data often show warmer periods than now half a century or more ago. Alas, this is often obscured by quaint “homogenization” of station records in the anomaly sausage-machine, which introduces segmented trend-adjustments around some “knee” date. Those expecting results reflecting unadulterated measurements can bury their heart in that knee.
There is something that I don’t quite understand about the middle (HadCru) map. According to the web site you linked it to:
If you look at roughly (100W, 60S), there are a + and a – in two adjacent girid blocks. Although records in opposite directions located next to each other in the same month might be possible, I would think it highly unlikely. However, that isn’t the strangest thing. At (120-140W, 80N), there are three minus signs in grid blocks which appear to be coloured yellow and red – indicating positive anomalies with one of them in the one to three degree range. Most of the other minus signs are in blue boxes. Since these anomalies pertain to the specific grid blocks that they occupy, this would appear to be an oxymoron: a “positive minimum anomaly”. Any idea on what’s going on here?
This is a very enlightening post!
Toronto is warmer in HadCRUT than in CRUTEM probably because, as Steve suggests, they included lake surface temperatures.
To test this hypothesis I used a primary synop station, located 100 km far away from where I live on the Adriatic sea coast in Italy – 16149 Rimini.
October was a warm month with an anomaly of +2.04 °C.
Correctly, CRUTEM and Gistemp show Italy and all eastern Europe as being quite warm.
Unfortunately, the second half of September and the first week of October were vey cold and the water of the Adriatic sea, a very shallow sea basin, got unusualy colder than normal. Cold sea water persisted during the month of October even if air temperature was very warm for the season.
HadCRU, accordingly, changed a warm October, experienced in Italy, in a cold one. That’s a miracle!
Now look at the British Isle. It was quite cold there but HadCRU says it was very warm. The same happened in France, in Morocco and in the Northern Atlantic around Iceland. In the SH the only place I found was in Queensland.
With the exception of Italy, all the other places warmed from CRUTEM to HadCRU and I’m sure this had an impact on global anomaly.
Sure, sst leads air temperature, but someone could tell Jones and his collegues at CRU that strong wind advection can and do produce a sustained situation in which sea water and the above air are decoupled.
They simple change facts!
A recent study published in Nature and based on a simple GCM is predicting that we will be going into a quasi permanent “big chill”. Hansen’s comment is as follows:
James E. Hansen, NASA Goddard Institute for Space Studies:
“Look at Figure 3 in our “Target” paper Target CO2: Where should humanity aim? [pdf]. Yes, the Earth has been on a 50-million-year cooling trend with superposed glacial-interglacial oscillations.* It would take only a small further reduction in climate forcing (less long-lived GHGs or whatever) to yield more ice during the glacial phase of glacial-interglacial oscillations. But that is entirely academic at this point, unless humans go extinct. Although orbital variations instigate the glacial-interglacial swings, the mechanisms for climate change are changes in GHG amount and surface albedo (as we show in Fig. 1 of our paper) — those mechanisms are now under the control of humans. Another ice age cannot occur unless humans go extinct. It would take only one CFC factory to avert any natural cooling tendency. Our problem is the opposite: we cannot seem to find a way to keep our GHG forcing at a level that assures a climate resembling that of the past 10,000 years”
Do I see an alternate causality (albedo i.e. land use) other than CO2 in the above statement that could explain any slight AGM over the last 150 years?
See link
http://dotearth.blogs.nytimes.com/2008/11/13/more-on-whether-a-big-chill-is-nigh/#more-503
Re: Ed (#7),
In contrasting the competing philosphies of climate-as-living-organism versus climate-as-mostly-machine, it is in Dr. Hansen’s opinion that humans could somehow prevent another ice age through C02 management practices wherein the climate-as-machine type of thinking reaches its most preposterous and absurd proportions.
Being a child of the 1960s television era, it is easy for me to envision a Smokey the Bear cartoon character pointing his finger out at the audience and intoning in a deep resonant voice, “Onlyyy youuu can prevent global warming.”
Now, it is apparent that even the goal of taking the earth’s temperature accurately while employing some acceptable level of scientific rigor and precision is a very difficult and expensive proposition, one which has not so far been entirely successful — let alone thinking we could somehow be successful at controlling the earth’s temperature and climate through carbon management practices.
In watching the ebb and flow of debate over the true affect and impact of these emerging issues over just how accurately and objectively the earth’s temperature is currently being measured, one might think to ask, is the instrumental approach necessarily the most “accurate” means we have at our disposal?
Is it not possible that with enough scientific rigor and discipline, that secondary, non-instrumental indicators might prove to be more useful tools in determining where temperature trends have been, and where they might be going, assuming of course that our examination of the secondary indicators is pursued in an intellectually honest and accountable fashion?
Well we know that GISTemp doesnt adjust for grids that are
part sea and part land ( at least I find no reference)
HADCRUT does…
Blending a sea-surface temperature (SST)
dataset with land air temperature makes an
implicit assumption that SST anomalies are
a good surrogate for marine air temperature
anomalies. It has been shown, for example by
[Parker et al., 1994], that this is the case, and
that marine SST measurements provide more
useful data and smaller sampling errors than
marine air temperature measurements would.
So blending SST anomalies with land air tem-
perature anomalies is a sensible choice.
I wonder if Parker looked at the the great lakes??
Continuing
Previous versions of HadCRUT
[Jones, 1994, Jones & Moberg, 2003] blended
land and sea data in coastal and island grid
boxes by weighting the land and sea values by
the area fraction of land and sea respectively,
with a constraint that the land fraction cannot
be greater than 75% or less than 25%, to
prevent either data-source being swamped by
the other. The aim of weighting by area was
to place more weight on the more reliable
data source where possible….
And then they go onto describe the new method I believe RomanM has addressed some problems with it elsewhere on CA ( am I remembering correctly Roman?)
But does SST include lakes?
suggests maybe not. I’ll have to go get the data.
Re: steven mosher (#8)
You’ve got it right! This violates the principles of stratified sampling (see here for example) where estimates from two or more different strata are combined into a single estimate. The weights that should be used are determined not by how much information is available from each stratum, but by the (fixed) proportion of the population that the particular stratum represents. If you combine your estimators otherwise, you introduce bias (systematic over- or under-estimation of the parameter in question) thereby losing any possible gain in the reduction of variability.
In the case of the temperatures, presumably the relatives weights for land and water in a grid box could also change over time as measurement sites are added or removed creating artificial differences in the anomaly sequence. Since the relative weights differ from grid box to grid box, changes in SSTs would also produce effects non-uniformly across the grid boxes.
Re: tty (#9),
Your explanation doesn’t hold water! (chuckle) I can’t imagine someone actually basing anomaly records on virtually non-existent information. 😉
RomanM:
“Although records in opposite directions located next to each other in the same month might be possible, I would think it highly unlikely.”
Not that unlikely. Remember that HadCRU uses surface measurements for SST. These gridboxes are down in the Southern Ocean where there is essentially no ship traffic. Say that there is one previous October measurement in each box. Now a ship passes through and takes two new measurements. One is slightly higher than the previous one and the other slightly lower. Presto! Two new records in opposite directions.
As for Hansens argument about ice ages, the problem is that we know quite well that it insolation changes that moves the climate both out of and into glaciations, and that the changes are quite abrupt, much faster than can be explained by changes in GHG (changes on the order of 5 degrees or more globally in 50 years). Apparently there are threshold effects where global circulation changes rather abruptly. As a matter of fact this change between glacial and interglacial climates is the only “tipping point” for which there is any actual evidence.
the source data has coastal marine data for the great lakes
need to read.
Click to access rayner_etal_2005.pdf
charts there seem to indicate no coverage??
SteveMc Might be instructive to do a map of
http://hadobs.metoffice.com/hadsst2/data/download.html
“But does SST include lakes?
suggests maybe not. I’ll have to go get the data.”
I don’t know if SST includes lakes, but there is at least three SST temperature records well inland in the October data. One near the southern Caspian might be understandable, but the other two would seem to refer to the lower Mackenzie River and Lake Peipus in Estonia respectively. “Curiouser and curiouser”.
Steve, can you help me clarify something. CRUTEM is land temperature from stations. HadCRU is land temperature from CRU plus Hadley temperature from the oceans. Is the Hadley ocean temperature from satellites? And what is GISS – land only, or sea data from satellite as well? Also, is there anywhere that the balloon temperature data since the late 50s can be accessed to compare it with the surface records?
D Elder,
HadCRU uses sea surface temps from ships for its ocean air temperatures (assumes air temp anomaly equals sea surface temp anomaly), GISS uses satelites.
Anomalous warmth over Siberia is the primary surface temperature signature of the Arctic Oscillation’s positive phase.
A large positive AO excursion occurred during October, as can be seen here.
With this Siberian stuff, don’t forget the “Divergence” between GHCN Monthly and GHCN Daily records for a spot check on Verhojansk, a site in the heart of the Siberian zone.
A very reliable, but UHI-contaminated station in western Maryland (USA) near me:
http://i4weather.net/oct08.txt
had October at 1.8F (.85C) below avg.
Oh wait, sorry, Hansen says UHI is insignificant.
Substitution of SSTs for air temps in HadCRUT time series is scientifically problematical in many respects. The two variables are seldom highly coherent even when co-located, as bouy measurements show. Ship reports are never truly co-located and, in most grid cells, are not frequent enough to provide a basis for constructing proper daily averages for every day of the month, necessary for avoiding aliasing errors. But most egregious is the wholesale adulteration of SSTdata by the unproven “adjustment” of bucket temperatures. No less than the “homogeneity” adjustment, this introduces a twiddle factor that is responsible for most of the trend seen in the annual global seWhile Hadley may have been more prudent than GISS in dealing with the recent monthly replication fiasco, they produce a data sausage whose ingredients are even more unreliable.
Steve,
wouldn’t it be better to use this GISS so all have the same 61-90 baseline.
When I do that and compare to met.no ,GISS and Hadley show Norway too warm. Judging from their maps Norway was at least 1 deg C warmer but the met.no says it was 0.4 C warmer. For the western parts of the country it’s even worse. GISS and Hadley show +0.5 to +1 but reality says -0.5 to -2.
Another example of inaccurate GISS here
Several areas of Siberia show negative anomaly at meteo.ru autumn of 2005, they are all positive at GISS.
Or is it just a matter of averaging?
Does anyone know how to get at meteo.ru station data in an ASCII or zip form? I used to be able to get at it, but they changed the formats and now I can’t find it.
Re: Steve McIntyre (#22),
Is this the place you are looking for?
http://meteo.infospace.ru/wcarch/html/e_sel_admin.sht?country=176
I had to turn off Google toolbar before it stopped blocking the popup box containing the results from appearing.
#23. Sort of. But you used to be able to access zip files with temperature histories for a given station which you could specify by id#. Now I can’t locate any url’s. I HATE it when agencies prevent being able to access data through hokey JAva stuff.
http://meteo.infospace.ru/wcarch/html/e_sel_stn.sht?adm=612
scroll to the location verhoyansk then the file icon
select zip. I’ll try it and send to you
here it is
http://www.panoramio.com/photo/8225894
conard,
Sorry, I hadn’t seen your question. I deal mostly with frequency domain signals so I’m not really qualified to answer this in a statistical context. However, from what I have read, there are good reasons to avoid smoothing at all. However, if you have to do it, it strikes me that a Gaussian filter would be far superior in this type of situation. Butterworth is best used when the corner frequency is far below or above the frequencies you are interested in. In the case of climate data, I don’t think that’s the case. If your filter window is 20 years and your data is 600 years long, that’s simply too close to the sort of period which will make a difference. A 10 year distortion at the end could affect the trend enough to matter. Even just a simple moving average seems to me like it would have better understood and milder effects than a reactive type filter.
To B Buckner re information in comment 13 thanks very much from D Elder. Pity the Mann men aren’t as cooperative!
Very nice write up in the UK Telegraph, with a corresponding link at the Drudge Report.
Written in haste, so I may not have seen if this was already discussed. Anyway, being UK-centric rather than Toronto-centric, I am concerned about the Hadcrut3 anomaly for UK apparently being positive (by the colour on the map), when we had our first London snow in October for 70-odd years, and CET daily max achieved its lowest value since 1974. Is this an example of grid cells overlapping land and sea, and the sea anomaly has somehow outweighed the land anomaly?
Will they still do the same combination in 20 years time when we have colder sea temperatures and higher land temperatures?
Expert explanations would be welcome.
Rich.
#25 Janama,
I am not going there for my holidays !
Hmmm…the October anomaly map from GISS seems to have changed yet again. The Siberian hotspot has shrunk and the Canadian Arctic hot spot that Steve noted had appeared in the GISS Nov 12th revision is gone. Are we starting to see some actual “quality control” from GISS?
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2008&month_last=10&sat=4&sst=0&type=anoms&mean_gen=10&year1=2008&year2=2008&base1=1951&base2=1980&radius=1200&pol=reg
for grins do 100km radius.
http://data.giss.nasa.gov/cgi-bin/gistemp/do_nmap.py?year_last=2008&month_last=10&sat=4&sst=0&type=anoms&mean_gen=10&year1=2008&year2=2008&base1=1951&base2=1980&radius=100&pol=reg
it would be interesting to do a sensitivity studies of radius versus anomaly over time
On RealClimate in response to Comment 221
RE. myself.
the 100kn radious is actually 250km. Over the “the historical” record, looking at the annual mean over 10 year periods, there is no real difference between 250km and 1200km averaging.
The piece also gets a shot in at the peripatic head of the UN globabl heating mission:
The Telegraph piece should get some attention among members of Congress!!
Here is the RSS temperature anomaly for October:
Let’s compare with the HadCRU map, where I put in evidence some areas that differ with the RSS map.
Now I try to give some explanation for each of these differences.
1) Barrow: all Alaska was very colder than normal with the exception of the northen coast. Possible explanations: more frequent wind from the sea to the north which is usualy warmer; UHI problems; later sea ice formation or snow cover in that area.
2) In the middle of Labrador: surface station problem.
3) USA Pacific waters: colder water with warmer air aloft; typical sea-air decoupling, wrong CRU’s assumption regarding near coast waters.
4) Eastern North Atlantic: cold air advection onto warm sea in the extratropics. Nothing unusual, wrong CRU’s assumption.
5) Italy-Adriatic sea: colder water with warmer air aloft; typical sea-air decoupling, wrong CRU’s assumption regarding near coast waters.
6) Eastern Equatorial Pacific: was sst so warmer than normal there? Anyway air temperature was normal. Further investigation.
7) Waters off Somalia: probable strong upwelling. Colder water with warmer air aloft; typical sea-air decoupling, wrong CRU’s assumption regarding near coast waters.
8) Waters south of Java: colder water with warmer air aloft; typical sea-air decoupling.
9) Eastern Australia: cold air advection onto warm sea in the extratropics. Nothing unusual, wrong CRU’s assumption.
10) Far Indian Ocean to the south of Madagascar: cold air advection onto warm sea in the extratropics. Nothing unusual, wrong CRU’s assumption.
11)Waters off Chile: cold air advection onto warm sea in the extratropics. Nothing unusual, wrong CRU’s assumption.
12)Argentina: it’s hard to understand what’s going on there. The only reason to have warmer temperature at the ground and colder aloft, I think, is a problem with the surface stations (UHI, omogenisation).
What’s the effect of all these assumptions on global mean?
I try again for the images.
RSS:
HadCRU
Steve said:
I don’t know who “they” are or what you mean by “patches on patches”. Perhaps you could clarify what you are talking about. I do hope that you understand that the change was in GHCN, not GISS:
Re: Deep Climate (#41),
When I checked a few days ago, the October mean for Resolute seemed a few degrees above normal. In November there were 3 days of data missing, so that may be why November is not used.
#41, I’ve discussed problems at GHCN in the past on numerous occasions. They are nearly 20 years late in updating many stations and their update patterns are erratic. GISS uses them as a supplier and like any other distributor, they need to take care that their supplier has proper QC. Like most readers here, I don’t get the purpose of fingerpointing.
I just examined versions of the data set and the GHCN version on Nov 14 12.30 am had removed the Resolute station (presumably because they went back to a version before it was added to try to fix the Russian problem.) It was in the Nov 12 version and is once again in the Nov 16 version.
But in either case, the Canadian Arctic had been discussed in commentary; the change was noticed instantly by many readers including myself. Gavin Schmidt had already noted that the information was available as of Thursday; the change was readily noticeable so it’s pretty goofy that GISS didn’t work with their supplier to deal with the problem for their update.
I just checked Resolute against Canadian source data. It looks like there was a carryover incident earlier in the year, operating the other way. I’ll do a post on it separately.
Re: Steve McIntyre (#44),
Am i missing something–aren’t these people paid to do this job? Is this kind of shoddiness common?
the giss quality checks are made manually. prior to processing. have a look at Karl 98 again for a flow diagram of the GHCN QC checks. figure 1.
“It looks like there was a carryover incident earlier in the year, operating the other way.”
I had a hunch there would be more of those. I wouldn’t be surprised to learn that there are all sorts of these errors sprinkled through the record. Sounds to me like it is time for an overhaul somewhere and a wholesale regeneration of that data from the original source files is in order if there is to be any trust in those data.
Re: crosspatch (#46),
Oddly enough, introducing extra noise above and beyond “weather noise” into the record makes it more difficult to demonstrate projections are wrong. Noise of any sort means means that we are more uncertain about mean trends.
That means that these errors will impact attributions studies and our ability to discern whether models are on track or off track.
re 48. You’ll note Lucia that for attribution studies the PIs reduce the number of models that they
use, for example screening models that exhibit too much drift during control runs. That way they get a tighter ensemble when then do comparison studies ( with and w/o C02 forcing) If they
ran all the models, like they do in hindcast studies, the probability of the with and w/o studies overlapping is higher
Re: lucia
‘introducing extra noise above and beyond “weather noise” into the record makes it more difficult to demonstrate projections are wrong’
It would seem to me that it would make it more difficult to demonstrate anything at all.
At the moment if I try to make a map – regular or polar view – at GISTEMP with the 250 km Smoothing Radius I now get broken links. Do others get the same?
Make map here Be sure to clear cache first.
(Mapmaker works fine for 1200km Smoothing)
crosspatch (#51),
Yes, It makes it more difficult to demonstrate anything at all. But, if you have been following the “not inconsistent” chronicals by Roger Pielke Jr. noisy measurements means one would constaintly decree all sorts of things are “not inconsistent” with data.
re 52 worked fine yesterday. I did about 20 of them
Re: steven mosher (#54), Thanks Mosh. looks like it is fixed now. The comment about the “grahics bug” has been removed as well I think. Just a hiccup!
Steve Mosher, if you feel like sticking needles in your eyes again — can you see anywhere in the Hansen programs where any QC is done? Hansen et al stated:
I took a quick look at Step 0 and Step 1 and didn’t see anything that could be construed as QC.
As a UK resident I know we just had a cold October – the official UK Met Office site has data for the month that shows the month was -0.7C below the 1961-90 average! Yet the HadCRU map (and this is produced by a research ‘arm’ of the UK Met Office apparently shows a positive temperature anomaly of about +0.5C. Seems very odd.
RE 55. I’ll redownload and have a look. Some of the QA/QC is done offline, some is manual by their own admissio. I’ll run through the code later tonight after dinner
Re: #57
Chris M
Several UK stations had the same Oct=Sep error as those in Siberia – though with less dramatic effect obviously.
RE55.
Most of the files were relatively easy to check, kinda like a bad dream. I’ll double check
it all again, but I couldnt find anything like outlier analysis for example. It’s clear
there are checks for “bad” data, but its data that has been labelled somewhere else as
“bad” No in situ QA. The approach is a file is prepped. Fed to GISSTemp. Gisstemp
checks for obvious blunders in file length, missing data, etc. So there Appear to be no checks within the GISStemp for bad data being ingested.
Most of the QA performed in GISS per their documentation happens offline. Like dropping stations,
dropping periods for stations. Steps that have never been fully documented.
A few more notes:
Step 1 has changed from my last download. new python code written May31 2008. There are a few
adjustments to records ( as mentioned in Hansen 2001, and calculation of stats functions.)
and undocumented routines like “dropstrange.py . I’m currently seeing nothing I would call QA, except rudimentary checks.. ( missing data, etc). I found this though..
C?*** Special title for Jim’s plots..
I’ll go over it again, but its appears that that SOME OTHER program or process was used to identify problem data.( like St Helena) and then input files were altered ( like start date end date) so that GISSTemp assume that the data AS INGESTED is good. Remember there are steps described in Hansen 2001 that are not captured by the source code published.
More later I want to go over the Python again since its new code since I looked at it last. I might not get to that until Thursday.
It sure would be nice if they used version control.
Can someone enlighten me on how the HADCRUT yearly anomaly figures are calculated. The most logical thing would seem to be to just take a simple average of monthly figures by adding the monthly anomalies together and dividing the result by 12. I started looking at the years that are similar to 2008 and found out that (apart from 1995) there are significant differences between the HADCRUT yearly figure and the average of monthly anomalies. For example (the first figure being the HADCRUT yearly and the second the average of months):
1990 – 0.248, 0.254
1995 – 0.276, 0.276
1997 – 0.355, 0.350
1999 – 0.262, 0.296
2000 – 0.238, 0.270
2008 – 0.304, 0.315
The largest difference is somehow in the years 1999 and 2000. I wonder how would the higher anomalies for these years influence the trend line? Would there be a decline since 1997 already? Strange anyhow…
I checked the trend lines. With official HadCrut3 annual data the trend lines starting with 97, 98, 99 and 00 are all going up and only with a starting point being 2001 and onwards, they are going down. But if I use simple averages of monthly data for all the years between 1997 and 2008, then the trend line with a starting point of 1997 is almost flat, a 1998 starting point shows downward trend and only using 1999 and 2000 as starting point do I get a clear upward trend…
4 Trackbacks
[…] #7 by “Ed” on Climate Audit on the source story of Long View #2 […]
[…] normal. But when expert readers of the two leading warming-sceptic blogs, Watts Up With That and Climate Audit, began detailed analysis of the GISS data they made an astonishing discovery. The reason for the […]
[…] error was so glaring that when it was reported by the US meteorologist Anthony Watts and Steve McIntyre…GISS began hastily revising its […]
[…] long before my own FOI requests for CRU station data, I discussed CRU calculations in more detail here, concluding with the observation that “if, like GISS, they are doing nothing other than […]