As is by now well-known, CRU lost or destroyed the “original” data that went into the construction of CRU station data. This doesn’t mean that analysis is totally compromised (though it is made more difficult.)
Let me explain this through a comparison to GISS methodology. A given station may have a number of (what I’ve called) scribal versions. The GISS “dset1” station record is a combination of scribal versions using Hansen’s reference station method – a method that has received virtually no close examination from the “community” though it has some important defects. (One such defect is its contribution to the “Great Dying of Thermometers”, which results in part from the interaction of Hansen’s reference station method with the changeover from World Weather Records provenance to CLIMAT provenance.)
The CRU station records are more or less conceptually equivalent to the GISS dset1 station record. Like GISS dset1, they are combinations of underlying scribal versions, though CRU’s method of combining versions is not the same as GISS. (It is much simpler and probably more sensible if combined with someone actually looking at the scribal versions.)
While CRU has not retained the original scribal versions, GHCN has. These undoubtedly connect to the “ur”-versions of CRU station data though CRU seems to have used data not archived at GHCN (not just some stations not at GHCN but occasionally data for GHCN stations not archived at GHCN.) Many stations at CRU survive the Great Dying of Thermometers at GISS. In some cases, this is due to not using Hansen’s reference station method (a thermometer “killer”), but, in other cases, it seems to be due to CRU using data for GHCN stations that has not been acquired by GHCN.
Merely from the point of view of craftsmanship, I think that it’s instructive to compare CRU and GISS dset1 versions for individual stations and have done so below for a couple of the first stations that I looked at – neither of which have entered into prior discussion.
Mersing, Malysia
The top panel is the CRU version; the middle is the GISS dset1 version; the bottom is GISS minus CRU. Values are identical from January 1961 to December 1965, but are erratically different from January 1966 to December 1970.
GHCN/GISS has values from Jan 1971 to Dec 1980 (while CRU is blank for this period), while CRU has values from Jan 1981 on, while GISS dset1 has none other than a few isolated values in 1991 – which exactly match CRU.
There are 4 scribal versions at GISS for this series which respectively cover the periods 1961-1991(but actually 1961-1970 with a few values in 1991) ; 1971-1980; 1966-1975; 1991-2007+. (I collated a version in 2007 and am using this collation – presumably updated since then to 2011.)
The overlap between the version 1991-on and the earlier versions is too short to qualify under the GISS formula and thus this thermometer “dies” at GISS. CRU’s values from 1991 on correspond to the GISS dset0 data that GISS didn’t use. It’s not that CRU had data from 1991 on that was unavailable to GISS; it’s just that Hansen’s methodology rejected the combination of the data – a discarding of data that surely merits close examination. Does this “matter”? Perhaps not, but it should offend anyone with any sense of craftsmanship.
The other interesting question here is the provenance of CRU’s data in the 1980s. Did CRU get this from WWR, getting something that was missed by GHCN? I don’t know, but this is the sort of thing that should be documented before the parties forget.
London, Ontario
Next is a similar comparison from London, Ontario, near Toronto. In this case, there is only one GISS dset0 record, but it has no values from Feb 1932 to summer 1940 and ends in mid-1991, though London, Ontario obviously collected records subsequent to this. This particular thermometer seems to have died at GHCN rather than as a result of Hansen’s reference method.
As in the Mersing case, CRU records in a period that GHCN/GISS are missing. Why? Dunno. Unlike Mersing, there is a step difference between the two versions, with the step difference being the same order of magnitude as the trend being sought.
Parsing of other station records will show other differences. Do any of these things “matter”? If the only question is whether the 20th century is warmer than the 19th century, then no. But some people are interested in the relative timing of the warming between the first half of the 20th century and the last half of the 20th century, with Thompson et al 2008 occasioning a substantial revision to the SST record in the late 1940s, 1950s and 1960s.
A couple of days ago, Trevor Davies of East Anglia said that the purpose of releasing station data was to “pull the rug out” from under skeptics. As I mentioned a couple of days ago, it seems to me that the proper reason for archiving station data is to ensure that the most accurate possible information is available on an important statistic.
But most of all, the global temperature index has become an important record. If this were a Consumer Price Index, the statistical agency would carefully examine the nuts and bolts of the information. I don’t get the impression that either Phil Jones or Jim Hansen are much interested in actually working on the details of the temperature indices that have brought attention to their respective institutions.
Climate Audit 
98 Comments
Steve,
Nice post. Already some interesting findings!
I have been thinking about this for a while. There is not much interest in the average absolute temperature of the globe. The main interest is the temperature INCREASE over any period. So why not use a differential approach?
Instead of calculating the average global temperature for every year, why not calculate the average global temperature INCREASE for every year? I.e. for every station, calculate the increase from previous year (if possible). And then calculate the average of that for all stations across the globe. The average would only include stations with complete data for the current and previous year.
When you now the global average temperature increase for every year it is easy to calculate the temperature increase of any time period merely by summing up the years.
This way there would be no need of infilling of data. Any changes in the average after a gap such as in the London A data would have no influence on the result. And the output would be calculated only from data, no guesses or interpolations. Also, (almost) all data could be used, no records would have to be thrown away. The only station data not used would be the one for years which are incomplete.
If there is a station move of something else that causes a change in the average temperature all you have to do is to split the data and treat it as two different stations.
Does this idea have any merit?
Martin A —
What you describe is known as the “First Difference Method.” Details are off topic here, but see
https://climateaudit.org/2010/08/19/the-first-difference-method/
Briefly, Jeff Id convinced me that it creates more problems that it solves. RomanM’s “Plan B” is a lot better.
Re: Hu McCulloch (Aug 1 14:33), I wonder why I missed your post on that. Very interesting. I will read it through. Thanks.
It is still an interesting suggestion, and since I had just posted the temperature data for Virginia it was relatively easy to generate a plot of the type you suggested. So if you want to see what it looks like, relative to the other plots of temperature for the state, it is the bottom graph at the end of the post.
‘
‘
So does anybody know what Jones actually does for a living? If he isn’t the world expert on this stuff and all its ramifications why are my taxes paying for his salary?
I’m beginning to suspect that academia in general and climatology in particular may not be full of the world’s finest minds and the most diligent and effective researchers after all :-).
My memory might be unreliable, but hasn’t Steve McIntyre previously postulated that release of data might reveal how little Prof Jones was actually doing in terms of ‘added value’?
Wernstedt’s compilation shows 116 stations for Malaysia with Mersing have 42 years of records.
Re: Jeff Alberts (Aug 1 14:27), Then, why are you reading this blog? It is all about the average temperature, calculated from proxies or thermometers. Is everything Steve has done meaningless?
Average temperature is well defined as a mathematical entity and it gives SOME information about the system for which we are calculating it. At the very least you can say that the minimum temperature is lower, and the maximum temperature higher than the average temperature (not much, but not meaningless).
Well, the comment you’re responding to got snipped, so this probably will too.
Steve: Chris Essex’ argument about a global average temperature is not covered at this blog. It’s a very subtle point that he and Ross were making. Please discuss elsewhere.
While following bender’s instructions to ‘read the blog’ I found the Oct 05 discussion on the subject useful here.
In the rarefied world of big name scientists, people like Harry or grad students write the code and input the data, not the big cheese. Such details are uninteresting. Doing one’s own analyses from scratch as Steve does certainly gives a better understanding of the data, but is far from universal. There certainly seems to have been almost no interest anywhere in opening up the box and seeing what is going on with all the different scribal versions, CRU vs GHCN vs GISS, weird outliers, where the dead thermometers went, etc.
Craig:
True, but it therefore leads some of us curious folk to see what lies in that box, and while I am not quite yet done, I have been comparing some of the GISS, USHCN homogenized and TOBS data for the different states over the past few months. I started with Missouri and have, as I noted above, just reached Virginia. Turns out there are some quite interesting things going on within all that data.
Dave
not sure if this helps but Chefio has had a good dig into GISS over the years, you may want to check out his findings.
http://chiefio.wordpress.com/category/agw-and-gistemp-issues/agw-gistemp-specific/
regards
dougieh
Hi Craig
Would it be terribly impolite for a layman to inquire exactly what the ‘big name’ scientists actually do all day if the grunt work of actually understanding and processing the data is beneath or beyond them?
For example, Trevor Davies of UEA has published over 270 academic papers in his career, as well as fulfilling his other duties as Pro-Vice-Chancellor for Research, Enterprise and Engagement…and prior to that as Director of the Climatic Research Unit. Surely he, of all people, should be the world expert on all this stuff. The absolute walking talking exemplar of ‘Trust Me – I’m a Climate Scientist’.
But now you suggest to my horrified mind that it may be actually the unfortunate Harry (of Read me fame) who is faced with the task of making some sense of the hodge podge of data collected by CRU. And that Trev doesn’t actually get too involved in the nuts and bolts?? But just sticks his name on a passing paper researched and written by others.
Say it ain’t so Craig, say it ain’t so! Because otherwise the scales of my admiration for Professors and other senior academics will fall from my eyes with a resounding thud. Surely it would not be possible for such people of irreproachable credentials to be taking credit for other people’s work?
/sarc
Re: Latimer Alder (Aug 2 01:29),
I am not sure how much of this is supposed to be ‘sarc’, but since you have raised this question twice already on this thread:
Typically an academic may have say 4 PhD students who they meet at least once a week. The student does the more handle-turning aspects of the work (as Craig says) but the interpretation, setting in context and directing of the work is done by the supervisor. When it comes to writing up the work for publication, the supervisor will often take the lead unless the student is particularly good at writing.
During term time there is also undergraduate teaching, tutorials, preparation of course materials, setting and marking coursework and examinations.
As they become more senior they are required to do more administration and management, which is often something for which they lack natural talent.
Other activities include reading the literature, reviewing papers, reviewing grant applications, acting as external examiner for other universities, attending conferences, or even responding to comments about academia on blogs 🙂
There is a slightly worrying tendency among some sceptics to extrapolate from the appalling behaviour of Jones and Davies to the whole of academia.
Hi Paul
Thanks for the input. Let’s assume for the moment that the appalling behaviour is limited only to senior climatologists, and other academics are squeaky clean.
And I don’t doubt that they are busy bunnies with lots of calls on their time. Having spent a lot of time as a good tekkie myself and then moving into tekkie management I recognise the increasing distance that perforce comes between you and the underlying problems as you move up the greasy pole. And the higher you get the greater the knowledge gap
But that really means that the guys who truly understand the data and its ramifications aren’t the middle or senior tekkie managers like Jones or Davies or Hansen or Mann. It is their mostly unheard junior staff – like the unfortunate Harry of Harry Read Me who really know how clunky it is and how uncertain the results are. Or others who have the time and space to get stuck into the work ‘just for fun’…Steve McIntyre, Willis Eschenbach, Douglas J Keenan all spring to mind as being free to do that. There are many others no doubt.
So it seems that those who really have the right to say ‘Trust Me, I’m A Climate Scientist’ are the latter group, while the Jones’s and Davies’s of the world should stand back and be quiet as they act as little more than technical admin managers. And in both their cases, evidence from Harry is that they ain’t much good at that either.
Before Physics grad school, I was among others in saying that it was unfair that the Professor got all the credit for the work of the grad students. After grad school, my tune certainly changed. Although there may be instances where there is a brilliant grad student that comes up with original research with little to no guiding, I believe they would be few and far between. Yes, I did almost 100% of the experiments, data analysis, computer modeling, etc… for my research. However, I was in significant contact with my professor who helped guide me to the proper way of analysis, new ways to look at things, what to look at, etc… My professor probably spent the majority of his time with classes, politicking, and applying for grants, but my research (especially during my first 3 years or so) would have been almost non-existent without him.
I do have serious concerns with the way these Climate Scientists have analyzed the data. However, there is little doubt in my mind that they are the driving force in the analysis done by their grad students and postdocs. The belief and skills of the professor are typically passed on to their grad students…
My major prof was pretty clever with math but never touched a computer–he had a tech and a bunch of Ph.D. students (like me) and post-docs who did that. I have seen a lot of that. I have no idea about Jones, Davies, or Hansen, but it does appear that poor Harry was rather out in deep water by himself.
I generally gathered from reading fairly well into Read_Me_Harry that Harry was trying like the dickens to match the results of what someone else had done, and that his exasperation was due to his confusion about where the data was that he was supposed to be using.
That is probably oversimplifying it, but that was my general understanding.
This could explain the defense of Mann 08 in PNAS, that Mike Mann doesn’t understand what his own code is doing.
Dougieh:
Yes it was his list of the GISS stations that are currently being used for the US that was one of the things that got me started. What I have found interesting (and it is something that he has commented on) is the number of these stations that start in 1948, rather than going back to 1895.
Couldn’t the missing temperature data for a place like London Ontario be pulled from the local newspaper archives or somewhere like that?
Steve: the data can be obtained from Environment Canada. One would think that the institutions compiling the data would do so just out of craftsmanship, regardless of whether it “matters” to the aggregate. Or if they don’t want to do things like that, an institution that is in the business of data collecting should take over the CRU temperature data.
In 1989, 487 Canada stations have records in GHCN. Previous years were slightly higher. Canada is a big country, but that is a lot. In 1990 there were 270, in 1991 71, in 1992 69. This is in the years leading up to the CLIMAT reporting system. I’ve said more about it here.
There were good reasons for cutting back, but they may have gone too far. According to Ogimet, 110 Canadian stations submitted CLIMAT forms in March 2011, which seems about the right number. As a separate matter, there seems to have been some issues in recent years with EC sometimes submitting incomplete forms.
Nick Stokes,
In your August 2010 blog you note “This makes the pattern clear. Australia, Canada, China, S Africa, Turkey and USA had been overrepresented, and were culled in specific events. Canada in 1989/90, Turkey and China 1990, S Africa 1991, Australia 1992 and the USA in 2004 and 2006.”
Personally, I feel that over-representation is preferential. Do you know why Australia reduced so much in 1992, apart from changing the name of the data set and rounding out to about 100 stations? Was there something wrong with the pre-1992 data, or had it simply not been verified adequately at that time? Or was it because of problems in matching earlier Hg thermometer records with later thermistor/thermocouple records?
Do you also know why Antartic stations were reduced, when Antarctica must be among the most under-represented, important global land masses?
You have better sources than I do, so I hope you do not mind me asking.
Geoff,
As I said below, it’s not really a matter of active reduction – it’s a matter of who signed on to the CLIMAT system and how much. I don’t know whether the WMO had to beat back volunteers or twist arms, but I suspect more the latter.
You might like to try the KML files that I have been promoting in recent posts. You can set them to show in Google Earth what stations were reporting pre-1990, post 2008 or whatever. If you look at pre-1990, the regional disparities are spectacular. Turkey is bristling wuth stations. S Korea and Japan likewise. Australia, US and Canada, despite their size, are also very dense in large parts. And of course Africa, S America etc, very sparse.
AS to Antarctica, I don’t know. It’s all tied up with international exploration and the associated politics. But there may well have been a shift to satllite orientation.
the pre-1990 density is because of a series of collection projects undertaken at the time. This has been discussed at CA in the past. GHCN undertook to update things at “irregular” intervals and have not done so. This is what happened – not your imaginary story about a “culling” decision.
Geoff, GHCN did a special assembly of Australian historical station data circa 1990, together with similar exercises in Russia and China – the other two legs of Jones et al 1990. As I keep saying, GHCN did not make a conscious decision to “cull” Australian data – for the most part, it simply hasn’t bothered updating any stations other than what it gets from CLIMAT. If they decided to cull stations, they would have culled USHCN.
It is quite evident to me that the decline in GHCN station population is primarily a result of bureaucratic inertia and perhaps laziness, rather than any rational process of station selection and culling.
Nick has no information on the actual decision-making and is merely confusing people.
Exactly Steve. And worse yet, just because a station reports in the CLIMATs doesn’t even guarantee that it gets updated in GHCN.
Back to the topic at hand – for London, Ontario, it looks like the GHCN data do come from means computed from Tmax and Tmin. I’m speculating on the CRU data, but the documentation says CHTD for the source – possibly ‘Canadian Hourly Temperature Data’? The CRU values are close enough to GHCN that it looks like the same station, but maybe just a different Tmean method? If so, the step changes in GISS minus CRU could be changes in the method over time.
Nick, do you know that there was a specific policy adopted at the time to “cull” Australia, Canada, China, S Africa, Turkey and USA? Is there a policy statement on the topic? Do you have personal knowledge that there was such a decision? What were the principles of the culling decision? Who did it?
Or are you just making this up as a pseudo-explanation?
You say that the US was ‘special”, but why was it “special”? It doesn’t make any sense to “cull” Canada without culling US.
Steve,
“Is there a policy statement on the topic?”
GHCN was actually, starting in 1992, an archiving process. AFAIK there were no strong guidelines on what could be included – just about everything they could get. When it became an ongoing project around 1997, they then had the CLIMAT reporting system. So “culling” means being archived pre-1992, but not going on to submit CLIMAT forms.
As to which stations did get to submit CLIMAT forms, I don’t know how that was decided, but likely the WMO coordinated it. They may have had to push a bit, as there is some commitment involved for the submitting countries.
The US is special, because of USHCN. There are very high station numbers (about 1500) in GHCN right through those years, and they were not cut back until 2004 and 2006. But because information of similar quality is being collected and is available through USHCN, whether it is formally included in GHCN is not so significant.
OK, I understand. You have no specific information on the history and simply fabricated a story about a “culling” decision. I wish you wouldn’t fabricate stories when you have no knowledge of what they did. It’s all right not to know something.
You’re also making stuff up about USHCN. You don’t “know” that USHCN was culled in 2006 as opposed to the updates not being incorporated yet. Again please don’t make up stories if you don’t have information on the topic.
In non-academic fields, people who don’t know something are urged to say so, rather than making up “plausible” stories as you have been doing here.
No, Steve, the fabrication is yours – you introduced the word “culling” in this thread. I simply noted the numbers.
And I didn’t say that USHCN was “culled” in 2006. In fact, that doesn’t make sense to me. I said the US station numbers in GHCN were cut back, as they clearly were.
Mr Stokes,
It wasn’t Steve McIntyre who introduced the word “culling” in this thread; it was Geoff Sherrington when quoting your own blog:
“In your August 2010 blog you note “This makes the pattern clear. Australia, Canada, China, S Africa, Turkey and USA had been overrepresented, and were culled in specific events.”
In the link referred to by you, and then cited by Geoff Sherrington, I remind you that you make this remark:
“It clearly consisted of a few major culling events. In the next table, the years with more than 100 stations ending are shown with a breakdown by country”
Had it not been for your statement, the term “culled” wouldn’t be the subject of any discussion at all.
So *you* made the assertion – and remember it “…clearly consisted…” so very little equivocation there, *you* referred readers of this blog back to that assertion, Geoff reminded you of the assertion that *you* made, quoting the exact phrase, and Steve referred to that.
But, naturally, you’re blameless.
Nick, you get more and more outrageous. The following retort is a fantasy in your own mind.
As another reader noted, the word “culling” was used in your blog (to which you yourself linked here) to describe the events. Reader Geoff Sherrington quoted from your blog where you used the word “culled” in respect both of Canada in the early 1990s and US in 2006:
You used the word. Own up to it.
Nor is there any evidence of a culling decision on rational grounds. It is evident that you just invented this “plausible” story, but have no evidence for a culling decision actually taking place.
At one time, I examined a number of GHCN documents and my own view is that there was no “culling” decision. Quite the opposite. Around 1990, GHCN carried out some special projects to collect NMS data e.g. China, Russia,… which they indicated at the time would be updated “irregularly”. The contemplated “irregular” updates don’t appear to have taken place. Nor do I believe that the USHCN data have in fact been “culled” as opposed to not being updated yet.
It seems to me that you just made up a story to “explain” the change in numbers bit have no historical knowledge or evidence for the story other than the change in numbers themselves and just guessed (i.e. made up an explanation.) In non-academic occupations, people take care to differentiate between what is known and what is merely guessed – a distinction that you seem to pay little heed to. It seems that far too often climate scientists confuse a “plausible” story for facts. Your fabrication of supposed ‘culling” decisions is merely another example.
Steve McIntyre (Aug 2 06:23):
That opens up a lot. Climate science seems to prefer areas where the best available theory is only plausible – for it is there that the appeal to authority is strongest. Hence the disdain for what satellites may be telling us about climate sensitivity. And the way they take like ducks to water to PR stories after Climategate that promote the plausible rather than the facts. In the latter there’s another word for it. But for lying to be effective it must of course be plausible.
Steve,
It would help if you would document better the sources of these claims of fabrication and fantasy. As here, my blog was simply a study of the numbers. I make no claims about anyone applying any particular culling policy.
It’s true that I used the word cull there, in the context of describing the appearance of the numbers. And someone pulled me up on it. Whereupon, I clarified:
‘Although “culling” isn’t the right word (and it’s a notion I’ve been deprecating anyway) I think the observations on country specifics stand’.
So let’s get facts straight
You did use the word ” culling ” in your blog
You linked the blog here in your post
Then you denied using the word culling here.
Then you justify your -snip – by talking about a response you made to a commenter when you said maybe culling was not the right word to use. But that does not make it true that you did not use the word.
-snip
Nick,
You hold some bizarre opinions. I am shocked you think 487 stations is a lot for Canada, a country larger than the US geographically. And you think 110 stations is about right. Where do you get this? I have never known a scientist to say they want less data. I have seen them ask for better data. I’ve seen them cull out instruments giving bad measurements. But I have never seen scientists ask for a reduction in the number of instruments and measurements without quality of data being a prime concern. But in climate science everything seems to be upside down. They don’t care if they are culling good stations, the goal was fewer stations – regardless of the quality of the stations. It is nonsensical.
Ron, there are currently about 1800 stations reporting world wide. Canada is about 3% of land area. 110 stations is about double the average density. But since some areas are clearly under-measured, that’s why I think it is about right.
Sure, more info is better. But there’s no point in uneven coverage. And getting those 1800 reports in every month is a big challenge. There are a lot of laggards. Indeed getting the Canadian info in has been patchy.
This is not a matter of climate science being upside down. They don’t run the system.
‘Getting 1800 reports in every month’ – let’s see, if you’re shipping that across a wire or an RF link that’s barely a few hundred bytes a day per station even once you add in protocol overhead, less total in a month than most people’s morning dose of email spam.
If this data were really considered important enough that its application means life or death for national economies, then you’d think it’d make sense to bring to bear a fraction of the technical capacity needed, say, to throw ads in your face when you play freeware ‘Angry Birds’.
It isn’t a matter of wire capacity – it’s a matter of getting the MO’s to send them in punctually and correctly. And they don’t.
“If this data were really considered important enough… “
Too much passive voice there – yes if you think more stations should be momitored, that’s a matter of resources – let your congressman know.
If it is difficult for the supposed greatest problem of this days, then maybe the problem is not so great, as it appears that is not so great need for this data.
I work on IT at a telecom, and the data on climate really seems tiny compared to the GB that we process everyday…
Nick,
In an email response to a question about the sudden dwindle in number of stations, Dr. Ruedy told me that it was a funding problem. They had had more in the early 90s and could handle the greater workload entailed. I’m paraphrasing my memory of his note and may not have the emphasis correct, but it is my sense of the thing.
in retrospect, people can always blame “funding”. However, it seems to me that CRU and perhaps GISS have treated the temperature statistics as a profit center and in effect used the funds not on temperature data, but to fund things that they’d rather do. CRU doesn’t seem to have done much quality control. It has a few simple programs to collate data and make a gridded average. Gavin Schmidt said that GISS spent only 0.25 man-years on quality control.
My own impression is that the collection failures have arisen out of laziness and inertia rather than “funding”. For example, has anyone at GHCN written Canada and asked them to update their historical data at GHCN? I suspect not.
Steve,
Doesn’t it look like they don’t think they need all the stations they used to have? If they limit their uptake to the ones that show up by themselves and which don’t need a lot of QC, then they can keep their QC burden as low as Gavin says and then, as you say, spend the money on something more interesting.
I don’t know if their staffing level over the years is accessible, but it would be interesting to see if they had more clerical-types, (grad students?) in the late 80s early 90s.
It is fair to call attention to the different skills that researchers and engineers bring to the table, to point out where an engineering mindset might lead to improvement, and to call for a move in the the day to day management of climate products away from researchers to specialists.
It is unfair to label CRU students, staff and faculty as lazy.
Dont forget the mail about using the controversy to get funding for a grad student to work on the next version..
More Canadian stations wouldn’t lead to over-representation because the data is geographically gridded.
On average, the Canadian stations with recent data have an average altitude of 240 while the stations that no longer report have an altitude of 360.
Would this not lead to a warmer bias?
Don’t know. Probably isn’t that clear cut.
Where I live, you have Whidbey NAS, which is a GISS station I believe, and is probably at less than 20 feet above sea level. My house is only a few miles away, at about 220 feet ASL, and is almost always 5-10 degrees warmer in the summer. For example, today at the NAS at 5:30pm it was 66f, at my house in the shade, it was 72f.
No. You use the anomaly.
Isn’t this the issue that this posting ahs been arguing about.
Is there any significance to the loss of stations? Is there a bias (geographic or otherwise) induced with the loss of stations or, as Nick Stokes appears to be arguing, the loss of statiosn makes no differrence to the calculations becuase of the extensive geographic sampling.
There are other issues that I see being brought up about the rigor of climate research etc. but the specifc issue that I see in the posting is the possible introduction of bias by the loss of stations.
So one concrete question to be asked is:
Is any bias being introdced by the loss of stations? And if so, how and of what quantative significance?
It’s a good question. I’m not arguing that more stations would be of zero benefit – just that they have opted for a reasonably uniform distribution, which is the best way to use the resources that they presumably have, and seems to work well. My latest blog post is on trying to quantify the question of coverage, via the presence (or absence) of stations in the gridcells that they use. It shows that there was a dip at about the time the station numbers in GHCN were declining, but it wasn’t nearly as pronounced as the dip in numbers themselves, and there has been some recovery.
I interpreted the issue as being about what CRU did with the original data to get to the “CRU” version – e.g. London – why the differences from GHCN, and why don’t the institutions care that there are these differences? It was Nick that hijacked the post and turned into a loss-of-stations issue.
The latest issue of the “American Scientist” has an article (“Quasirandom Ramblings pp 282-287 American Scientist July-August 2011) on the surprising effectiveness of Monte Carlo techniques. Very high dimensional problems can be addressed by these techniques with a degree of effectiveness that is not yet understood. According to the article, the effectiveness seems to be related to the effectiveness of various forms of random sampling on these high dimensional spaces.
To understand the effectiveness of sampling a measure of how well the sampling points have been distributed has been defined. This is called “discrepancy”. For two dimensional problems, It is calculated by determining the maximum difference between the number of selected sampling points in any possible rectangle in the domain with the number of pints that would be expected if the distribution were truly random.
Since using available stations is a form of sampling and the gridded average technique is a form of Monte Carlo estimation, perhaps “discrepancy” could be used to determine if the available station sampling technique is sufficient.
The American Scienstist articel on Monte Carlo sampling is at
Click to access 2011631353538566-2011-07CompSciHayes.pdf
Thanks, Tom, that is an interesting article. The issue with station measures is that the density is not spatially uniform. I’ve been trying Monte Carlo thinking to try to inversely estimate the density parameter of that distribution. It’s what I need for least squares weighting.
It seems to me that the discussion in the article indicates that if one wishes to know the quality of an set of sampling points and the effect of culling sampling points on them, one must calculate the discrepancy of the resulting sets and the changes with culling. That is, one treats this as a Monte Carlo problem and uses the concept of discrepancy as described in teh American Scientist article noted above.
If temperature trend is a constant value across an entire region (say the US), then only one sapling point is required. However the trend will vary from place to place and be relatively constant over areas of related geography. Thus, ideally, the number of sampling points should be enough so that each geographic region is sampled and that the number of points in each region is proportional to the area of that region. This would minimize the discrepancy and as the article indicates would minimize the error in the estimation.
So the quality of any trend estimate (ignoring the issue of bias in individual sampling sites) can be estimated by calculating the discrepancy and teh effect of any culling could be estimated by the change in discrepancy.
What I see from this is that adding more sampling stations could increase the discrepancy (by over sampling an area) and make the estimate worse. Culling could also in certain cases improve an estimate by reducing discrepancy.
The use of gridded estimates would seem then to be useful in that it is a method of minimizing discrepancy in the Monte Carlo estimate.
It looks to me that the culling issue is not important with the suse of gridded averages. The main point for the use of large numbers of stations is the ability to remove random error from the gridded averages. Since the errors of concern in the issue are not random but systemic (UHI etc), this approach is not of use. A close analysis of indvidual stations and the compostion technqiues used on them would seem to me to be the only viable methods.
agreed. in this case, one of the points of issue is whether the airport stations (which dominate in CLIMAT) are biased. Not in the sense that the entire increase is due to the bias, but is some portion of it? One way of examining this is to examine against non-CLIMAT stations that can be shown to be rural and non-airport.
GHCN’s failure to properly collect and archive non-CLIMAT stations leaves the indices open to criticism on this front. Nor does the collection of the data seem like an insuperable enterprise. I assume that the main reason is that they haven’t tried.
Nick,
Yours is a nonsensical answer. It bears no relationship to how many thermometers is actually needed to sample an area, just lots of subjective opinion about areas you feel are undersampled and extrapolations from there. Pure armwaving.
Steve has shown here on CA that undersampled regions show the most warming. Perhaps you will forgive me for thinking the fewer thermometers reporting, the easy it is for climate scientists to commit mischief with the records.
Many of these stations are still reporting. Someone at GHCN or somewhere is culling these stations and it is not being done on the basis of poor quality stations because plenty of poor quality stations are left.
I don’t think anyone is “culling” or deciding not to use particular stations but rather that the “system” (and I use quotes intentionally) for importing the data is casual–whatever shows up. There does not seem to be any awareness from descriptions I have seen that one can actively go get most of the data on line.
Here is a WMO report describing the current GCOS observing system, which is what GHCN uses (more specifically, the Global Surface Network GSN component). It is a selection of stations around the world – the document describes what is expected of them. It is quite informative. Some extracts:
“In practice, meteorological measurements are made at thousands of places all over the world, more or less regularly. The most essential subset of these observing stations is operating under the regime coordinated by the World Meteorological Organization (WMO), involving clear commitments regarding the site, the exposure of instruments, error handling, units of measurement, coding and exchange of reports. In practice, this WMO Global Observing System (GOS) is implemented by National Meteorological and Hydrological Services (NMHSs) of WMO Members1. The original prime purpose of the system was the provision of data in support of weather observation and forecasting, but it of course serves many other potential users particularly in this case climate and climate change research.
…
The GSN is intended to comprise the best possible set of land stations with a spacing of 2.5 to 5 degrees of latitude, thereby allowing coarse-mesh horizontal analyses for some basic parameters (primarily Temperature and Precipitation). The criteria for selection include:
• Commitments by NMHSs with regard to continuity;
• Geographical representativeness of observations;
• Length and quality of historical time series;
• Available parameters;
It is recognized that the coarse network density limits the applicability for some applications…
…
Generating a monthly CLIMAT report is a Minimum Requirement for stations. “
Nick,
I thought “more or less regularly” a disturbing metric. Do you suppose that is where the QC burden creeps into the picture? This is not intended to be sarcastic, but I keep thinking that if you are running a temperature data accumulator and you always get good data from 10% of the stations out there, maybe you quit fussing with the others.
I think “more or less regularly” is a reference to the totality of temperature measurements that are taken, by many different groups for different needs, and they go on to say why a more organised subset is needed.
Nick, again you are fabricating a story of GHCN deciding to “cut back” on the collection of non-CLIMAT records. As I’ve said repeatedly, GHCN undertook to “irregularly” update the historical (non-CLIMAT) data but have failed to do so. This might well be due to bureaucratic inertia rather than “good reasons”. Please do not confuse your ex-post rationalizations with facts.
Steve,
If you’re going to persist with these allegations, please quote what I actually said that is alleged to be a fabrication. It will save time.
To Nick Stokes, here are the recent cases of you simply making stuff up without being in possession of the details. There’s no mystery about the claims that I objected to as they were replied to or quoted.
or this – to my knowledge, USHCN was not “cut back”; it just isn’t updated yet. I don’t believe that you have any knowledge of the matter and are asserting something as though you have knowledge of the policy when you have none:
Or the following bald-faced disinformation. You used the term culling on your blog in a post to which you linked. You also used it about the US. And in this context, the ‘distinction” between culling and cut back is a distinction without a difference. The sort of tiresome wordsmithing that seems to characterize the “community”. And when challenged on it, the obvious thing to do was to apologize, instead you wheedled. Again, all too frequently, the mark of the modern climate “community”.
I’m normally pretty patient. And if this were an isolated incident, I’d ignore it.
Steve,
“There’s no mystery about the claims that I objected to”
It’s been quite hard to respond to this post, because it changes as I write. You originally said:
“Nick, again you are fabricating a story of GHCN deciding to “cut back” on the collection of non-CLIMAT records.”
and I was in the process of pointing out that I did not speak of a (bolded) GHCN decision when that disappeared. So there is a little mystery.
But you seem to be saying that I am making some inference about decision processes behind the numbers that I set out in detail. I am not, and you have not shown any. I use “cut back” in the sense of reduced. The reductions in numbers that I detailed did occur. I did not speculate on the underlying decision processes, or even whose they were.
The reference to USHCN is just one case of putting words into my mouth. I simply observed that the numbers of US stations in GHCN were cut back in 2004 and 2006. They were. And I noted that there was an alternative database, USHCN. I did not speculate on its future.
As to “culling”, you used the word without reference to my blog, or even to Geoff’s comment, which at that stage I hadn’t read. So it was a mystery to me, and I pointed out that you had introduced the word in this thread, not me. And OK, I was part wrong – Geoff was 6 minutes earlier. But I hadn’t used it. When I did track it back to the blog (a year ago), I was able to point out that any suggestion that I was describing a policy-based culling process was not intended, and that I had thoroughly corrected that at the time. That’s not “fabricating a story”.
OK I see that the bit about GHCN deciding hadn’t disappeared – it had just wandered off to somewhere else in my browser. So I’ll reinstate my response – I did not speak of GHCN deciding to cut back on non-CLIMAT records.
as we speak environment canada website is being scraped by a hacked out R program. about 7000 stations to go..
some very short records some with no temps only snow rain and other stuff.
But a researcher as noted to me that they dont do monthly updates with any kind of regularity.
arrg. connection timeout. This could take a while
Craig, There’s quite a lot of work going on in Australia that is relevant to this post, but much less than is needed. We can produce similar examples to Steve’s, I suspect, though I personally have not tried yet. We have been more concerned with adjustments that might have been made to Australian data before it went to CRU or NOAA.
It is notable that the work most visible here is done by volunteers, while official sources seem more likely to repeat tired old dogma. In general, the BOM seems to have sent monthly data overseas but I’m not sure of methods used to compile daily to monthly beforehand, e.g. treatment of missing daily values.
Has there been any significant new data allowing extension of the proxies in your comparison paper?
MartinA and others – please remember that a major use for historical thermometer data is to calibrate proxies. A subjective adjustment of a step change – as if Steve’s example of London Canada was levelled to ‘look better’ – could have a major effect on proxy calibration. (BTW, an academic question: how are proxies to be calibrated for the last 15 years when there has been very little temperature slope globally?)
It would of course be hilarious if Steve reverse engineered their “missing” records in short order, whereupon they’d steal it back without credit and while denying he’d done anything original they had to acknowledge at all.
It’s funny that you should bring up the consumer price index, Steve. Here in the U.S., as you may or may not know, it has been politically adjusted and rejiggered so many times that it has lost most of its meaning as a measure of a constant standard of living.
Now there are all these “hedonics” and “substitutions” involved in its calculation. For just one example, if the price of beef goes up, the BLS simply assumes that people have switched to chicken, and voila, the index need not actually reflect the rising price of beef at all. That’s substitution, and “hedonics” is when they “adjust” the price of something down, say, computers, to “compensate” for how much better and faster they are than they used to be.
Pretty wild stuff. There’s an economist who covers all this, his name is Walter J. “John” Williams and his site is shadowstats.com
I guess that what I’m saying is that maybe these temperature indexes or records or whatever they are, are politicized just like the CPI is. In fact, the powers that be in the administration and congress have recently laid the groundwork for yet more “adjustments” to the CPI in order that it, once again, show even less inflation than before so that COLA’s can be reduced.
The parallel motivations seem to be as follows:
With the CPI, make it show less and less inflation – this could be seen as benefitting the government by reducing entitlement outlays.
With the temp and proxy records, make them show more modern warming and get rid of the MWP – this could be seen as empowering the government to be bigger and do more to solve/mitigate the problem of AGW.
What I find interesting is that instead of an official government agency like the BLS, the climate team is relying on more of an ad hoc method, largely promoted by individual scientists under their own names and on supposedly non-official blogs like RC. Although I freely admit I don’t understand all the details, I think the rolling up/consolidating/eliminating some of the temp data and stations could be a step in the direction of a more centralized control of all things climate. In the end, I fear that no-one will be able to get any raw data at all, only the adjusted and corrected versions.
tomdesabla: You are referring to the Chained Consumer Price Index.
Steve has often said that climate statistics should be calculated and published by a government organization just like Cost of Living etc. Methods and data made public. Very public. Independent. Reliable. Rational. Pundits occasionally complain about the CPI or some other statistic and call for changes. However they never question how it is calculated.
Ha Ha. Title is one of your best. Good to have a chuckle while reading your post.
I eagerly await the publication of Watts et al analysis of the stations when we can finally learn if any of them merit the appellation Grand Cru or if we are stuck with boatloads of plonk. Methinks that the latter will be more appropriate.
Re “culling” and the cost of keeping up with the data. My question is, how do the various entities (GISS, Hadley, GHCN) keep their data up to date? Since the data are gridded, there is no sense in which “over represented” countries skew the world data. Are data manually sent and manually entered? Why on earth would anyone do that? Much of the data for US, Canada, even Russia is available online to be automatically grabbed–though probably needs some QA. Is any QA done on incoming? Who does it? If QA is expensive and data is entered manually, then yes, it might make sense to limit # stations (but better to actually fund it properly)–but even better to automate most of it. As I have mentioned before, the Wolfram Mathematica tool can access tons of world weather stations with data up to the current day or even hour. Custom tools could do the same. Have they heard of tools like this?
“Since the data are gridded, there is no sense in which “over represented” countries skew the world data.”
Thanks, Craig. This answers definitively the business Nick is putting forth about over representation and a decision to cut back.
Craig,
Due largely to pressure from people such as yourselves, the process of updating is formalised. Countries submit monthly CLIMAT forms – you can read them at Ogimet. GHCN will not alter the numbers on its own initiative – if they fail QA it bounces the forms back to the originating MO. This all takes time.
Yes, overrepresentation does not bias the result. It is just not an optimal use of resources.
Are the CLIMAT forms paper? Manually processed? Or automated?
I’ve done a quick comparison of differences in data sets for three sites. There are two main differences:
1. Different data sets use different data periods for the same station.
2. There is a tendency for the values of the GHCN and CRU data sets to show step increases for recent years.
I forget to tell you all where to find the analysis. It is at:
http://www.climatedata.info/extras/cru-compare.html
Am I to understand that nick is saying that it’s better to have less data than more data when trying to reconstruct an average temperature?
I wonder if Watts is going to plan to show pics of all these excellent stations which could not possibly be found next to air conditioning vents.
That was the essence of Nick’s original claim, although he didn’t state it that way — rather, that the Canadian cutback made sense due to some over representation, which Craig points out is nonsense.
He has now shifted his position to the idea that getting data from all 1800 stations is too hard.
There are some simple facts. Here is a map (2002 – the most recent one I could easily find) of stations for which CLIMAT forms were submitted. There were then 1785. What do you think determines that number?
I don’t know if MO’s are reluctant to make the effort, or if someone like WMO regulates the input. But it’s what GHCN has to work with.
Re: Eric Anderson (Aug 2 16:40),
Nick Stokes (Goldilocks?)
1. This data is tooooo big!
2. This data is tooooo hard!
3. This data is juuuuust right!
Yes, the Team seems ever vigilant to eliminate oversampling yet seems to not have a worry in the world about under sampling. This emphasis seems curiously misplaced. Its like a police force who spends all their time giving out tickets to people who are driving too slow and never gives tickets to people who drive too fast.
However, it seems to me that CRU and perhaps GISS have treated the temperature statistics as a profit center and in effect used the funds not on temperature data, but to fund things that they’d rather do.
Ouch that’s got to hurt. Need some iodine on that one.
Having worked on one longish term collaborative research project involving several university departments and other research bodies, and witnessed other such projects, it would come as no surprise if CRU did do that.
Anyone seen Anthony Watts’ work on this?……….
http://wattsupwiththat.com/2011/08/04/analysing-the-complete-hadcrut-yields-some-surprising-results/
Analysing the complete hadCRUT yields some surprising results
Posted on August 4, 2011 by Anthony Watts
From The Reference Frame, 30 July 2011 via the GWPF
HadCRUT3: 30% Of Stations Recorded A Cooling Trend In Their Whole History
The warming recorded by the HadCRUT3 data is not global. Despite the fact that the average station records 77 years of the temperature history, 30% of the stations still manage to end up with a cooling trend.
In a previous blog entry, I encouraged you to notice that HadCRUT3 has released the (nearly) raw data from their 5,000+ stations.
Lubos Motl did the grunt work on the reference frame
Yes he did!
Please, could someone free my response to Speed (aug 3 2:50am)from moderation purgatory?
I am resubmitting the comment without the link – maybe that will help:
For some reason, my browser – or something – is not letting me respond to individual posts. I seem to only be able to post at the end, so please forgive me if I respond to Speeds above post at 11:56 AM here. If anyone has any idea what is causing the problem, please let me know.
–
Uh, Speed, let me clarify. No, I was not referring to the chained CPI specifically, and I don’t know why you would assume that. According to Williams, who is a recognized authority on the subject, and also according to my wife the financial economist who works at Treasury, there are 3 main CPI measures currently published by the BLS – the CPI-U, the CPI W, and the C CPI U (chained CPI).
There are NO CPI measurements published by the BLS that measure a constant standard of living, as the index was intended to do. If the methodological changes made since 1980 are backed out of the calculations, the CPI would be reading 7 percent higher. That’s seven (7) percent, not .7 percent.
The entire point of my post, which I guess you missed, was that the CPI is not what most people think it is. This quote from you regarding the CPI is a great example, and is IMHO totally incorrect in every respect:
“Methods and data made public. Very public.
Ha. Not one person in a thousand understands how a single one of the CPI measures are calculated. The concepts of subsitution and hedonic adjustment are completely unknown to the average American, so how can the BLS’s methods be “very public”? Publicly available maybe, if you know what you’re looking for and where to look, but not publicly known.
“Independent”
No. If you think a measure that is at the mercy of the whims of politicians is independent, well, what is it independent of? It’s been changed many times already, and is being targeted for changes again, and every single change since 1980 has always been to show lower inflation, which benefits government. That may be “independent” in your mind, but it certainly isn’t in mine.
“Reliable”
If it were so reliable, then why has it “needed” to be changed so many times? If it were so reliable, then why does it show inflation that is so much lower than common experience? How many things do you know of that have only gone up in price by 3% or so in a year? Very few, yet that’s roughly what every CPI measure indicates.
“Rational”
How rational is it to take a metric, which is supposed measure a fixed basket of goods, and then make so many adjustments to it that it now indicates an inflation number that is 7 percent lower than it indicated 30 years ago. It has become a dishonest metric misleading the people, and I don’t think that misleading people about something so important is rational.
“Pundits occasionally complain about the CPI or some other statistic and call for changes. However they never question how it is calculated.”
No. Besides shadowstats, which I noted above, check out Chris Martenson’s opinion – he wrote an article sometime ago entitled “Inflation is so much worse than we are told” He is pretty well respected, so check that out.
The CPI is a complete laughingstock outside of the MSM. What do you think the changes are that pundits call for anyway? Changes to the way the index is calculated of course. Just like all the changes that have happened already. In any case, yeah, there are many people complaining about the CPI being useless because it understates inflation, but none of those people has the power to change the way the index is calculated. The people who do have the power, are trying to do the same thing to the CPI as has been repeatedly done in the past, make it show less inflation than there obviously is.
Bottom line, using the CPI as some sort of model that climate science should follow is way misguided and naive, because the CPI is not some transparent, reliable independent metric like some people might believe or wish it to be.
Sorry to burst anyone’s bubble.
2 Trackbacks
[…] Politico, 8 August 2011Green Beatle Gas Guzzler Henry Payne, The Michigan View, 7 August 2011A Couple of CRU Stations Steve McIntyre, Climate Audit, 1 August 2011Senate Testimony: The Question of Green Jobs Kenneth […]
[…] Source: https://climateaudit.org/2011/08/01/a-couple-of-cru-stations/ […]