Since my initial analysis didn’t show a robust HIE, I looked bias in the dset1 data.

Details here:

http://sextant.blogspot.com/2010/01/seeking-heat-island-effect.html

Using groups for urban/suburban/rural, altitude band, latitude band, station id, and year in the R Project lmer function applied to the giss.dset0 annual temperature data. It nicely shows the effects of each of the above factors. The details and graphs are here. From raw data to unbiased results in one R command! No doubt there are better ways do this, but these results are not bad. Changing the model formula around and using various different groups, didn’t seem to make much difference to the random effects calculated for time. I’ve done the same for giss.dset1 data and got much the same results.

A post that begins with “As Steve McIntyre has so brilliantly shown…” can’t be all bad.

Steve: Nice analysis. I’ll take a closer look at your methodology sometime soon. The term that I’ve frequently used for Team statistical methods is “artisanal” – the term doesn’t mean that the methods are “wrong”. The purpose of trying to place artisanal methods in a statistical context is to simplify analysis of things like UHI by having an algorithm that can be understood. The battleground issue in station data is UHI. My objection to Hansen’s methods is that the purported UHI adjustment isn’t one. Simply doing a mixed-effects style average of dset0 station data doesn’t do a UHI adjustment either. But the algorithm style looks like a very nice enhancement of what I did in the old post. Nicely done.

This post may sound a bit ungracious, and possibly even egotistical, but I have a real gripe with the new list of the Top 100 science blogs on Wikio at the moment. Others are very happy with it – Greg Laden celebrated getting to number 9 (which curious…

How is an elephant like a gazelle like a lion like a human? Yes, they all have four limbs, two eyes, and a mouth.

Gee whiz.

In this case, picture it as the following model:

where i is station, m is month, t is time
is the mean for month m at station i
is the anomaly for time t

Mixed effects is a standard and effective way of estimating models like that. Now your inclination is to probably say, well, duhhh, so what. What’s so hard about that? And you’d be right. But this simple point seems to have eluded a lot of people so far.

1) The reticence to allow outside questioning of climate analysis/adjustment methods and software (and the related push to communicate that ‘the science is settled, and deniers should be ____’)

and

2) Policy PR recommendations, such as published here:

…we need to work in a more shrewd and contemporary way, using subtle techniques of engagement. To help address the chaotic nature of the climate change discourse in the UK today, interested agencies now need to treat the argument as having been won, at least for popular communications. This means simply behaving as if climate change exists and is real, and that individual actions are effective. The ‘facts’ need to be treated as being so taken-for-granted that they need not be spoken…Ultimately, positive climate behaviours need to be approached in the same way as marketeers approach acts of buying and consuming. This is the relevant context for climate change communications… (Executive Summary, page 8, IPPR report “Warm Words: How are we telling the climate story and can we tell it better?” August 2006)

I do NOT want to hijack this thread into context of politics or marketing.

However, it seems worth at least noting that certain arenas of scientific work today just possibly may be influenced by marketing PR, and not purely emerging from the available data.

When considering the quality of work done, perhaps we can best compare what we’re seeing to the kind of analysis that supports marketing of a new automobile** rather than a trillion dollar investment in the future of the planet.

**Apocryphal story: when the Chevy Nova didn’t sell in Latin America, who was harmed but a few shareholders? Who woulda thunk that “No Va” means “Doesn’t Go” in Spanish? (It’s actually not true… but perhaps that’s the point. AGW research is no laughing matter, folks…)

Willis: ….. Hansen’s method may prove to be error-free, or self-canceling … although the first site analyzed is hardly encouraging. My main problem is the laziness of the investigators.

They are anything but lazy. A growing body of evidence indicates the AGW climate modelers and the dendro-climatologists are actually dedicated, productive artisans in lab coats who are using the computational sciences as productivity-enhancing tools in sculpting raw, unpolished data sets into pretty curves which fit their artistically-motivated ideas as to what they want to communicate to their artisan peers and eventually to the public.

Supplying a user manual which documents in good detail how the computational tools are actually being employed would expose these data sculpting methods for what they actually are.

This means we are not likely ever to see the AGW modelers willingly adopt any kind of self-policing software quality assurance ethic. If they ever did so, they would be voluntarily telling us what it is they are actually doing.

Supplying this kind of information could ultimately prove fatal to their careers as AGW data artisans, assuming some group of economically and politically influential people ever began taking the low-level details of their work seriously enough to challenge it.

In this particular case, Hansen’s calculations, viewed as statistical estimates, do not coincide with the maximum-likelihood estimates, which are arrived at in the lmer calculation. At the present time, it’s hard to tell whether the Hansen method would tend to cause the introduction of an upward trend relative to the maximum-likelihood calculation (as in the above case) or whether the trend differences from maximum likelihood estimates are randomly distributed between positive and negative values.

Would it be possible for you to either link a description of what the lmer calculation does, or better, to post an explanation of it?

I find the problems with Hansen’s ad hoc method to be illustrative. We have a small,tightly knit and perhaps even insular community that everybody is relying on for accurate information on how the Earth’s climate is changing over time, and what influence human activity is having on it, yet there appears to be little of self-auditing activity in that field, but rather, if anything, more of a wagon-circling going on…