Roundup

Well, welcome back.

The operating problems have resulted either from an increase in volume, or a DOS attack or both. Any time that we’ve tried to open up the site, the site has been knocked over. This has been true even on the weekend. Even after the climateaudit IP address was pointed away from the server, the hitting of the server continued unabated.

Notwithstanding this, you’ll notice a few comments over the past week. For most of the week, the site has been in partial lockdown as any attempt to open up has ended with the site being swamped. For the most part, anyone who’s been able to see the site has the server as me (e.g. my neighbor, bigcitylib) or John A in England or Anthony in California.

During this time, instead of my usual fairly quiet circumstances, I have had my 15 seconds of publicity and have been very busy particularly with Canadian radio talk shows. By contrast, there was virtually no Canadian coverage of the Hockey Stick although there was extensive international and American coverage.

I seem to have got onto the Canadian radio talk show play list for a day or two. At one point today, I could be heard simultaneously on two different Toronto talk shows (one taped, one live). The story seems to have got traction here, not because of American coverage, but because of an article in The Toronto Star, a very liberal and Liberal paper, entitled “Red faces at NASA over climate-change blunder: Agency roasted after Toronto blogger spots `hot years’ data fumble”. This story then got linked by the Drudge Report.

Today I taped a segment for CBC Toronto at 2 pm, taped a segment for CBC national radio at 2.30, appeared on John Moore CFRB Toronto at 3.30, – all of which required pre-interviews with producers during the morning, was interviewed by Marcel Crok from NWT Holland at 4 pm. Oh, yes, I also was on Calgary radio last Friday, Minneapolis radio today and interviewed by a U.K. newspaper on the weekend. Tomorrow I’m scheduled on Washington radio at 7.20 am, for John Oakley (AM640 Toronto) at 8.15, then CTV national television is coming to my house for an interview at 10, another interview at 11.30; plus I’ve got about 3 more radio invitations I haven’t coordinated yet. And about 5 requests that I haven’t responded to yet – just in: a magazine in Brazil, television station in US.

I think that the “NASA” error is hitting some hot buttons. One of the calls came from a reporter who was in Florida covering the space shuttle and was intrigued with the idea of a “Toronto blogger” identifying an error for NASA. He didn’t know Hansen from Hansel and Gretel.

On blog operations, we’ve received suggestions that we should have re-located to blogspot. Blogspot uses “Blogger” software, while this site has been constructed in WordPress and a changeover is not something that is really very practical at this point.

Adding to the difficulties, about 2 weeks ago, John A asked me to start the process of finding new technical support for CA; his efforts had been all volunteer and he had pressing personal and work obligations. Needless to say, instead of being able to accommodate this request, we were immediately hit by a deluge of problems. Despite his request, John A has worked long hours to resolve matters without a whisper of complaint, all on a voluntary basis and without complaint or compensation.

We’ve re-located to a server in California with Anthony Watts’ able assistance. I am grateful to Anthony for his help in re-locating the website, but especially to John A both for his present efforts in re-locating the website and for his past efforts. I hope that the re-location will enable me to accommodate John A’s request to spend far less time on CA. I would particularly appreciate it if people would recognize John A’s efforts by making deposits to the PayPal jar; such contributions will be forwarded to John A as an honorarium rather than used for other operating expenses. I’m sure that John A would also appreciate a few kind words from readers who have benefited from his efforts.

Does Hansen’s Error “Matter”?

There’s been quite a bit of publicity about Hansen’s Y2K error and the change in the U.S. leaderboard (by which 1934 is the new warmest U.S. year) in the right-wing blogosphere. In contrast, realclimate has dismissed it a triviality and the climate blogosphere is doing its best to ignore the matter entirely.

My own view has been that matter is certainly not the triviality that Gavin Schmidt would have you believe, but neither is it any magic bullet. I think that the point is significant for reasons that have mostly eluded commentators on both sides. Continue reading

“Lights Out Upstairs”

Hansen has published an online letter entitled A Light On Upstairs? The letter concludes by saying:

My apologies if the quick response that I sent to Andy Revkin and several other journalists, including the suggestion that it was a tempest inside somebody’s teapot dome, and that perhaps a light was not on upstairs, was immoderate. It was not ad hominem, though.

I haven’t seen the original letter and don’t know who the comment was about. However, it certainly sounds like an ad hominem remark and one that is highly inappropriate for a federal civil servant. I have a number of comments about other aspects of the letter. Continue reading

If it seems a little quiet…

I’ve had to radically filter the connections to the weblog until we get our webhosting sorted out (hopefully this weekend).

If you can read this, then you’re fortunate to be on a restricted number of subnets. The rest are getting a “403 Forbidden” error.

We’ve had to block pretty much everybody because of the enormous traffic from Instapundit, digg, slashdot, Rush Limbaugh and Fox News.

And a few hundred blogs.

Steve has just been on Calgary radio.

I’m sorry for the block, but it was necessary in the circumstances. Hopefully by Monday we’ll be on the new server.

Unthreaded #18

Continuation of Unthreaded #17

Sorry for the loss of service

The server slowed down and stopped repeatedly as CA was either a) under a DDOS attack and/or b) got mentioned by Instapundit (the result is the same, the question is intent).

I’ve changed the theme temporarily to make the site a bit faster.

We are currently building a new server able to withstand these sorts of events.

Update: We also got mentioned by Rush Limbaugh, Digg, Slashdot and quite a few other blogs. Tonight its going to be covered by Fox News. Oh the joy.

A New Leaderboard at the U.S. Open

There has been some turmoil yesterday on the leaderboard of the U.S. (Temperature) Open and there is a new leader.

A little unexpectedly, 1998 had a late bogey and 1934 had a late birdie. (I thought that they were both in the clubhouse since the turmoil seemed to be in the 2000s.) In any event, the new leader atop the U.S. Open is 1934.

2006 had a couple of late bogeys and fell to 4th place, behind even 1921. I think that there’s a little air in the 2006 numbers even within GISS procedures as the other post-2000 lost about 0.15 strokes through late bogeys, while it lost only 0.10 strokes. It is faltering and it might yet fall behind 1931 into 5th place.

Four of the top 10 are now from the 1930s: 1934, 1931, 1938 and 1939, while only 3 of the top 10 are from the last 10 years (1998, 2006, 1999). Several years (2000, 2002, 2003, 2004) fell well down the leaderboard, behind even 1900. (World rankings are calculated separately.) Note: For the new leaderboard see http://data.giss.nasa.gov/gistemp/graphs/Fig.D.txt. The old data has been erased; by sheer chance, I had the old data active in my R-session but I can’t give a link to it.)

GISS U.S. Temperatures (deg C) in New Order

Year Old New
1934 1.23 1.25
1998 1.24 1.23
1921 1.12 1.15
2006 1.23 1.13
1931 1.08 1.08
1999 0.94 0.93
1953 0.91 0.90
1990 0.88 0.87
1938 0.85 0.86
1939 0.84 0.85

Here’s the old leaderboard.

Year Old New
1998 1.24 1.23
1934 1.23 1.25
2006 1.23 1.13
1921 1.12 1.15
1931 1.08 1.08
1999 0.94 0.93
1953 0.91 0.90
2001 0.90 0.76
1990 0.88 0.87
1938 0.85 0.86

Will the Real USHCN Data Set Please Stand Up?

The GISS homepage formerly said:

The NASA GISS Surface Temperature Analysis (GISTEMP) provides a measure of the changing global surface temperature with monthly resolution for the period since 1880, when a reasonably global distribution of meteorological stations was established. Input data for the analysis, collected by many national meteorological services around the world, is the unadjusted data of the Global Historical Climatology Network (Peterson and Vose, 1997 and 1998) except that the USHCN station records included were replaced by a later corrected version.

In econometrics, you couldn’t use loosy-goosy phrases like “replaced by a later corrected version.” You’d have to identify the version. Actually AGU policies (which apply to JGR) require proper data citation, although climate scientists publishing in AGU journals flout this policy, including Hansen here. Following an email to Hansen and Ruedy of GISS, they changed the introduction at the GISTEMP intro from:to the following:

Input data for the analysis, collected by many national meteorological services around the world, is the unadjusted data of the Global Historical Climatology Network (Peterson and Vose, 1997 and 1998) except that the USHCN station records up to 1999 were replaced by a version of USHCN data with further corrections after an adjustment computed by comparing the common 1990-1999 period of the two data sets. (We wish to thank Stephen McIntyre for bringing to our attention that such an adjustment is necessary to prevent creating an artificial jump in year 2000.)

This doesn’t really clarify the provenance of the data. In his email, Ruedy added:

In 2000, USHCN provided us with a file with corrections not contained in the GHCN data. Unlike the GHCN data, that product is not kept current on a regular basis. Hence we used (as you noticed) the GHCN data to extend those data in our further updates (2000-present)

Well, I hadn’t really “noticed” that they had used GHCN data to extend the USHCN data. I’ve done lots of cross-comparisons with different variations trying to identify exactly where the GHCN raw data came from. I’ve put up many plots at CA showing GISS raw as compared to different USHCN versions and I’ve done many more that I’ve not posted up. At his point, there’s one thing that we can say for sure: I’ve now looked at GISS raw as compared to GHCN raw data in the post-2000 period and in the few sites that I’ve examined since receiving this email (Detroit Lakes, Port Angeles), there is an exact match. So at least we’ve tracked down one aspect of the provenance of GISS raw data. I’ve done a comparison plot below for Detroit Lakes MN, a series that we’ve looked at before. After 2000, the match is exact (the delta is 0.0). However, before 2000, the USHCN TOBS/adjusted series sort of match for a while but the match tails off in the earlier portions, with the GISS raw version being quite different.

hansen48.gif

So where does the GISS raw version come from? At this stage, I think that we can declare that it doesn’t come from any of the USHCN version 2 (raw, TOBS, adjsted). If it did, then the versions would match as exactly as the GHCN raw and GISS raw after 2000. So where does it come from? Another stupid climate science guessing game, although hopefully it will solved in a shorter time than the unsolved MBH99 confidence intervals.

Ruedy’s letter has opened up the possibility of really obsolete data being used – I hadn’t thought to look at really obsolete data. I browsed through some obsolete data in connection with Swindle – who would have thought that we’d be doing so again. Two possibilities spring to mind: (1) maybe he’s using something from USHCN version 1; this is online, but I don’t think that the dates are right. (2) Maybe there’s an earlier USHCN version 2 that’s been overwritten (3) maybe he’s using an old GHCN version before 2000. Maybe none of the above – hey, it’s climate science.

Scraping USCRN Data

The US CRN (Climate Reference Network) appears to be a generally well-designed network for measuring 21st century temperatures. Its mission statement includes an undertaking to make its results available online. Here as with GISS and the metadata, system designers have provided webpages – in a format that may be interesting for very casual users, but a nightmare for serious data processing. People interested in monthly data are apparently expected to cut and post innumerable monthly data sets.

I’ve written a short read function to scrape monthly CRN results by individual station at http://climateaudit.info/scripts/station/read.uscrn.txt . This uses the style of Nicholas’ function to scrape information from GISS.

You can look up station IDs either at http://mi3.ncdc.noaa.gov/mi3report/MISC/CRN-STATIONS.TXT (outdated but ASCII readable) or http://www.ncdc.noaa.gov/app/isis/stationlist?networkid=1 (up-to-date but inconvenient for searching. I’ve scraped a more usable form of this as well).

Because of the crummy CRN interface, you have to individually download every month to extract the monthly data. So it ends up taking a fairly long time to extract the tiny amount of data of interest. (I’ve written to Karl suggesting that they provide a sensible ASCII-based archive).

Here’s an example (also showing a download of GHCN daily data for Detroit Lakes for comparison) and comparing the Goodridge MN CRN site and the Detroit Lakes MN USHCN site, yielding the figure below:

source(“http://climateaudit.info/scripts/station/read.uscrn.txt”
goodridge=read.uscrn(id=1039) #Goodridge MN

source(“http://climateaudit.info/scripts/station/read.ghcnd.txt”
detroit=read.ghcnd(usid=212142) #DEtroit LAkes

plot(c(time(Y)),Y[,4],type=”l”,ylab=”deg C”)
lines(c(time(X$monthly)), X$monthly[,”ushcn.area”],col=”red”)
lines(floor(detroit$time/100)+(detroit$time%%100 -1)/12,detroit$tmean,col=”red”)
lines(floor(detroit$time/100)+(detroit$time%%100 -1)/12,detroit$tmean,col=”red”)

combine=ts.union(ts(detroit,start=c(floor(detroit$time[1]/100),detroit$time[1]%%100),freq=12),goodridge)

nf=layout(array(1:2,dim=c(2,1)),heights=c(1.1,1.3))
par(mar=c(0,4,2,1))
plot(c(time(combine)),combine[,10],type=”l”,ylab=”deg C”,xlim=c(2002,2008.3),xlab=””,axes=FALSE)
lines(c(time(combine)),combine[,6],col=”red”)
legend(2007.2,-10,fill=1:2,legend=c(“Goodridge CRN”,”Detroit L USHCN”),cex=.7)
axis(side=1,labels=FALSE);axis(side=2,las=1);box();abline(h=0,lty=2)
title(main=”Comparing CRN and USHCN in Minnesota”)
par(mar=c(3,4,0,1))
plot( c(time(combine)),combine[,6]-combine[,10],ylim=c(0,3.75),xlim=c(2002,2008),type=”l”,ylab=”Deg C”,xlab=””)
points(c(time(combine)),combine[,6]-combine[,10],pch=19,cex=.8)

detroi47.gif
Figure 1. Top – Detroit Lakes MN data from GHCN daily and Goodridge MN CRN data (both as average of max and min); bottom – Detroit Lakes minus Goodridge MN

BTW the NOAA robots.txt file – not that I agree that this is relevant to acquiring temperature data from a site that is supposed to provide temperature data – does not disallow access to the crn/ directory and subdirectories.

A "lights=1" USHCN station

Don Healy sends this photo of Port Angeles, Washington’s USHCN station #456624:

Port Angeles USHCN station

Increasingly, non standard equipment is being observed as substitutes for max/min thermometers and MMTS systems. In this case, vandalism issues with the MMTS forced a change to this station setup in 2002 that was less accessible.

Port Angeles pole station

In 2002 a change shows up on the GISS graph for the station. A step of about 2 degrees C seems apparent at that time, but other stations in the area also have a similar event, so it is unclear just how much of a magnitude this equipment change, elevation change, and location change may contribute to the Port Angeles record.

Oddly though, while the change in lat/lon shows up in NCDC’s MMS, not a single mention of the new equipment can be found, nor a mention of the new observing height. MMS still says all the original equipment is in use:

2003-05-17 ] 9999-12-31 PRCP F&P PRIMARY FISCHER/PORTER RRNG PRECIPITATION — 84-880-067 COOP HPD
SRG STANDARD RAIN GAGE — COOP SOD
TEMP MMTS MMTS ELECTRONIC SENSOR TEMPERATURE 4829

According to Don’s survey notes, the old MMTS is still at the old site:

“In 1987 the station was moved to the location on the lawn at the Southeast corner of the Port Angeles City Hall (see photos of Prior Location). It remained there until 2002. The exposed location allowed for repeated acts of vandalism, and while the MMTS still remains on the site, it no longer functions. In 2002, the City of Port Angeles purchased a weather station from the Davis Instrument Company and installed it on the top of the utility pole. NOAA was made aware of the change and accepts the data.”

I wonder if this new equipment gets regular inspections at this height to check for wasp nests in the IR shield or bird excrement in the rain gauge?

Complete survey photo essay and details here on http://www.surfacestations.org

UPDATE( Steve Mc):
I’ve done a few figures checking the adjustments at Port Angeles WA. Station moves recorded in 1952, 1985, 1988. TOBS changed from midnight to morning in 1935; to afternoon in 1944, except for a short interval at 10 pm in the last 1940s. MMTS was introduced in 1984. The USHCN stage adjustments are shown here:

portan46.gif

Next is a comparison of GISS stage adjustments – Port Angeles has a significant Y2K error. It also has a very large “UHI” adjustment to unlit areas (it has GISS-brightness of 28) that reverses the Y2K and more.

portan44.gif

Here is a comparison of the USHCN TOBS version and the GISS adjusted version. Neither shows any particular trend; both show a very warm 1998 with something of a mean-reversion since then.

portan45.gif