Comments on: Carnage

By: Nic L

Nic L — Mon, 16 Feb 2009 18:57:49 +0000

#150. Steve, I’m afraid I have no idea why the Dec 2002 data wasn’t QCed until Oct 2008. That is in fact the last month for which QCed AWS data is stated to be available from the University of Wisconsin – Madison: see ftp://aws.ssec.wisc.edu/pub/aws/antrdr/readme.aupdates. AWS which Madison deals with seem to be listed at ftp://aws.ssec.wisc.edu/pub/aws/biglist.
I have checked the BAS READER HTML web pages and can confirm that (as of last Friday at least) there seems to be no QCed data (in black) yet for January 2003 to January 2007 inclusive for any of the stations on their list, with the exception of Ferrell (Nov & Dec 2006 and Jan 2007), Marble Point (Oct, Nov & Dec 2006 and Jan 2007) and Larsen Ice Shelf (Jan 2003 to Mar 2004 and Nov & Dec 2004). I have checked this back to the ftp directories at Madison, which are empty for 2005 and have only a few monthly data files for 2004 and 2006.
It is difficult to see how Steig could have downloaded his data in 2006, since it extends to December 2006. I suppose he could have downloaded data up to an earlier date during 2006, before the pre-2002 missing data points that I identified were in the READER text files, and then done another download in 2007 to get subsequent data up to December 2006 and not replaced the data for earlier months that he had downloaded in 2006. Or maybe the missing pre 2002 data points were not after all added to the READER text files until later in 2007 than Steig downloaded his data.

By: conard

conard — Mon, 16 Feb 2009 17:46:10 +0000

Re: Steve McIntyre (#127),

I know that you like to do as much as you can in R but here is a script (extracted from my GISTEMP step0 emulation) that downloads and diffs the data. Once I have the data local I go ahead and start my calcs

http://www.climateaudit.org/phpBB3/viewtopic.php?f=11&t=678

By: Steve McIntyre

Steve McIntyre — Mon, 16 Feb 2009 16:17:36 +0000

#149. Nic, does that bracket the “vintage” of Steig’s data version to some time in 2006 or thereabouts? Do you have any idea why the Dec 2002 data wasn’t QCed or whatever until Oct 2008?

By: Nic L

Nic L — Mon, 16 Feb 2009 15:33:19 +0000

Further to my post #148, I have now ascertained from BAS that most of the December 2002 all-day monthly average temperatures were displayed in red and so did not appear in the text files until October(?) 2008, when quality controlled data became available. Therefore, it is unsurprising that they are missing from Steig’s reconstruction. Likewise, the missing late 2006 data points for Ferrle and Marble point were probably not included in the text files when Steig obtained the data, as quality controlled files for them only became available in December 2008. Accordingly, contrary to my previous thinking it appears that Steig was not responsible for any of these points being missing from his AWS reconstruction. However, BAS have confirmed that the earlier data I identified as missing were available for download in 2007.

By: Nic L

Nic L — Thu, 12 Feb 2009 23:58:11 +0000

Re: Ryan O (#147),

Personally, I think Steig removed some data points that looked like outliers and simply calculated the anomalies based on all available data. There’s also the possibility that he just used all the available data and there were quite a few changes between his data set and Steve’s. I think the former is probably more accurate based on which data points were removed. There’s probably some of the latter mixed in as well.

I have worked out that the following points for which actual readings existed, at the time that Steve archived them, were not used by Steig. Once these points are removed in doing the spliced calculations, everything agrees to his reconstruction (subject in the case of some AWS and particular months of the year to constant differences for all years that actual data for that month has been used – see below).

The data points (months and actual mean temperatures) that do not appear to have not been used by Steig are:

Butler Island: Dec02 -3.0

Cape Denison: Jan01 -0.1, Feb01 -5.9, Jun01 -17.7, Jul01 – 16.9, Aug01 -18.2, Dec02 -1.2

Clean Air: Dec02 -29.3

D47: Mar84 -21.2, Apr84 -30.3, Jul84 -34.0, Aug84 -27.9, Sep84 -29.1,Oct84 -27.6,Dec84 -13.7, Apr96 -27.9, May86 -31.8, Apr88 -26.6, Jan93 -12.1, Aug94 -34.5, Sep94 -25.7, Oct94 -25.1

Dome CII: Dec02 -29.2

Elaine: Dec02 -9.2

Ferrel Dec02 -7.0, Nov06 -13.1, Dec06 -6.3

GEO3: May83 -33.8

Gill: Dec02 -9.1

Harry: Dec02 -11.8

Henry: Dec 02 -27.0

Larsen Ice Shelf: Aug95 -24.7, Dec02 -1.4

LGB20: Jan03 -50.3

Manuela: Dec02 -4.9

Marble Point: Dec02 -4.9, Oct06 -19.2, Nov06 -8.1, Dec06 -3.9

Marilyn: Dec02 -8.0

Minna Bluff: Dec02 -10.7

Mount Siple: Dec02 -1.9

Pegasus North: Dec02 -5.7

Pegasus South: Dec02 -3.5

Port Martin: Dec02 -2.1

Possession Island: Dec02 -1.0

Racer Rock: Dec02 0.3, Dec05 -12.5 (this is the old Racer Rock Dec05 temperature)

Relay Station: Dec02 -29.8

Schwerdtfeger: Dec02 -9.4

Whitlock: Dec02 -5.8

The huge concentration of deleted data points in December 2002 certainly seems to point in the case of that month to something Steig did rather than to any changes in the READER files before Steve archived them, but it could just be a processing error and lack of checking that is to blame rather than a deliberate decision to exclude the points. It does appear that almost all the missing December 2002 data points were below the December average for the station concerned, but I am not sure that the missing points are outliers.

As mentioned above, in some cases the amount that has to be deducted from each calendar month’s actuals to get the temperature anomoly per Steig’s reconstruction does not exactly equal the average up to December 2006 of that month’s temperature readings. The AWS affected are Butler Island, Cape Ross, GEO3, GF08, LGB35, Larsen Ice Shelf, Law Dome Summit, Limbert and Terra Nova Bay. Almost all the differences disappear if the averages for these AWS are calculated up to June 2008 (July 2008 for Cape Ross, October 2007 for Larsen Ice Shelf) instead of to December 2006, but it remains in 7 of the 756 cases. Very strange.

By: Ryan O

Ryan O — Thu, 12 Feb 2009 20:12:03 +0000

It’s still kind of backwards, but when I removed from the raw data all points that didn’t lie on zero after calculating using the mode, and then recalculated anomalies using the data that was left, I get Steig’s anomalies exactly.
.
Prior to realizing the data changed between Steve’s archive and my download, I was mystified because it looked as though Steig used varying baselines. But when I used Steve’s archive, that problem disappeared (BAS has done a LOT of data changing over the past week apparently).
.
Personally, I think Steig removed some data points that looked like outliers and simply calculated the anomalies based on all available data. There’s also the possibility that he just used all the available data and there were quite a few changes between his data set and Steve’s. I think the former is probably more accurate based on which data points were removed. There’s probably some of the latter mixed in as well.
.
But in the end, once I used Steve’s archived data, I’m convinced Steig just calculated anomalies using all of the information available – minus any outliers he didn’t like.

By: Steve McIntyre

Steve McIntyre — Thu, 12 Feb 2009 19:42:43 +0000

#144. Ryan, other than getting at the data backwards, can you figure out how Steig did a forward calculation of anomalies? In this sort of detection, I usually try short files and see if there are any clues.

By: Ryan O

Ryan O — Thu, 12 Feb 2009 19:33:22 +0000

Correction: missing a paren

modev=names(sort(-table(data[i,j])))[1]

By: Ryan O

Ryan O — Thu, 12 Feb 2009 19:32:11 +0000

Re: Nic L (#142), Yep.

I would guess that most of the category 1 differences are more likely to be due to errors made by Steig than to changes in the READER files between when he obtained the data and when Steve did.

.
That was my conclusion as well.
.

It is more difficult to work out what the differences in the first category are, as we don’t have direct access to Steig’s data, and the adjustments he will have made to get to temperature anomolies from actuals will be based on the data he has used.

.
This actually isn’t as hard as it might appear at first. You can get Steig’s anomalies exactly by simply subtracting the RECON from the ACTUAL and taking the mode of the result for each month. You don’t even have to know which months he used for the calculation.
.
What’s odd is that sometimes he only used partial data for the anomaly calculation, yet included the full data in the recon.
.
To get the mode of the data, you can use this:
.

modev=names(sort(-table(data[i,j]))[1]

.
The sort produces a vector of frequencies, ordered from most frequent to least frequent, and the names of each quantity is the value associated with the frequency.

By: Steve McIntyre

Steve McIntyre — Thu, 12 Feb 2009 19:15:45 +0000

Noted at BAS today:

(11 Feb 2009) Incorrect data for Bonapart Point for the whole of 1999 and January 200 have been updates with values from the quality controlled files, as the values received on the GTS appear to come from a different station. All of the incorrect values were displayed in red and did not appear in the downloadable text file.

(11 Feb 2009) The name of Bonapart Point has been changed to Bonaparte Point as this is the correct name.

(11 Feb 2009) The station height for Terra Nova Bay has been corrected from 81m to 92m.

(11 Feb 2009) It has been decided not to calculate AWS wind data from the GTS data as the sensors can sometimes freeze. Only values obtained from the quality controlled files provided by Madison will be displayed.