ICOADS – Hawaii

Sep 1, 2010 – 8:24 PM

Although the formatting of the SST datasets needs to be completely freshened up, once again, before commenting, I commend the SST collaters for honoring their data by ensuring the preservation of comprehensive metadata – as opposed to their cousins at CRU and GISS. Unfortunately, there don’t seem to be any statistical analyses of SST measurements – by this, I mean, where the authors analyse the actual bias of changing measurement systems and provenance. (Although Thomson et al [Nature 2007] challenged earlier bucket adjustments, they didn’t do the sort of patient data analysis that the field cries out for. Read More »


By Steve McIntyre | Posted in Uncategorized | Tagged , , | Comments (4)

A First Look at ICOADS

Aug 30, 2010 – 4:27 PM

For quite a while, I’ve urged people interested in gridded temperatures to really look at the SST data – realdata not adjusted data. SST makes up 2/3 of the record, but temperature critics spend 99.99% of their time on land data. In part, it’s because the data sets are much larger, but increased power of ordinary laptops is making these data sets accessible without work stations, whereas this was not the case even a few years ago. I’ve taken a first look at ICOADS – and since we’re talking climate science – naturally the data has some peculiar features. ICOADS collaters deserve great credit for their care with metadata – they obviously feel a responsibility towards the data that wasn’t felt by CRU (who notoriously kept only the “value added” version.) The data sets are large and rich and deserve a great deal of statistical analysis – basic cross-classifications as opposed to rushing off to make bucket adjustments. (I’m going to put this file away unfortunately, but commend it to others.) Read More »

By Steve McIntyre | Posted in Uncategorized | Tagged , , , | Comments (30)

Tar and Z

Aug 30, 2010 – 11:45 AM

Over the weekend (before I picked up my “regular” files), I started looking at Steve Mosher’s use of raster and zoo – both of which intrigue me a great deal, but got intrigued by something else and ended up finally figuring out how to extract .Z files within an R script without having to handle them manually. (R has utilities for .zip and .gz files, but not the older .Z format.) This isn’t anything other than a nuisance with GHCN which only has one .Z file to worry about, but was a big problem with the very large ICOADS SST data where every month of data is in its own .Z file and manual processing isn’t an alternative. It’s further complicated since the 12 monthly .Z files for each year and packaged into an annual .tar file. Read More »

By Steve McIntyre | Posted in Uncategorized | Tagged , , , | Comments (7)

Back from Erice

Aug 30, 2010 – 10:28 AM

Got back from the annual WFS conference at Erice, Sicily a couple of days ago. It is an interdisciplinary conference on world issues, in which climate is only a part, but an important part. As in previous visits, it was a very enjoyable visit – the conference attracts a lot of spouses, resulting in more mixing between disciplines than would occur otherwise as the spouses create their own mixing.

Antonino Zichichi, who organizes the conference, is not convinced that climate sensitivity is very great and the climate invitees had a much stronger “skeptical” bent than previous years. I ended up as a panelist in a discussion on climate with Richard Lindzen, Will Happer and Richard Wilson (of Harvard, another prominent nuclear physicist who’s taken an interest in climate.) I spent quite a bit of time with all three.

The conference had interesting presentations on science issues related to the BP oil spill – a discharge estimate of about 70,000 bbl/day seems to be the final number.

The conference always has interesting presentations from nuclear scientists and engineers and this year was no exception. I find the discussions of designs and design improvements fascinating and reassuring.

While climate sensitivity was the large scientific issue, Climategate was on a lot of people’s minds – both for its fallout on climate science and science more generally. I ended up focusing my presentation on Climategate and the inquiries, neither of which made people very comfortable.

Obviously, the tide of climate news continues. I haven’t parsed McShane and Wyner yet and need to do so. As a relaxation when I got home a couple of days ago, I spent some time looking at Steve Mosher’s recent blog – Steve has done some slick applications of R packages raster and zoo (which I haven’t explored) to the analysis of data sets. I see that the IAC report is out today and is one more thing to read – I hope that it’s less bad than the others.

By Steve McIntyre | Posted in Uncategorized | Comments (25)

Kriging on a Geoid

Aug 26, 2010 – 1:24 PM

Geoff Sherrington and others on the First Difference Method post have requested a post for discussing Kriging.

I am new to Kriging myself, so please correct me if I make any errors here. Steve McIntyre (who may be on the beach at the moment!) is far more knowledgeable, and has posted about the topic frequently on CA. See, for starters, “Toeplitz Matrices and the Stahle Treering Network”, 3/22/08, “Antarctic Spatial Autocorrelation #1″, 2/20/09, “Steig Eigenvectors and Chladni Patterns”, and follow-up posts.

Wikipedia has a useful post on Kriging, in which it draws a distinction between “Simple Kriging”, which assumes a known constant unconditional mean for the random field, and “Ordinary Kriging”, where the unconditional mean is unknown. In the former case, the predicted value at an observed point in space will be a convex combination of the unconditional mean and an average of the observed values, while in the latter case it will just be an average of the observed values, with weights summing to 1. We are mostly concerned with “Ordinary Kriging”, though the distinction should be kept in mind.

Read More »

By Hu McCulloch | Posted in Uncategorized | Comments (40)

Replicating McShane and Wyner

Aug 23, 2010 – 9:08 PM

R coder mind of a Markov chain has replicated portions of the M&W work.

They write:

There are a bunch of “hockey sticks” that calculate past global temps. through the use of proxies when instrumental data is absent.

There is a new one out there by McShane and Wyner (2010) that’s creating quite a stir in the blogosphere (here, here, here, here). The main take out being, that the uncertainty is too great for the proxies to be any good.

Here’s an output from the replication:

More including R code here:

http://probabilitynotes.wordpress.com/2010/08/22/global-temperature-proxy-reconstructions-bayesian-extrapolation-of-warming-w-rjags/

By Anthony Watts | Posted in MBH98, Replication, Scripts | Comments (36)

The First Difference Method

Aug 19, 2010 – 9:36 PM

Over on WUWT (http://wattsupwiththat.com/2010/07/13/calculating-global-temperature/), Zeke Hausfarther and Steven Mosher have been discussing the calculation of global temperature from station data. They list several methods of combining records, noting that most of the major indices use the Common Anomalies Method (CAM). They mention, but do not discuss, the First Differences Method (FDM).

In fact, FDM is far superior to any method based on averaging anomalies. Simply averaging anomalies relative to each station’s mean (as in the much-discussed Steig et al 2009 study of Antarctic temperatures — see below) greatly understates any trend there may be in the data, while using a common period unnecessarily restricts the available data. At the same time, FDM eliminates the need for opaquely complex adjustments for TOBS or MMTS, and automatically takes care of station moves.

Suppose, to take a purely hypothetical example, that four stations, A, B, C and D, have partial data on years 1-5, as indicated in the following table:

Table 1: Temperatures (°C)

Station Yr 1 Yr 2 Yr 3 Yr 4 Yr 5
A 13 14 - - -
B - 10 11 - -
C - - 22 23 -
D - - - 15 16

Clearly all the stations point to a +1°C/yr uptrend in temperature.

If we compute anomalies for each station, as was done by Steig et al (2009) (to the best of my knowledge), and then average over available stations (S09 did something much more elaborate), we obtain the following:

Table 2: Anomalies

Station Yr 1 Yr 2 Yr 3 Yr 4 Yr 5
A -.5 +.5 - - -
B - -.5 +.5 - -
C - - -.5 +.5 -
D - - - -.5 +.5
Avg. -.5 0 0 0 +.5

The Least Squares trend in the average anomalies is only 1/6 1/5 °C/yr. [(-2)^2 + (-1)^2 + 0^2 + 1^2 + 2^2 = 10, not 12 as I computed last night!]

The First Difference Method, on the other hand, first computes annual differences as available for each station and then averages the first differences as available across stations. The average first differences are then cumulated from an arbitrary starting point (Year 1 in the following table), and then may be adjusted as anomalies relative to any desired reference period (Years 1-5 in the last row of the table):

Read More »

By Hu McCulloch | Posted in Uncategorized | Comments (135)

Signal to Noise Ratio Estimates of Mann08 Temperature Proxy Data

Aug 19, 2010 – 9:49 AM

Guest post by Jeff Id from The Air Vent (used by invitation)

Occasionally when working on one thing long enough, you discover something unexpected that allows you to take a step forward in understanding.  At the ICCC conference, I met Steve McIntyre and took time to ask him how come Mann07 “Robustness of proxy-based climate field reconstruction methods” didn’t show any variance loss in the historic signal.  The paper makes the claim that CPS is a functional method for signal extraction, which I’ve long and vociferously contested ;) .  Neither of us had a good answer, but I had to know.   In Mann07 – Part II at the Air Vent, the mystery was solved.  The M07 paper uses model data as a ‘known’ temperature signal and adds various levels of noise to it.  While the work oddly uses white noise in most operational tests, it does present the example of ARMA (1,0,0) ρ = 0.32 models, and it showed very little variance loss.  Replicating M07 using CPS wasn’t difficult and the results were confirmed – no historic variance loss so no artificially flat handles for the Mann hockeystick.

With white noise or low autocorrelation noise, there will be none variance loss (HS handle) reported in VonStorch and Zorita 04, Christiansen2010, McIntyre Mckitrick o5 or numerous other studies.   This is because low AR noise doesn’t create signal obscuring trends on a long enough timescale to make a difference.  However, if red noise having autocorrelation which matches observed values in proxies is used, we get a whole different result overturning the conclusions of Mann07. But, this isn’t the topic of this post.

Read More »

By Anthony Watts | Posted in Mann 2008, Mann et al 2007 | Comments (17)

Erice 2010

Aug 17, 2010 – 4:39 PM

I’m off to Sicily tonight for the 2010 conference of the World Federation of Scientists, hosted by the redoubtable Antonio Zichichi. I’ll be a bit spotty checking in.

I never did finish reporting on the 2009 conference as we got overtaken by Yamal and then by Climategate and the inquiries. I’ve got a better computer this year (enough battery life to get through the day and I’ll try to post some conference reports.)

I’m making a presentation in a session on Improving IPCC.

By Steve McIntyre | Posted in Uncategorized | Comments (50)

Briggs on McShane and Wyner

Aug 17, 2010 – 8:34 AM

As usual, a good analysis from Matt Briggs here

By Steve McIntyre | Posted in Uncategorized | Comments (53)