For quite a while, I’ve urged people interested in gridded temperatures to really look at the SST data – realdata not adjusted data. SST makes up 2/3 of the record, but temperature critics spend 99.99% of their time on land data. In part, it’s because the data sets are much larger, but increased power of ordinary laptops is making these data sets accessible without work stations, whereas this was not the case even a few years ago. I’ve taken a first look at ICOADS – and since we’re talking climate science – naturally the data has some peculiar features. ICOADS collaters deserve great credit for their care with metadata – they obviously feel a responsibility towards the data that wasn’t felt by CRU (who notoriously kept only the “value added” version.) The data sets are large and rich and deserve a great deal of statistical analysis – basic cross-classifications as opposed to rushing off to make bucket adjustments. (I’m going to put this file away unfortunately, but commend it to others.)
While I was at Erice, I chatted with William Kininmonth of Australia quite a bit. I’m mildly interested in examining what the most unchanging tropical gridcell looks like – looking closely at different data sets for such a gridcell. I was thinking of something in tropical Pacific. Will suggested the gridcell containing Honiara (near Bougainville) about 9S, 160E. Almost at the border of two gridcells.
The range for Honiara in GISS dset1 (monthly) is from 25.1 to 27.5 deg C (data at GISS only for 1951-1986 for some inexplicable reason.) Now that I could extract ICOADS SST into R, I thought that it would be interesting to see what SST realdata looked like. It’s climate science, so naturally a surprise awaited.
I collated ICOADS data from ftp://ftp.ncdc.noaa.gov/pub/data/icoads/ into R-objects from 1900 to 1980. This was a very short script, but it probably took over 8 hours. The data sets get very large as one approaches the present. In 1900, the data set is 6 MB, by 1980 is 160 MB and in 2006 was over 1 GB. The data sets have all sorts of information – wind speeds, lots of coded meta data and several versions of the data seem to be carried in the same record ( I haven’t decoded all the fields yet.) Each measurement has a different record, so you see why it is such a large field.
The most relevant fields for SST purposes seem to be the time of measurement (year, month, day, hour), lat, long, air temp, SST, plus metadata such as country and, as I found out, the “deck” from which the measurement originated. I extracted the data from the two gridcells adjacent to Honiara (157 and 162E; 7.5S), yielding 21,138 records from 1900-1980 with SST values ( I didn’t bring it forward only because the later data sets get very large and I was experimenting.)
As a plain vanilla exercise, I plotted the average temperature by month for these gridcells with minimal variation yielding the following strange series.
Figure 1. Average monthly SST from ICOADS for two Honiara gridcells
For comparison, here is the GISS dset1 version for the Honiara station plotted on the same y-scale. Why were these so different?
Figure 2. Honiara temperature (GISS dset1)
Re-examining the data, the SST data included records with both negative SST and air temperature. This is easy to show after the fact, but not so easy in the first pass. I experimented with permutations of typos in the SST column before it became clear to me that both air temperature and SST were negative. Here is data for January 1973 during a problem period. Obviously something was wrong if Honiara has below zero SST and air temperature.
Figure 3: Honiara gridcells+ SST versus Air Temperature
Parsing the January 1973 data, it turned out that all the negative data came from “deck” 732. Here’s a barplot of the January 1973 average of SST measurements by “deck”.
Figure 4. Honiara gridcells January SST by deck.
I tracked down information on deck identifications at ICOADS here , which showed that deck 732 derived from:
Russian Marine Met. Data Set (MORMET) (received at NCAR)
Which perhaps explains the negative values, though it’s not the most obvious source of data for Honiara gridcells.
The problem seems to arise in the lat-longs of the Russian data as transcribed by NCAR. I presume that this sort of thing gets screened out as unphysical when the gridded versions are made, but it seems like the sort of thing that people in the trade might have picked up by now – I noticed it in my first couple of hours of handling the data. It is very much to the credit of ICOADS compilers that they’ve preserved metadata with such diligence and making this sort of check possible. Collectors of marine data seem to feel an obligation towards their data that was obviously not felt by CRU when they discarded their underlying station data (keeping only the “value-added” version).
I don’t know how the SST grids check for unphysical values but one would presume that this error has not arisen in the original data but in the handling of it at NCAR and is correctable.
More problematic is that even if one scraps the obviously unphysical Russian data, the differences between values by “deck” remain very large as shown below – ranging from 28.3 deg C to 30.6 deg C in this very stable gridcell.
The largest value came from “International Marine (US- or foreign-keyed ship data)”, the lowest value from “Autodin (US Dept. of Defense Automated Digital Network)”.
I did a similar experiment with data extracted from gridcells near Hawaii (where I compared CRU and Ersst data on another occasion.) Here is a similar example for gridcell 22.5N , 157.5W, where there is 3.2 deg C difference in the average between Netherlands and Germany/USSR. There is a similar pattern in gridcell 17.5N 157W where Netherlands was again the lowest.
It seems odd that the scientists in this field are able to make bucket adjustments to tenths of a degree in the 19th century when differences as large as this arise in the 1970s. Since Netherlands accounts for a larger proportion of early data than later data, it would be interesting to check whether this is a potential bias.
Please do not interpret these as anything more than notes. The ICOADS data set is huge; the compilers have taken care to preserve enough metadata to make possible some interesting statistical analyses. I don’t think that I’ll follow up much further right now as these sorts of data sets, no matter how interesting, quickly become black holes for time. However, I urge some of the people who worry about UHI to start looking at SST data. I’ve posted up some tools to facilitate access to ICOADS.
Climate Audit 
30 Comments
Fascinating detective work. It seems that this data deserves some very careful analysis. Shouldn’t Jones et al have isolated these issues?
Steve – Jones didn’t do analysis of marine data. Actually he didn’t do much analysis of land data either.
However, Jones co-auhtored Thompson et. al., A large discontinuity in the mid-twentieth century in observed global-mean surface temperature, Nature 453, 646-649 (29 May 2008), doi:10.1038/nature06982 ( http://www.nature.com/nature/journal/v453/n7195/abs/nature06982.html )
As I understand, they are claiming that SST drop in 1945 can be explained by the fact that percentages of SST observations from US and UK ships changed from 1945 US: 80%, UK: 5% to 1949 US: 30%, UK: 50%, and that there were clear measurement bias between US and UK observations (engine intake vs. bucket).
That is reasonable hypothesis but it seems like for some reason or other the authors did not check the actual ICOADS data. Instead, they made a model based on assumed measurement bias and the change of country of origin – and (surprise, surprise) found out that their model “explains” the abrupt cooling. Furhermore, they told that they are planning to make adjustments to HadSST according to these results.
I have been wondering why they didn’t check the ICOADS data – or why the reviewers did not require that. They could have divided the ICOADS data to US and UK portion – and if there were no rapid SST cooling in either data set, then it would be quite certain that the cooling was not a real but just an artifact caused by improper merger of US and UK data.
So I guess that would be one rather interesting thing to check?
That is what I was thinking about when I posed the question. I always assumed that they had to do some data inspection before they compiled their grids. But after all, iiocs!
Re: kse (Aug 31 09:14), That was the infamous paper largely lifted from a climate audit post “The team and pearl harbour”, see also “Nature discovers another climate audit finding”.
@bernie, I think the speculation is that gross errors of this sort ae corrected by sanity checks when the raw data undergo processing at point of use.
Still, useful to get some sort of handle on the mist fundamental sources of error so early in the process.
@smc what spec. PC does this stuff get run on?
Compilers of marine data also report evidence of equally large systematic errors in their measurement data, that are removed only as ad hoc means. This, also, makes a mockery of reporting temperatures and trends to tenths of degrees.
This is truly value added research.
A guest investigator looks at what the data says, without preset agenda or thesis to pursue.
Immediate insights are achieved regarding data quality.
Taxpayers own a debt of gratitude to Mr McIntyre. He works for free correcting the omissions of well funded research establishments.
I notified ICOADS of the potential error in deck 732. I’ll let you know if and when I hear back.
Re: Steve McIntyre (Aug 30 21:51),
where be the tools? that pdf is mind numbing
you could also just use the netCDF files.
ftp://ftp.cdc.noaa.gov/Datasets/icoads/1degree/
Steve: Not the same thing at ALL. I wanted to look at the underlying data; you’ve linked to the gridded product.
From some of the emials, I would be surprised if there isn’t at least one bucket of rats in this dataset. 🙂 The methods have changed so much over the years. I didn’t know the raw stuff was available and never even thought to look. It’s going to be very interesting what you find.
Re: Steven Mosher (Aug 31 06:04),
ok its also good to note that they supply the data with no QA
Temperature differences may be explained by the use of different sensors that measure the water at various depths, all the way from near surface with IR sensors to maybe 10 m below the surface with a thermometer on a buoy. Also, the location of the sensor, ie, deep water vs a shallow cove. If the metadata is good, someone may be able to translate data to a standard level.
Historical SST data are for climate research nowadays presumably not worth a straw, which, in my view, should be clearly distinguished from the time the SST were taken, as they served their purpose at that time. The following material may be of interest:
In one article of the „Annalen der Meteorologie“ Vol. 10-12 from 1951 (p.479), the design of a special bucket with thermometer for SST measurement that was in use since 1900 is described, including a drawing, here: http://www.2007seatraining.de/Archiv/images/aug2b_10.jpg
Another article in the same volume (p. 439-443) deals with the accuracy of deck and engine SST data, which had been investigated on the fishery protection vessel MEERKATZE (length 56,6 m; 673 register tons). The summary says:
“Of 410 comparative measurements of water temperature made on deck and in the engine room, in the area of the North Sea and the Norwegian Sea from June to October 1950, the differences between both methods, in 73,5% of all cases, were less than plus/minus 0,3°C. Systematic adulteration of the measurements in the engine room was avoided by suitable measuring methods. The negative difference between deck and engine occurring on wind force of more than 4 Beaufort may be due to incorrect water temperature measurement on deck in consequence of heat exchange and evaporation. By means of wind tunnel investigations, corrections are deduced which reduce the average deviations (deck – engine) to less than +/- 0,12°C.”
The daily practice on merchant ships was surely much less precise, but that was good enough for the weather services at that time, which got the SST immediately transmitted together with SAT, wind, clouds, air pressure, dew point.
The following script downloads data from ICOADS and extracts the subset that I’m using. I ran this overnight as I hadn’t extracted the deck information in my first pass and I saved a lot of information that I don’t anticipate using. The R-object saved here is about 25% of the size of the tarred and zipped ICOADS version, but large enough to make analysis slow for me unless I run this sort of download overnight.
The format information from the paper is collated into the table info_icoads.tab.
I’ve extracted subsets for two gridcells near Honiara and two near Hawaii – see climateaudit.info/data/icoads – also a count of measurements by gridcell (arranged in HadCRU order hour hand – N to S, minute hand – dateline to dateline).
I’m not nearly as worried as most people about how many decimal places of precision are in the measurement but biases from changing techniques and changing provenance does concern me.
Indeed the decimal do not matter, even +/- 1°C would be helpful if all other parameters would be fairly equal. But that is by far not alone a matter of “changing techniques and changing provenance”, but was effected by many other factors, e.g.
___taking water form a plain sea surface, or from a high wave;
___on a sunny day (when the most upper layer was particularly warmed, or at night.
At any time a very significant temperature difference is possible within a very thin surface layer of less than three meters, see the measurements by the FINNMAID in the Baltic Sea on 28-29 August 2010: http://www.itameriportaali.fi/en/tietoa/algaline_seuranta/mittaustulosarkisto/2010/en_GB/SalTemp100828/
measured at 3 to 4 meters depth, showing a trop in water temperatures from higher than 17°C to 10°C in the east of Bornholm towards Öland, over a distant of about 150 km, while the SST for the area ( southern Baltic Proper) is indicated with 15-17 deg.
The current SST forecast is here: http://www.itameriportaali.fi/en/itamerinyt/en_GB/pintalampoennuste/
snip – editorializing on policy
Not technical and not SST, but conceptually similar to this inquiry is the new book by historian Edward Kohn, ‘Hot Time in the Old Town,’ about a heat wave in the eastern US, and especially New York City, in 1896, where he relates that the Weather Bureau official temps were regularly 10 degrees cooler than the (nearer to the ground) recordings made by the Herald newspaper, and 5 degrees cooler than a long-running private research station in the city.
Sometimes, you don’t even need a computer.
Steve – This post is not about relative hotness.
An interesting exercise (not directly related to Steve M’s points, sorry) would be to plot the reported gridcell SST increase (1900-2000) versus the number of samples taken in say 1900-1950 in those gridcells.
I wonder if the regions of greatest reported SST rise over the last hundred years are also the ones which were poorly-sampled in the early part of the record.
Steve: counts are uploaded to climateaudit.info/data/icoads
David: A quick way to see where you’d want to look. COADS monthly coverage maps:
http://www.cgd.ucar.edu/cas/guide/Data/coads.sst.f1.html
And:
http://www.cgd.ucar.edu/cas/guide/Data/coads.sst.f2.html
Versus trends in HADISST/Reynolds OI.v2 via GISS:
or trends in ERSST.v3b:
Steve
Completely OT.
Was that stuff I sent you from the CRU FOI requests of any use?
Steve – yes, but I’m overloaded with things.
It is instructive to read and note the assumptions, corrections, errors and caveats in the document that comes up when you click ‘here’ in Steve’s
“I tracked down information on deck identifications at ICOADS here ,”
For the odd deck 732 of Steve’s figure 3, read the understatement:
[NOTE: Russian MORMET (deck 732) data are an exception to the
otherwise uniform storage of data in the supplemental attachment
using the “ship” character set as described above. For deck 732,
the data were written directly into the attachment rather than
translated into the ship character set. Since this may produce
unexpected results if deck 732 supplemental attachment data are
unpacked or printed using standard software, we intend to correct
this minor discrepancy as part of a future update.]
Hmmmm. Minor discrepancy.
Overall, it appears that an unfortunate amount of data was rounded then the originals lost. But in any case, reconstruction from a clean slate would be a massive job. The number of weather variables from the multi-choice sheet exceeds 100 and is a demonstration of how to design an off-putting, user hostile recording system to be used by many people who spoke different languages and used different conventions, like the number of points on a compass. It is such a shame that the records are in a shape so poor that they should not be used with optimism for serious work, like altering the world economy.
Even the 2010 description is wrong, for example on compression,
“Using the sea surface temperature (field 32) true value
28.6 degrees C as an example, (28.6/0.1) – (-1000) = 1286.
Once a given field has been extracted into a coded value, the true value can be
reconstructed by reversing the process:
true value = (coded + base) * units
The above true value example is reconstructed by (1286 + (-1000) * 0.1) = 28.6
degrees C. ”
The last equation gives a value of 1186 degrees C, not the intended 28.6, because it does not code the worded expression above it.
Wow. What a bower bird nest this is.
Hi Steve and all
First time posting here, hopefully this will be helpful for people looking at ICOADS as this information is buried away in the documentation on the ICOADS website and not immediately obvious.
When using the raw ICOADS data the trimming flags (characters 149 to 154 in the IMMA files) can (and probably should) be used to remove gross errors. Further information on them can be found here
http://icoads.noaa.gov/e-doc/stat_trim
Using this flag for SST does remove the problem identified by Steve above – that’s not to say that there aren’t other problems.
Steve: which columns is the relevant flag in?
It should be column 149 (if my maths is correct although it’s quite late in the day here).
Column 149 = SF (SST Trimming flag)
Column 150 = AF (air temperature flag)
Column 151 = UF (u wind component flag)
Column 152 = VF (v wind component flag)
Column 153 = PF (sea level pressure flag)
Column 154 = RF (relative humidity flag)
The fields are coded in base36, details can be found in table 1 (page 6) of the imma.pdf document.
Man you got some patience. At some point somebody needs to document all the columns in a short readable fashion.
Re: DaveBerry (Sep 1 11:34),
Here’s how the Russian data for Honiara gridcells is encoded:
SQZ- 1
SQA – 4
SF – 6
SQZ is the number of standard deviations that the value is from climatology and SQA is some sort of quality control flag that I haven’t figured out yet, as is SF.
In their base36, they use the 10 digits and 26 letters to make 36 digits base 36. I did the following brute force look-up table (I’m sure that there’s a simpler way but this works and it isn’t worth the effort trying to figure out a slicker way.)
To code SQZ, they divide the number of standard deviations by 0.5 (min of -8.5 sd and max of 8.5 sd), add 18 and round to get an integer. They convert this to “base36” alphanumeric. The following function recovers the std deviations from the alphanumeric:
In the case at hand, qcz(“1”) is -8.5 sd: appropriate for Russian Arctic data.
The corresponding conversion for SQA is given by:
This gives a value of 0.15. The manual says:
Hard to say what this means in the present context.
Like GISTEMP, some of these methods seem to have been developed for much smaller computers. Screening values by std deviation is a sensible precaution, the prevalence of extreme values should also prompt the keepers of the data set to question their programming of the Russian data.
How did you select the gridcells you used?
Steve – I explained Honiara in the post. I did a post last year on Hawaii gridcell.
From: International Comprehensive Ocean-Atmosphere Data Set (ICOADS): Release 2.5 Long Marine Reports/Fixed-length (LMR6/LMRF6) 15 March 2010 as linked above:
“However, if no mapping was defined by Table 14, the original 8-bit ascii character, or a translation of the original 8-bit ebcdic character into ascii, was stored according to rule d) above (ebcdic translation was accomplished using the reversible mapping described in ).
The statement about reversible mapping should be applied with caution.
At the foundation of ebcdic, there were two binary characters available to assign sign as + or -. Clearly, only one character is needed, so one can make a convention that 0 means positive. Countries or organisations can comply with this convention, but maybe some did not. Let’s assume all did and that the other mathematical portions of reverse mapping are ok.
Next, consider the surveying part of this statement using latitude as an example. The Equator is designated 0 degrees and the poles up to +90 deg at North Pole and – 90 deg at South Pole. Data entry in the southern hemisphere thus required many entries for the minus sign (the plus sign was no punch). Some spatial data users in countries like Australia altered the local rules in the early days of computing and decided that for them, all SH latitude data would be positive. This also simplified some calculations in geodesy and gave more intuitive mathematics.
Crew of a ship visiting Australia might not have known about, or declined to use, this convention. Therefore, theoretically, each SH ebcdic conversion should be checked to see what happened. A German ship visiting Honiara at 9 deg 23 min south would be a candidate for this type of “convention error”. Honiara became part of British Solomon Islands Protectorate in 1893 and so the main rules could be expected to come from Great Britain. But trade, commerce and warfare at times were strongly liked to Australia. Places close to the Equator would be especially susceptible to this convention error.
This is but one example of a global problem that could be difficult to correct by a global instruction in soft-lmr or similar. It is a problem of local convention, needing a case by case check.
It arises again with longitude, which can be expressed as Greenwich meridian = 0, other places up to + or – 180 degrees; alternatively, Greenwich = 0, all other places up to + 360 degrees. By convention, different countries or even different ships’ navigators might have adopted one of the conventions. The converter needs to know the convention in each case. It arises again for day dates, because the International Date Line does not follow the 180 degree meridian. Part of Fiji, for example, is East of 180 and part is West, but it is all at to the west if the IDL.
This type of “change the convention” complication is likely to occur in other applications, such as the number of points on a compass rose and is particulay a worry with “anomaly data” or deltas which are a shorthand way to represent data.
The erratic results shown in Steve’s vanilla graph one above have the appearance of errors of conventions. There are other explanations. I write this note as theoretical, not having checked to see what has already been documented as corrected and how.
Re: Geoff Sherrington (Sep 5 05:22),
It arises again with longitude, which can be expressed as Greenwich meridian = 0, other places up to + or – 180 degrees; alternatively, Greenwich = 0, all other places up to + 360 degrees. By convention, different countries or even different ships’ navigators might have adopted one of the conventions. The converter needs to know the convention in each case.
Not to mention the occasional habit of the French of measuring longitude from Paris, rather than Greenwich.
2 Trackbacks
[…] A First Look at ICOADS For quite a while, I’ve urged people interested in gridded temperatures to really look at the SST data – […] […]
[…] Ron Broberg Leave a Comment Steve McIntyre begins a tour of ICOADS sea surface temperature data: A First Look at ICOADS ICOADS – […]