Climate Audit

There is a vast amount of available station data. Versions have usually been adjusted and care has to be taken to be sure exactly which version has been used. Categories of information that I’ve tried to link here include both data and metadata. This is for orientation only. This does not include information on SST estimates or gridcell estimates.

Anthony Watts/ Surface Stations

http://www.surfacestations.org

John V http://www.opentemp.org/_results/_20070919_RuralCRN12_Revised.zip

Standards

http://www.campbellsci.com/documents/apnotes/siting.pdf

http://www.wmo.int/pages/prog/www/IMOP/meetings/Surface/ET-STMT1_Geneva2004/Doc4.3(2).pdf

http://www1.ncdc.noaa.gov/pub/data/uscrn/documentation/program/X030FullDocumentD0.pdf (CRN)

Station Data in the Major Gridcell Composites
There are three major global indices of temperatures that incorporate station data: CRU, GISS and NOAA. Each of these groups primarily relies on the GHCN (Global Historical Climatology Network) for their input data. GHCN has two versions: 1 and 2. Each version contains max, min, mean and adjusted mean. A large proportion of the GHCN network is composed of the USHCN (US Historical Climatology Network). The USHCN network has two versions: 1 and 2 (not yet released), which do not coincide with GHCN versions. USHCN version 1 has raw, time-of-observation adjusted, adjusted and urban adjusted variations. Daily information is available for a subset of GHCN. Identification numbers are not consistent between USHCN and GHCN (and elsewhere). I know of no official concordance of USHCN and GHCN identifications and have archived my own.

GHCN Monthly see also GHCN Daily and USHCN
Readme http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php . See also http://www.ncdc.noaa.gov/oa/climate/research/ghcn/ghcngrid.html
Directory ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2
An inventory of 7280 stations together with particulars such as lat, long, altitude, population etc,: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.temperature.inv

Look at the directory for a list of available versions. There is a medium-sized zipped data file that is updated all the time (my present download – June 20, 2007) ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z

GHCN carries out their own adjustments: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean_adj.Z – also updated all the time. About 25% smaller in size than the raw data.

GHCN has 12 identification numbers: the first 3 are country code, the next 5 the nearby WMO station and the next 3 a station identified. All 11 digits are needed to identify a station. This 11 digit code links back to the Station Inventory File. The 12th digit identifies the “duplicate number” since GHCN raw archives versions that are scribally distinct.

There is a mirror at KNMI: http://climexp.knmi.nl/selectstation.cgi?someone@somewhere (I haven’t verified that versions correspond yet.)

Academic references:
http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php http://www.ncdc.noaa.gov/oa/climate/ratpac/index.php?name=ratpac-a Holder et al: CCSP chapter 3
ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/zipd

GHCN Version 1 (1992)
Directory http://cdiac.ornl.gov/ftp/ndp041/
Station List http://cdiac.ornl.gov/ftp/ndp041/temp.statinv
Data http://cdiac.ornl.gov/ftp/ndp041/temp.data.Z

GSOD
Stations ftp://ftp.ncdc.noaa.gov/pub/data/gsod/ish-history.txt

CRU
Station list only – my collation http://data.climateaudit.org/data/station/cru/cru.info.dat

GISS
Readme http://data.giss.nasa.gov/gistemp/

Station Lists: There are two station lists at GISS. The list http://data.giss.nasa.gov/gistemp/station_data/v2.temperature.inv.txt has 7364 stations and is the more comprehensive. The 7280 GHCN stations are all present; the others are nearly all from Antarctica plus 2 Southern Ocean stations. The other list http://data.giss.nasa.gov/gistemp/station_data/station_list.txt are the stations “as used”; I have not located an explanation of the reason for the subsetting.

Data: There are 3 GISS datasets: 0: raw; 1- combined; 2- adjusted.
dset=0 contains multiple versions and for the most part, this information seems to match GHCN versions, including the duplicate number. As noted above, different versions of one station may only have scribal variations.
dset=1 is the “combined” time series at one station. Very occasionally, there are two dset=1 versions for one station. The form of “combining” records is very idiosyncratic, involving pervasive adjustments; there is a lengthy discussion at climateaudit on the topic. For USHCN stations, the “raw” GISS version is generally similar to the USHCN adjusted (FILNET) version.
dset=2 – “adjusted” using the GISS adjustment, which supposedly adjusts the trend of urban stations to the trend of rural stations within 1000 km.

There is no comprehensive version of the data. Each individual time series can be scraped from the GISS website
http://data.giss.nasa.gov/gistemp/station_data/ . It is VERY time-consuming to scrape the entire data set: I’ve done so on a high-speed cable connection and the downloading of 16 (net) MB of dset=0 data took nearly 24 hours.

I’ve posted up my collations of scraped data as follows:
dset0: http://data.climateaudit.org/data/station/giss/giss.dsete0.tab

References: http://data.giss.nasa.gov/gistemp/references.html http://pubs.giss.nasa.gov/

USHCN
USHCN: CDIAC Version
- this is not as updated as NOAA
Prefix http://cdiac.ornl.gov/ftp/ushcn_monthly/
Readme http://cdiac.ornl.gov/ftp/ushcn_monthly/ushcn_monthly_doc.html
Station inventory http://cdiac.ornl.gov/ftp/ushcn_monthly/station_inventory
Note: http://cdiac.ornl.gov/ftp/ushcn_monthly/hcn_doe_mean_data.Z (20 MB archived May 10, 2005; up to Dec 2002). Includes three versions for individual stations: Areal Edited (Raw), TOBS (Time of Observation adjusted), and FILNET (adjusted). Max and min are also available.Downloaded MAy 25, 2007
Calculated mean data hcn_calc_mean_data.Z (12 MB archived May 12, 2005 at CDIAC; different version at NOAAto Dec 2002) Areal Edited, TOBS and FILNET Tmean data calculated from max and min versions ( hcn_doe_max_data.Z and hcn_doe_min_data.Z). So far I’ve not compared this to the hcn_doe_mean version.
Urban heat-adjusted data http://cdiac.ornl.gov/ftp/ushcn_monthly/urban_mean_fahr.Z (archived – 5.2 MB Feb 2007) I haven’t examined this data set yet or the UHI adjustment yet.
Daily http://cdiac.ornl.gov/ftp/ndp070/ http://cdiac.ornl.gov/ftp/ndp070/invent.txt stations http://cdiac.ornl.gov/ftp/ndp070/ istory http://cdiac.ornl.gov/ftp/ndp070/history.txt
Station history http://cdiac.ornl.gov/ftp/ushcn_monthly/station_history . Further details at http://mi3.ncdc.noaa.gov/mi3qry/login.cfm
Landuse metadata http://cdiac.ornl.gov/ftp/ushcn_monthly/station_landuse
Population metadata http://cdiac.ornl.gov/ftp/ushcn_monthly/metrof_orig
Old version (Revision 3) http://cdiac.ornl.gov/r3d/ushcn/ushcn.htm

USHCN: NOAA Version
Readme http://www.ncdc.noaa.gov/oa/climate/research/ushcn/
Directory http://www1.ncdc.noaa.gov/pub/data/ushcn/ updated Oct 11, 2007.
Prefix ftp://ftp.ncdc.noaa.gov/pub/data/ushcn http://www1.ncdc.noaa.gov/pub/data/ushcn/
Monthly mean data: ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/hcn_doe_mean_data.Z dated March 1, 2007 is different than the version with the same name at CDIAC and goes up to late 2006. Includes three versions for individual stations: Areal Edited (Raw), TOBS (Time of Observation adjusted), and FILNET (adjusted). Max and min are also available. Urban http://www1.ncdc.noaa.gov/pub/data/ushcn/urban_mean_fahr.Z
Downloaded June 13, 2007.
Versions at NOAA have same nomenclature as CDIAC.
History (archived 1995) http://www1.ncdc.noaa.gov/pub/data/ushcn/station.history.Z

USHCN:Old Version
Readme http://www.ncdc.noaa.gov/oa/climate/research/ushcn/ushcn.html

USHCN:Daily Version
Stations: http://www1.ncdc.noaa.gov/pub/data/ushcn/daily/invent.txt

GHCN Daily
Directory ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily
Three very large zipped files:
all: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd_all.tar.gz 1.0 GB
gsn: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd_gsn.tar.gz 59 MB
hcn: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd_hcn.tar.gz 153MB
These three files appear to bundle the contents of 3 large subdirectories:
all: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/all
gsn: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/gsn
hcn: ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/hcn
The file structure is: 10500065516.dly etc using GHCND identification
hcn: 42500455946.dly 890 KB 8/2/2007 1:38:00 PM
Station Lists:

http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt

Station history versions not necessarily consistent. For example information on Aberdeen WA is at cdiac, but not MI3.

GHCN Daily:OA Version
Readme http://www.ncdc.noaa.gov/oa/climate/ghcn-daily/index.php
Daily directory ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily
all: Directory with “.dly” files for all of GHCND
gsn: Directory with “.dly” files for the GCOS Surface Network
hcn: Directory with “.dly” files for U.S. HCN
Daily Station Inventory: http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
Raw Data http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/hcn/
GCOS stations ghcnd-gsn.tar.gz
USHCN ghcnd-hcn.tar.gz
All ghcnd-all.tar.gz
Academic references: http://www.ncdc.noaa.gov/oa/climate/ghcn-monthly/index.php

NOAA Climvis
Station list: http://www1.ncdc.noaa.gov/pub/data/climvis/ghcn/v2.comp.srt

GSN (part of GCOS)

GSN = GCOS Surface Network
About 1100 stations, mainly airports. This is very up-to-date. GCOS also has upper-air measurements. GCOS is described in Peterson et al 1997 as a network of “good” stations.
Directory http://www1.ncdc.noaa.gov/pub/data/gcos/
Readme http://www.ncdc.noaa.gov/oa/hofn/global-insitu.html See also http://www.wmo.ch/pages/prog/gcos/
Station List http://www.ncdc.noaa.gov/hofngsn/HOFNGsnStn See also http://cdo.ncdc.noaa.gov/cdo/gsnmonstn.txt ;
Also see info at http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt with collated IDs
List of 1086 station numbers http://www1.ncdc.noaa.gov/pub/data/gcos/GCOS_GSN_STN_LIST
Station ranges http://www1.ncdc.noaa.gov/pub/data/gcos/COUNTRY_XREF_LIST11302006.prn
Summary of availabilities in 7 regions: http://www1.ncdc.noaa.gov/pub/data/gcos/WW_REG7_POR_summary etc
Webpage http://gosic.org/ios/GCOS-main-page.htm
Historical http://cdo.ncdc.noaa.gov/pls/plclimprod/cdomain.DS3500

GSN: Ron Ray Version
Directory: ftp://ftp.ncdc.noaa.gov/pub/data/gsn
Daily Data: ftp.ncdc.noaa.gov/pub/data/gsn/gsndy.zip (Very large; over 80 MB zipped;’ over 500 MB unzipped)
I’ve extracted info from the gsndy zip file and it is generally not up to date. This appears to be the historical collection.
To red in R from zip file, use: data_handle < – unz(url, "\\project\\gsn\\data\\gsndy.dat", "rb");
Raw Data Directory: ftp://ftp.ncdc.noaa.gov/pub/data/gsn/rawdata (By country)
Reformated Directory: ftp://ftp.ncdc.noaa.gov/pub/data/gsn/reformated In files like /reformated/07130.dly.Z

GSN: GHCND Version
I haven’t determined yet what differences exist between Ron Ray and GHCN Versions.
Station Data Directory: http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/gsn/
File nomenclature: 22200029612.dly ccciiiwwwww : country code; id; 5 -digit wmo number
For US stations, they use the wmo number and not the GSN number. A (partial) concordance of GSN and WMO numbers is
at http://www1.ncdc.noaa.gov/pub/data/ghcn/daily/ghcnd-stations.txt
Readme ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/readme.txt