Climate Audit

Over the weekend (before I picked up my “regular” files), I started looking at Steve Mosher’s use of raster and zoo – both of which intrigue me a great deal, but got intrigued by something else and ended up finally figuring out how to extract .Z files within an R script without having to handle them manually. (R has utilities for .zip and .gz files, but not the older .Z format.) This isn’t anything other than a nuisance with GHCN which only has one .Z file to worry about, but was a big problem with the very large ICOADS SST data where every month of data is in its own .Z file and manual processing isn’t an alternative. It’s further complicated since the 12 monthly .Z files for each year and packaged into an annual .tar file.

On many occasions, I’ve expressed regret at the disproportionate interest in land datasets relative to SST datasets, which are more important proportionately but about which there’s been negligible third party analysis. We’ve all raised our eyebrows a little at the bucket adjustments, but I, for one, hadn’t handled original data. For a start, the data sets were too big for the computer that I had a couple of years ago, but they are practical on my new computer (though, if I were doing much on this, I’d need to upgrade.)

A couple of years ago, CA reader Nicholas experimented with the extracting .Z files in R, contributing the package compress, which, unfortunately, I couldn’t get to work on the GHCN data set. I lost interest in the issue at the time, but the inability to handle .Z files automatically has been at the back of my mind for a while.

Mosh started one of his posts as follows:

In the last installment we started by downloading the v2.mean.Z file unzipping it and processing the raw data into an R object.

Mosh linked to another blog who had used the R command system to unzip a .gz file as follows:

# ———– UNCOMMENT ME TO DOWNLOAD HADSST2 ————-#
#if (! file.exists(hadsst2_data)) {
# hadsst2_gz <- paste(hadsst2_data, ".gz", sep="")
# download.file(hadsst2_data_url, hadsst2_gz)
# # windows users will need gunzip in their PATH
# system(paste("gunzip ",hadsst2_gz, sep=""))
#}
# ———————————————————-#

While browsing through some sites, I noticed the following comment at gzip.org

gunzip can decompress files created by gzip, compress or pack. The detection of the input format is automatic

.Z files are produced by compress. So maybe I thought that a simple expedient for extracting .Z files was staring me in the face. I’d gone through the process of installing a gunzip.exe commend on my old computer but hadn’t done it on my new computer and had to retrace my steps. I found a version of gzip here http://www.powerbasic.com/files/pub/tools/win32/gzip124xN.zip – there are other versions around. You have to unzip this file, which yields gzip.exe, but not gunzip.exe. I was stumped by this for a while. A webpage here explains:

Note that this archive contains only gzip.exe — to get gunzip.exe, you must copy gzip.exe to gunzip.exe (a silly Unix trick– don’t ask).

I uploaded a version of gunzip.exe to climateaudit.info to save others the labor. The following short script downloads the .Z file for GHCN and the gunzip.exe program to the working directory. Worked like a champ for me:

download.file(“ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v2/v2.mean.Z”,”v2mean.Z”,mode=”wb”)
download.file(“http://www.climateaudit.info/scripts/gzip/gunzip.exe&#8221;,”gunzip.exe”,mode=”wb”)
system(paste(‘”gunzip.exe”‘,’v2mean’ ))
working=readLines(“v2mean”,n=20)
working[1:3]
#[1] “1016035500001966-9999-9999-9999-9999-9999-9999-9999-9999-9999-9999 133 110”
#[2] “1016035500001967 100 112 129 143 181 193 239 255 225 201 166 107”
#[3] “1016035500001968 109 123 130 160 181 207 247 241 223 191 162 128”

This was handy for GHCN but is a nice to have, but not a need to have, whereas it’s essential for ICOADS. I checked it out on the .Z files extracted from the ICOADS .tar files and again it worked fine. The .Z files have to be in the working directory for the R system command to work.

It turned out that one could use tar.exe the same way as gunzip.exe. The structure of tar commands is described in a manual here. I located a version of tar.exe in a zip package at http://sourceforge.net/projects/unxutils/ and downloaded it (and uploaded it unzipped to http://www.climateaudit.info so it could be readily used.)

The following commands downloaded and extracted the contents of a tar file. (This is a small file – recent years get huge.)

loc=”ftp://ftp.ncdc.noaa.gov/pub/data/icoads/IMMA.1784.tar”
download.file(loc,”JMMA.tar”,mode=”wb”)
system(paste(‘”d:/climate/scripts/compression/tar.exe”‘,’-xvf’,'”JMMA.tar”‘ ))
#IMMA.1784.02.Z
#IMMA.1784.03.Z
#IMMA.1784.04.Z

Each of the monthly .Z files could than be unzipped. To do so, I found it convenient to rename each .Z file in turn to temp.Z and unzip, read an R-object and then remove this file, saving the read R-object (which takes up less space than the original Z file anyway.) The following read-instruction reads each of the .Z files and then saves the result by year.

index=list.files()[grep(“IMMA”,list.files())];index
K=length(index); n=nchar(index)
sst=rep(list(NA),K)
for(i in 1:K) {
file.rename(index[i],”temp.Z”)
system(paste(‘”gunzip.exe”‘,'”temp”‘))
working=readLines(“temp”,n=-1)
X=read.fwf2(working,widths=info$width,colClasses=info$class )
names(X)=info$id
sst[[i]]=X
file.remove(list.files()[grep(“temp”,list.files())])
}
save(sst,file=file.path(destdir, paste(“sst_”,year,”.tab”,sep=””) )

The above assumes that information about ICOADS formats has been extracted into a info file info. This is not a small job see here. I downloaded and extracted data from 1900 to 1980 over the last couple of days (tar files go from a few MB in 1900 to over a GB in 2006.) I got some odd results, which I’ll mention in another post, but will have to move on as looking through SST data is a big analytic job.