Comments on: Tar and Z

By: Steve McIntyre

Steve McIntyre — Fri, 24 Sep 2010 21:57:26 +0000

They’ve removed the data from the site. It is now at http://dss.ucar.edu/datasets/ds540.0/ behind a well of Java access screens that are hard to penetrate with R.

By: Pablo

Pablo — Fri, 24 Sep 2010 20:02:49 +0000

This data is no longer available on the ftp site you ( ftp://ftp.ncdc.noaa.gov/pub/data/icoads/ ). Is it available somewhere else?

By: Handling .Z files « Steven Mosher's Blog

Handling .Z files « Steven Mosher's Blog — Mon, 13 Sep 2010 06:31:59 +0000

[...] Handling .Z files September 13, 2010 Steven Mosher Leave a comment Go to comments A while back Steve Mcintyre was looking for a way to handle .Z files in R [...]

By: drj11

drj11 — Fri, 10 Sep 2010 15:39:25 +0000

Have you tried asking the ICOADS folks to provide the files in .gz?

When ccc-gistemp had this problem (decompressing a .Z file in Windows), I read the C code for the OS X version of compress and reimplemented the same algorithm in Python. Such fun.

Steve: I asked one climate group to provide in.gz, but they didn’t do anything. The method set out here works fine – so I’ve worked around the problem.

BTW I’d emulated quite a long way through GISTEMP steps in R in 2007 and placed the code online. Their so-called UHI adjustment seemed totally ineffective – as it effectively presumed that other inhomogeneities had already been resolved.

By: Jane Coles

Jane Coles — Wed, 08 Sep 2010 10:52:43 +0000

Code that relies on remote data really needs to calculate a hash on that data and record it along with other (more interesting) results. The remote data might change. If someone downloads the code (a year after it was published, say) and gets different results then, absent the hash, it may not be obvious that this is due to a change in the remote data. The md5deep utility provides the relevant functionality and is freely available for all operating systems.

By: johnl

johnl — Fri, 03 Sep 2010 22:00:19 +0000

This might be a time to consider writing the R script to work from a local copy of the data, and to preprocess the data beforehand into .gz (rather than .Z) files with a .bat or .sh script.

By: Ron Broberg

Ron Broberg — Mon, 30 Aug 2010 23:45:49 +0000

Mosh linked to another blog who had used the R command system to unzip a .gz file as follows:

You’re welcome. I’m glad it was of some use to you. I’ve learned much from your R code as well.

Ron Broberg
http://rhinohide.wordpress.com

By: Emil

Emil — Mon, 30 Aug 2010 21:50:54 +0000

http://stat.ethz.ch/R-manual/R-patched/library/base/html/system.html

By: Chad

Chad — Mon, 30 Aug 2010 21:32:03 +0000

Steve,

I think the core of the problem is that you’re running a Windows port of R. R was originally developed in Linux and decompressing files within R is easy because the system() command can call binaries that are already present by default on any Linux system. But you have what you need installed so problem solved.

By: Emil

Emil — Mon, 30 Aug 2010 21:25:41 +0000

“gunzip file_name |tar xvf”

if the pipe is the problem, try “tar xzvf file_name”

Steve: as I said before, the Unix commands were not at issue. It was the R interface. No more discussions of Unix commands please.