Pedantically, I would also tend to avoid having “data” and “Data” around at the same time meaning different things, and stylistically either “<-” or “=” should be used throughout, so perhaps as a model the code could be

handle <- url(“ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/station.inventory.Z”, “rb”)
raw_data <- readBin(handle, “raw”, 9999999)
close(handle)
uncomp_data <- uncompress(raw_data)
Data <- strsplit(rawToChar(uncomp_data), “\n”)
Data <- unlist(Data)

More seriously, I do not seem to get the whole NOAA file into Data, as it breaks at records 143 or 458 depending how I run the code. Copying the file to my hard disk first, it breaks at record 1222.

Anyway the bigger problem with calling external programs, besides inconvenience and performance, is that we (and specifically Mr. McIntyre, I think) want these scripts to be reproducible. If you’re executing an external program it requires the user to have that program installed in the same place as you do, and then you have to deal with the case where the program may be executing on different operating systems.

However, with this R package, all a user has to do is install it via CRAN, whether they are using Linux, Windows, MacOS, or some other Unix variant. It should just work. If it doesn’t hopefully they will let me know and I can fix it.

If there’s another processing step required for external data in future which is hard to perform in R then I can write another package to deal with it. Hopefully we can reach a point where practically all climate data is easily imported into R for processing. We may already be close to that point.

It turns out that you can run DOS commands out of R by using the system() command.

Sure, but just try and call a Cygwin executable using system() from Microsoft C++ .NET…and passing it a filename, including path, as an argument. Recalling, of course, that DOS wants to use backslash in path descriptors, but Cygwin treats backslash as an escape.

Observe:

> handle <- gzfile(“IMMA.1784.02.Z”, “rb”);
> data <- readBin(handle, “raw”, 9999999999);
> close(handle);
> rawToChar(data)
[1] “37\235\2201n\340\24001B\206\f\202…
> library(“uncompress”);
> rawToChar(uncompress(data));
[1] “1784 224 3913 28782 02 4 10EMPRE*_C 1 23…
>

I think what is happening is the commandline program “gzip” can handle .Z data but not via zlib – it uses separate code – and gzfile() uses zlib. However that’s just a guess. I’m pretty sure I tried this before I wrote the uncompress package since I was trying to find the easiest way to solve this problem

The Mac OSX version of my package is available now too. I believe that means “uncompress” will run on any platform that R will, unless I’ve made a mistake with Unix portability. I haven’t tested it on 64 bit, but theoretically it should work.

The packages are here:

http://cran.r-project.org/web/packages/uncompress/index.html

I’ll check again if gzfile will open .Z files – as I said I don’t think so – but I’ll report on the result.

Re-attempting to show the code that got cut off:

Following your example, is something like the following possible without requiring the addition of a package?

…
handle <- url(“ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/station.inventory.Z”, “rb”)
uncomp_data <- gzfile(handle)
close(handle)
Data <- strsplit(rawToChar(uncomp_data), “\n”)
…

Steve: gzfile does gz files, not Z files. That’s why the package ws necessary. My guess is that Nicholas’ function will ultimately be added into the base R along with gzfile and such.

Following your example, is something like the following possible without requiring the addition of a package?

…
handle

It’s possible to add locations to the path via, I believe, Control Panel’s “System” panel. However personally I’m lazy and I usually just copy such files into C:\WINNT\SYSTEM32 (or C:\WINDOWS\SYSTEM32 as appropriate). Then you can run them from anywhere.

Of course R has gzfile() so in this case it isn’t strictly necessary, but there may be other times you need to execute an external program. Then again, now that I know how to make R packages, it should be less necessary

Steve: I was just using gzip to experiment with the system() command on something that worked (my interest then being in the compress.exe command), not because I was having trouble with gz files.

The private sector runs rings around most other institutions.