Comments on: Another R-Script for a Zip File (NOAA)

By: Nicholas

Nicholas — Fri, 27 Apr 2007 09:17:28 +0000

MurrayK, it’s tough to rely on anything which isn’t a standard part of R, because as soon as you do, it makes it more difficult for people to download & run Steve’s scripts. I think it’s OK if he requires some commonly used and widely available third party R packages, but would be extremely hesitant to require people to install znew on their system, especially for Windows users, where there may be no tidy GUI for installing that program.

That is why I wrote the R uncompress routine, because it was the only solution I could think of that wouldn’t require users to jump through hoops to get the script to run. It works, but I don’t think most people have the patience to wait for 2-4 minutes while the data decompresses. It’d be even worse if the script had to deal with multiple .Z files. I’d much rather just recompress those files myself and upload them to my web server, and point the scripts there. Less hassles for all of us.

By: MurrayK

MurrayK — Fri, 27 Apr 2007 03:21:53 +0000

znew recompresses files from .Z (compress) format to .gz (gzip) format. There's an example here. Perhaps the R 'system' interface can call this utility first?

By: Nicholas

Nicholas — Fri, 27 Apr 2007 01:37:03 +0000

mhcoffin, you’re right, but the problem is while gzip may support uncompressing .Z files, the built-in R function gzfile does not work on .Z files. You can’t rely on there being gzip installed on the system, since R is often run in Windows, where there is no gzip command-line program.

I wrote an R implementation of a Z decompressor (it’s just dictionary-based so it wasn’t too hard) but it takes several minutes to decompress a 1MB file. There’s no way to I know avoid a for-loop since the compressed data is stored in variable length bit strings, and R is slow with for-loops.

The idea of renaming the file is interesting, but I don’t think gzfile cares about the extension. It probably just calls the deflate algorithm, which returns an error because the data is not compressed with deflate. The gzip program is probably smart enough to detect the extension and call the modified-LZ78 routine instead, but R’s built-in function doesn’t seem to have that capability.

By: mhcoffin

mhcoffin — Thu, 26 Apr 2007 16:50:44 +0000

Files with a .Z suffix are probably generated by “compress”. Compress was the standard Unix compression utility until Unisys, which owned a patent on Lempel-Ziv compression, began trying to enforce their patent. (That patent expired a few years ago, but by then gzip had become standard.)

The good news is the patent didn’t cover decompression, so gunzip will happily uncompress .Z files. If R supports gzip, it can probably be tricked into decompressing .Z files pretty easily. Maybe by merely renaming the file to .gz and pretending it’s gzipped.

By: Skip

Skip — Thu, 26 Apr 2007 15:57:47 +0000

The “wb” is likely just passed through to the c runtime library fopen() function. Likely values are:

“wb” – open for writing as a binary file
“w” – open for writing as a text file. This will change end of line characters to be appropriate for the system that it’s downloaded to.
“ab” – same as “wb” but appending instead of creating the file new
“a” – same as “w” but appending.

You need the “wb” if it’s a zip file because otherwise it will likely corrupt the file.