That is the link that I was attempting to find as I indicated in my email to you.

Are you referring to the following?

http://www.unur.com/climate/giss-1880-2006.html

Let me know here (if it is OK with Steve) if this is not it. You can also contact me by sending email to +nanis+ at unur dot com. (Remove the ‘+’ signs, and reverse to get my first name).

– Sinan

I have not been able to download a nc file directly to R, but only after downloading it locally to my computer and then loading it into R with normal prescribed commands – same for zipped nc files.

This is way OT, but since you are commenting here, Sinan Unur, I need help in finding the link to your code for using the gridded GISS temperature data.

I am not in a position to disagree, but your review was very helpful in allowing me to better understand the important issues involved in that part of the analysis. It is most appreciated by this blog reader when an “involved” poster/analyzer takes the time to review what has been covered.

Here is a list of some things I have learned about the dataset. Some are firmer than others and I have tried to express my uncertainty in the comments below. Please let me know if anyone disagrees or has any additions. I hope I’m not rehashing things that are obvious; the intent is to put what I think I know in one place.

1) The files are 0200 and 1400 hours. I originally though these were UTC, but the text says they are local times. This appears to mean that satellite passes from different times over different areas are patched together to form a “night” composite and a “day” composite (relatively speaking).
2) The monthly means in this dataset was compiled using CASPR, documentation for CASPR can be found here and here. For this dataset, CASPR used the daily 5 km data parsed to every 5th point to create the 25 km data. Dr. Jeffrey Key of U Wisconsin is the author of CASPR and appears to have compiled this dataset.
3) An extremely detailed description of the daily 5 km x 5 km dataset can be found here here. This page has a good description of the data, its shortcomings and known flaws, missing days and processing methodology.
4) CASPR has multiple cloud-masking options for determining cloud cover. I can’t find a specific description of what setting was used for this dataset, but the text seems to indicate is was something of a “default” setting. The link in #3 has a good description of CASPR’s cloud masking methods.
5) These data files are “all sky” (as opposed to “clear sky”). In all sky mode, CASPR uses an interpolation algorithm to infill areas it thinks are cloud-covered. It uses information from adjacent clear sky areas for the infilling. The author states that this infilling can have large errors as the clear sky areas can be at a considerable distance. They don’t quantify the error magnitude.
6) From reading Comiso 2000, it’s not clear if Dr. Comiso used CASPR. He does seem to indicate that he discarded cloud covered areas. This could be akin to the clear sky mode of CASPR as opposed to the all sky mode used for this data set.
7) In the SI, Dr. Steig describes the use of a daily filter where values that exceed +/-10 deg C of the climatological mean are considered cloud covered and discarded. When asked at RC, Gavin stated that “climatological mean” meant the monthly average over the entire satellite record. Since this dataset is already in monthly means, we obviously can’t apply a daily filter as Steig describes in the SI.
8) The dataset contains a data grid called “cloudtype”. This value ranges from 0-99 and describes the percentage of time each cell experienced cloud cover during a given month. I’m working with this dataset to see if there are any obvious spatial/temporal patterns that might explain discrepancies with the Comiso dataset.

I’ll put anything else up that I find. Some of the points above are speculative based on my reading of the documentation. Please let me know if anyone disagrees.

http://cran.r-project.org/doc/manuals/R-data.pdf

5.1 Binary data formats

Packages hdf5, RNetCDF and ncdf on CRAN provide interfaces to NASA’s HDF5 ( Hierarchical Data Format, see http://hdf.ncsa.uiuc.edu/HDF5/ ) and to UCAR’s netCDF data files (network Common Data Form, see http://www.unidata.ucar.edu/packages/netcdf/ ). Both of these are systems to store scientific data in array-oriented ways, including descriptions, labels, formats, units, … HDF5 also allows groups of arrays, and the R interface maps lists to HDF5 groups, and can write numeric and character vectors and matrices. Package ncvar on CRAN provides a higher-level R interface to netCDF data files via RNetCDF.

There is also a package rhdf5 available from http://www.bioconductor.org.

gzip is a standard format.

— Sinan

It is now pretty clear to me how the data is structured and what is in the file. For example, there are 25 pieces of data of “vectors” or “matrices”. Each of them comes with an annotation, information about the type of numbers in the vectors, their length, and the data themselves.

See this PDF printout of a Mathematica notebook for a self-explanatory treatment of the structure of the file:

http://lumajs.googlepages.com/cdf-import.pdf

And please, don’t panic about GZ – it’s a standard triviality and if one decoder has problems, others surely don’t.