You asked for e-mail addresses in response to this from the webmaster:

GISTEMP research group to explain your needs. E-mail addresses for the GISTEMP research group are located at the bottom of the page at GISTEMP Research Group

Not exactly true. There are links to their bio pages which do have their e-mail addressess. However, good luck with getting a response from them. I wish thee well.

Contacts
Please address scientific inquiries about the GISTEMP analysis to Dr. James Hansen. (jhansen@giss.nasa.gov)

Also participating in the GISTEMP analysis are Dr. Makiko Sato (makikosato@giss.nasa.gov) and Dr. Ken Lo (klo@giss.nasa.gov)

Dr. James E. Hansen : jhansen@giss.nasa.gov
Dr. Reto A. Ruedy : rruedy@giss.nasa.gov
Dr. Makiko Sato : makikosato@giss.nasa.gov
Dr. Kwok-Wai Ken Lo : klo@giss.nasa.gov

Contacts
Please address scientific inquiries about the GISTEMP analysis to
Dr. James Hansen.
Please address technical questions about these GISTEMP webpages to
Dr. Reto Ruedy.
Also participating in the GISTEMP analysis are Dr. Makiko Sato and Dr. Ken Lo.

How would you like me to get you the data set? Direct e-mail’s out since it’s too big, but perhaps we can switch to e-mail to coordinate an upload?

Let me know,

-rv

On March 5, soon after you posted the original entry, I successfully downloaded all three datasets for all stations using a variation of the process you described. I throttled my requests to a couple every few seconds so that I would not hammer the site (and hopefully wouldn’t get disconnected) and ended up taking 36 hours to grab everything. I figured it was only a matter of time before the multi-step process stopped working, or they started blocking people using your technique, given the obvious lengths that they went to to make downloading difficult.

I would be happy to send you the combined and compressed station data if you like. As a tar.bz2 file it’s a 26 MB archive containing ~27,000 separate files. I originally intended to combine and process it into one file with a WMO number column, but the day job interfered. If that would be helpful, I could try to get to it maybe this weekend.

-rv

source(“http://data.climateaudit.org/scripts/gridcell/read.giss.station.txt”)
idgiss=scan(“http://data.climateaudit.org/data/giss/idgiss.dat”);N=length(idgiss)
giss=rep(list(NA),N) #
##I got stopped at 1615
i=1614

K=i+1
for (i in K:N) {
station.giss< -download_giss_data(idgiss[i])
giss[[i]]=station.giss
names(giss)[i]=id[i]
}

Steve,

Although you did not provide any further details about
your problem, I will assume that you are the person on
the ***.com network who has been running a
robot for the past several hours trying to scrape
GISTEMP station data and who has made over 16000 (!)
requests to the data.giss.nasa.gov website.

Please note that the robots.txt file on that website
includes a list of directories which any legitimate
web robot is _forbidden_ from trying to index. That
list of off-limits directories includes the /work/
and /cgi-bin/ directories.

Because the robot running on the ***.com network
has rather obviously and blatantly violated those rules,
I placed a block on our server restricting its access
to the server.

If you are indeed the person who has been running that particular web robot, and if you do need access to some large amount of the GISTEMP station data for a scientific purpose, then you should contact the GISTEMP research group to explain your needs. E-mail addresses for the GISTEMP research group are located at the bottom of the page at http://data.giss.nasa.gov/gistemp/

rbs

I was not running a robot but an R-project. I’m blocked from the webpage in question so maybe someone can send me the email address.

Nicholas, I’ll email you with what I was running.

We might end up getting blocked too, eventually, but hopefully after we’re each finished 1/8th or 1/4 or some fraction of the sites.