Lucia has been experimenting with model data downloaded from KNMI and I thought that I'd try to experiment with this a little from time to time.
Unlike Benjamin Santer, Ph.D. of the U.S. federal Lawrence Livermore Labs, who refused to provide any assistance whatever in providing the data used in Santer et al, Geert Oldenburgh, the KNMI scientist responsible for their public interface, is pleasant and cooperative though it seems that they aren't used to automated data inquiry of the type that I expect to do.
Their webpage is designed for manual retrieval, but it can be pinged so that data can be directly downloaded into R. KNMI had no objections to me trying to do this and Oldenburgh helped as much as he could. I've managed to write a neat little retrieval script that picks up ensemble combinations but I'm stumped as to how to retrieve individual runs. It looks like it should be possible; KNMI has no objection to my doing so; maybe one of the computer-oriented readers can solve this little puzzle.
Here's what I can do so far.
The access point are cgi commands in the URL that instruct the KNMI computer to compile averages. This works very quickly. Examples of these commands are:
climexp.knmi.nl/get_index.cgi?email=yourname@you&field=tas_bcc_cm1_20c3m&standardunits=true
or
climexp.knmi.nl/get_index.cgi?email=yourname@you&field=tas_bcc_cm1_20c3m&standardunits=true&lat1=-20&lat2=20&lon1=0&lon2=360
If you insert this sort of URL in your browser, it will ping the KNMI computer to produce ensemble temperature averages for the region specified by the lat-long (default - global.) The HTML page that is produced contains information on the location of the sought data file. The information can be grep'ed from the HTML page and a new URL can be constructed where the ensemble average is located. This file can be read fairly easily - there's an annoying comment at the end of the file that requires extra handling, but this is programmed easily enough. I've inserted the program below.
The ensemble averages is fine. Here's where I get stuck on reading individual runs. Go to climexp.knmi.nl and then ping Scenario Runs on the right frame under Select A Field. Now as an example, check the radio button under tas for BCC_CM1 and click Select Field.
If you click Make Time Series in the first panel, this will make the ensemble average, producing a new page. Above the third panel in the new page is a hyperlink "raw data" which contains a time series of anomaly data. This is what I can retrieve with the script below.
If you now go back to the previous page, you'll see hyperlinks at the bottom of the page which produce results from each of two different runs. If you click the hyperlink of "analyse ensemble member 1 separately", it will generate a webpage looking just like the one we just had, except this time if you click Make Time Series in the first panel, it produces the results of this run located in a webpage structured as before.
I presume that there is a cgi command generated by one of these links, but I haven;t been able to figure out the structure. Maybe one of the computer programmers who visits here can figure it out.
Here are my scripts. Please paste in a real email address in the appropriate spot below (requested by Oldenburgh) so that usage of the access tools can be reported.
##FUNCTIONS
read.ensemble=function(model,scenario,prefix=myprefix,region="GL",suffix= "&standardunits=true") {
if (region=="TRP") {suffix="&standardunits=true&lat1=-20&lat2=20&lon1=0&lon2=360";suffix2="0-360E_-20-20N"}
if (region=="NH") {suffix="&standardunits=true&lat1=0&lat2=90";suffix2="0-360E_0-90N"}
if (region=="SH") {suffix="&standardunits=true&lat1=-90&lat2=0";suffix2="0-360E_-90-0N"}
if (region=="arctic") {suffix="&standardunits=true&lat1=60&lat2=90";suffix2="0-360E_60-90N"}
if (region=="antarctic") {suffix="&standardunits=true&lat1=-90&lat2=60";suffix2="0-360E_-90-60N"}
if (region=="GL") {suffix="&standardunits=true";suffix2="0-360E_-90-90N"}
url=paste(paste(prefix,model,scenario,sep="_"),suffix,sep="")
#http://climexp.knmi.nl/get_index.cgi?email=yourname@you&field=tas_bcc_cm1_20c3m&standardunits=true
# "http://climexp.knmi.nl/get_index.cgi?email=yourname@you&field=tas_bcc_cm1_20c3m&standardunits=true&lat1=-20&lat2=20&lon1=0&lon2=360"
my_info=download_html(url)
grep("raw data",my_info)
y= my_info[grep("raw data",my_info)][3]
# "raw data, "
n=nchar(y)
loc=file.path("http://climexp.knmi.nl",substr(y,10,n-16))
Sys.sleep(2)
test=readLines(loc)
count=as.numeric(substr(test[1],19,26))+1
writeLines(substr(test[3:length(test)],1,20),"temp.dat")
test=read.table("temp.dat");
x=test[1,1];x
read.ensemble=ts(test[,2],start= c(floor(x),round( 12* (x%%1),0)+1),freq=12)
read.ensemble}
download_html =function(url) {
download.file(url, "temp.html");
html_handle < - file("temp.html", "rt");
html_data <- readLines(html_handle);
close(html_handle);
unlink("temp.html");
return(html_data);
}
Here is a sample use. This can easily be modified for bulk retrieval.
knmi.info=read.csv("http://www.climateaudit.org/data/models/knmi.info.csv",sep="\t")
scenario="20c3m"
temp=(knmi.info$scenario==scenario);sum(temp) #24
id=unique(knmi.info$alias[temp]);K=length(id) #24
email="yourname@you" #
myprefix=paste("http://climexp.knmi.nl/get_index.cgi?email=",email,"&field=tas",sep="")
i=1; test=read.ensemble(model=id[i],scenario,region="GL"));
| Category: Data, Modeling, Santer | Comments (32)
The Finnish "Supra-Long" Chronology #1:
Gavin and the Big Red Dog:
The Supra-Long Finnish Chronology #2:
Wordpress upgrade imminent:
Royal Meteorological Society Considers Data Archiving:
2008 Tropical Temperatures:
M&M Return:
Sea Ice - End of Game Analysis:
Briffa and Sodankyla Church:
Unthreaded #31:
Glenn McGregor: Data Archiving not required by the International Journal of Climatology: