My new script for scraping KNMI model makes it very convenient to look at model data without a lot of setup overhead. Up till now, I’d only downloaded air temperature data (tas) and I tested downloading SST data (tos). KNMI’s collection of tos data is unfortunately quite spotty and this information is not consistently available. I don’t know whether this is incompleteness on their part or at PCMDI (from which they derive their data) or on the part of the contributors.
For example the cccma_cgcm3_1 model has 5 20c3m runs with tas (air) values but only one with tos (SST) values. Here’s a plot of the difference between tas (masked to sea only) and tos for the overlapping run. For some reason, the model air temperature over the ocean has increased relative to the SST during the 20th century.
Figure 1. Difference between TAS and TOS. Dotted red are major volcanos.
Climate Audit 
26 Comments
Here’s another example of how my new toy works for downloading model information:
source(“http://data.climateaudit.org/scripts/utilities.txt”)
source(“http://data.climateaudit.org/scripts/models/collation.knmi.functions.txt”)
#load data
download.file(“http://data.climateaudit.org/data/models/knmi/ensemble.a1b.tab”,”temp.dat”,mode=”wb”)
load(“temp.dat”)
#this is a R-list of 24 items (24 models), each item in the list being a time series of the model runs varying from 1 to 7
sapply(ensemble.a1b, function(A) A[2100-tsp(A)[1]+1,])
#input tas and tos example
model=”cccma_cgcm3_1″
tas= read.knmi.models(field=”tas”,model=”cccma_cgcm3_1″,scenario=”20c3m”,landtype=”sea”,
lat=c(-20,20), long=c(0,360), version=”anomaly”) ;dim(tas) #1813 5
tos= read.knmi.models(field=”tos”,model=”cccma_cgcm3_1″,scenario=”20c3m”,
lat=c(-20,20), long=c(0,360), version=”anomaly”) ;dim(tos) # 1812 1
tas=window(tas,end=2000.99)
c(cor(tas,tos))
#[1][1] 0.9992076 0.8941663 0.9153710 0.9135053 0.8916370
#GDD(file=”d:/climate/images/2009/models/tas_vs_tos.gif”,type=”gif”,h=320,w=480)
par(mar=c(3,4,2,1))
plot(c(time(tas)),tas[,1]-tos,type=”l”,main=paste(model,”: TAS (Sea) – TOS”) ,xlab=””,ylab=””,col=”grey80″)
smoothf=function(x) filter.combine.pad(x,truncated.gauss.weights(51))[,2]
lines(c(time(tas)),smoothf( tas[,1]-tos),col=2,lwd=2)
abline(v=c(1883,1902.15,1963,1982.378,1991.45),col=2,lty=3)
grid();abline(h=0,lty=2)
# dev.off()
I see TOS is also called SST here. There’s also ‘ts’ or skin temperature which over sea I guess is also SST, but they’re different. I think TOS is the marine air temperature. Also, try differencing ‘tos’ with the other runs. KNMI for at least one model (NCAR CCSM 3.0) doesn’t pair the runs properly. Here’s what you find on PCMDI for that model: 20C3M – runs 1,2,3,4,5,6,7,9 SRESA1B – runs 1,2,3,5,6,7,9. KNMI doesn’t have all the runs and since the meta data isn’t included, you don’t know which runs are paired with which.
Quick question: I notice in your script, you’re grabbing the data only in the tropics. Why is that? What do we get out of studying the tropics rather than the globe?
Re: Chad (#2),
I’ve been working on the Tropics in connection with Santer and because of the tropical tropo controversy.
In addition, the biggest difference between model sensitivities is said to arise out of the handling of tropical clouds, so I was familiarizing myself with this portion of the data.
One thing you need to be aware of – at least for CCSM3 and PCM, “tos” is the temperature at the midpoint of the first ocean model layer, not the temperature of the water surface (z=0). That could have an impact.
Re: Gary Strand (#3),
Do you know why a measurable discrepancy will build up?
Re: Steve McIntyre (#5), perhaps because it takes longer for the SST to warm given the relative heat capacities of water and air. Also, for CCSM3 and PCM, I had to interpolate the data from its original displaced-pole grid to a rectangular grid (as required by CMIP3 requirements) so there’s a potential for differences from that.
Re: Gary Strand (#7),
yes, in a world where air warms the ocean. Earth is not that world!
Steve
Here are some thoughts, as time goes by and as I write more and more code I remember
less and less about the details and need help of commentary. Additionally as I discover errors
and improvements I need versions. I try to not be cheap and save bytes of memory nor do I want to be too verbose as it is then a weight that needs to be pulled forward for the life of the program.
##########################################################################
#—overview of downloader and plotter Steve McIntire 14 June 2009
#My new script for scraping KNMI model makes it very convenient to look at model data without a #lot of setup overhead. Up till now, I’d only downloaded air temperature data (tas) and I tested #downloading SST data (tos). KNMI’s collection of tos data is unfortunately quite spotty and this #information is not consistently available. I don’t know whether this is incompleteness on their part #or at PCMDI (from which they derive their data) or on the part of the contributors.
#For example the cccma_cgcm3_1 model has 5 20c3m runs with tas (air) values but only one with #tos (SST) values. Here’s a plot of the difference between tas (masked to sea only) and tos for the #overlapping run. For some reason, the model air temperature over the ocean has increased relative #to the SST during the 20th century.
############
#—Downloader and Plotter Version 1.0 Steve McIntire 14 June 2009
#–revision 1.01 TrBixler add commentary (I am not too smart about your stuff so my
# commentary is much weaker than you would do, but as a programmer your commentary should
# help me understand what is happening before my eyes)
#–Set up source driver utilities and functions (these require versions as well)
source(“http://data.climateaudit.org/scripts/utilities.txt”)
source(“http://data.climateaudit.org/scripts/models/collation.knmi.functions.txt”)
#load data (although the data is implied from the URL there is no version)
download.file(“http://data.climateaudit.org/data/models/knmi/ensemble.a1b.tab”,”temp.dat”,mode=”wb”)
load(“temp.dat”)
#this is a R-list of 24 items (24 models), each item in the list being a time series of the model runs varying from 1 to 7
sapply(ensemble.a1b, function(A) A[2100-tsp(A)[1]+1,])
#input tas and tos example
model=”cccma_cgcm3_1″
#–generate the tas dataset
tas= read.knmi.models(field=”tas”,model=”cccma_cgcm3_1″,scenario=”20c3m”,landtype=”sea”,
lat=c(-20,20), long=c(0,360), version=”anomaly”) ;dim(tas) #1813 5
#–generate the tos dataset
tos= read.knmi.models(field=”tos”,model=”cccma_cgcm3_1″,scenario=”20c3m”,
lat=c(-20,20), long=c(0,360), version=”anomaly”) ;dim(tos) # 1812 1
tas=window(tas,end=2000.99)
c(cor(tas,tos))
#[1][1] 0.9992076 0.8941663 0.9153710 0.9135053 0.8916370
#GDD(file=”d:/climate/images/2009/models/tas_vs_tos.gif”,type=”gif”,h=320,w=480)
par(mar=c(3,4,2,1))
#–gauss smooth the data then plot
plot(c(time(tas)),tas[,1]-tos,type=”l”,main=paste(model,”: TAS (Sea) – TOS”) ,xlab=””,ylab=””,col=”grey80″)
smoothf=function(x) filter.combine.pad(x,truncated.gauss.weights(51))[,2]
lines(c(time(tas)),smoothf( tas[,1]-tos),col=2,lwd=2)
#–put some dates on the x axis??
#–There is no date time or version in the output or authorship
abline(v=c(1883,1902.15,1963,1982.378,1991.45),col=2,lty=3)
grid();abline(h=0,lty=2)
# dev.off()
#################################################################
One other thing – there are two surface air temperature fields in the CMIP3 archive – “tas” (2m temp) and “ts” (“skin” temperature). You may want to compare “tos” to “ts”.
Re: Gary Strand (#8),
KNMI doesn’t carry ts. And I haven’t figured out how to access data within PCMDI other than downloading terabytes. If you’ve got a few minutes, could you write down an access template for obtaining the ts information for cccma_cgcm3_1 20c3m run 1. I can probably figure things out if I ever got started.
I’d like to be able to scrape data within programs at some point.
PCMDI drives me nuts because of their gigabyte-sized output. My connection here is far too slow to access any of their data, so I have to steal it on hotel email when I travel. There is no way that I can find to download summaries or subsets. I’ve been working on the temperature of the atmosphere for a paper I’m writing. The HadAT2 balloon dataset gives zonal averages, the whole dataset covering back to 1958 is about 100K or something on that order. It’s frustrating (and often impossible for me) to have to download a half a gigabyte just to get 100K worth of data …
w.
Re: Willis Eschenbach (#10),
You can access PCMDI through opendap and grab a subset without having to download the whole file. I was planning on constructing synthetic MSU temps for the period left out from Santer et al (2000-present), but I just couldn’t store all the data (~100 GB), so that project is on hold until I figure out how to construct urls for Matlab to communicate with PCMDI.
More suffering for the Pacific Islanders from climate researchers.
jeez, yes, climate change is hard on us Pacific Islanders, but at least the island isn’t sinking …
Chad, if I knew what opendap was, I’d do that … I’ll do some research to see if I can figger it out. Any clues would be most helpful.
w.
Re: Willis Eschenbach (#13),
OPeNDAP is just a protocol for retrieving data on servers. What programs do you use for your analysis?
Chad, I use Excel for the quick’n dirty and R for the heavy lifting.
w.
Re: Willis Eschenbach (#15),
Willis, I looked quickly at the OpenDap site. They have a Matlab Plug-in, but as far as I can tell, no plug in for R or Excel. http://opendap.org/download/ml-structs.html
From my own experiments, if someone could provide a template for how a browser shakes hand with any source, I could probably create commands in R for some retrievals that are relevant to us.
PErhaps Chad or someone else can provide some actual URLs and CGIs for a working retrieve.
Re: Steve McIntyre (#16),
Here is how the HTTP request / response (hand shake) works with a working example:
On a raw socket connection to port 80 of the web server the browser sends the following “GET” request:
GET /{web page URL with parameters} HTTP/1.0{CR/LF}Host:{the IP address of your computer}{CR/LF}Connection: Close{CR/LF}{CR/LF}
For example, to request storm data from NOAA’s web server, send the following HTTP request:
GET /www.nws.noaa.gov/view/national.php?prod=HUR HTTP/1.0{CR/LF}Host: 10.10.1.10{CR/LF}Connection: Close{CR/LF}{CR/LF}
In the above example, the host address of 10.10.1.10 is my computer NIC’s IP address. You would need to substitute your computer’s IP address here. Notice also that the question mark “?” tells the web server that the following text are the parameter assignments. In this case, “prod” is the parameter name and “HUR” is the parameter value. If there is more than one parameter, each parameter will need to be separated with a “&”. For example, ?prod=HUR&time=1045.
I’m representing a carriage return (ASCII 13) and Linefeed (ASCII 10) as {CR/LF). At the end of the GET request, you send two sequences of a carriage return and line feed to tell the web server that the get request is completed. If the resource requested is available, the response data will be returned to you in whatever format the CGI application formats it in. Once the response has been fully sent, the web server then forces the connection closed (dropping the connection). To send another request, you must repeat the above sequence, substituting the web page URL and parameters as needed.
If you use Putty as a test tool, you can manually interact with the web server and save the response data to file. Here is the full set of configuration settings to make a working GET request from NOAA using Putty, saving the entire session dialog (with response data) to file for your analysis:
1) Using a text editor, copy and paste the above example into the editor. Where I use a {CR/LF}, make that a new line in the editor. For example:
———–[Editor Text Window]————
1: GET /www.nws.noaa.gov/view/national.php?prod=HUR HTTP/1.0
2: Host: 10.10.1.10
3: Connection: Close
4:
5:
6:
——————————————
Note: Make sure lines 4 through 6 exist because they represent the sequence of two carriage return and line feeds terminating the request.
2) Now highlight lines 1 through 5 above and copy them to your clipboard
3) Run the Putty application
4) Under the Session option tree, set the following options:
* Set the Host Name to http://www.nws.noaa.gov
* Set the protocol to “Raw”
* Set the Port number to 80 (the standard HTTP protocol port)
* Set the “Close window on exit” to “Never”
5) Under the Session > Logging tree, set the following options:
* Set “Session Logging” to “Log all session output”
* Set “Log file name” to what ever path and file name you want to use.
6) Now, click on the open button
7) In the session window, paste the lines copied from your editor (from step 2 above) into the session window.
8) If the above is done correctly, the server will return the data and close the connection. You can view the data by inspecting the log file created in step #5 above.
Hope this helps. Let me know if there is anything that needs more explaining.
Hank
Re: Hank Hancock (#20),
thanks, I’ll experiment with that on the weekend.
Steve,
You may not always get what you expect.
Using KNMI I tried to retrieve giss er 20c3m tas 0-360E 43-65N using the land only button, and it returns no data. Yet the everything button and the sea button return different data sets. Maybe a KNMI problem?
PS
Both north of 65N and south of 43 the land mask returns data.
Oops, I made a slight copy error. In the working example instuctions above, the host name should be set to “www.nws.noaa.gov” (ommit the http://) as below.
4) Under the Session option tree, set the following options:
* Set the Host Name to http://www.nws.noaa.gov
It appears your forum software is adding in the HTTP:// in front of the URL in step #4. In any case, just ommit the HTTP:// and go with just the “www.nws.noaa.gov” part. Sorry for not being adept at forum postings.
Re: Hank Hancock (#23), ? Oh, wait, I understand now…
Easy way to do links, btw, is to simply type in the phrase you want to represent the hyperlink, then select it and click on the Link Quicktag above the comment window. A winder will pop up requesting you to enter a URL, which you can edit as you like (it defaults with http:// on the line). I never can manage to get the text entered properly any other way.
Mark
Re: Mark T (#24),
This is the first time I’ve tried to post information where I wanted the format maintained. I’ll remember that “Link” button in the future. Thanks for your assistance.
Moderator, you can delete my posts #20 and #22 if you want to clean up the clutter.
You’re welcome. Now if I can just get Anthony Watts to put the Quicktags into his WordPress blog… 🙂
Mark