I’ve just updated my HADCRU2 dataset and, for good measure, also the current edition of CRUTEM2. I’ve updated my scripts and, for those brave souls that want their very own copy of HADCRU2 or CRUTEM2 in R format, I’ve posted up scripts to make R tables organized like time series usually are, with each column being a gridcell time series. I’ll post up some utilities as well for locating a gridcell-column from lat-long coordinates and the inverse and some plotting layouts that I’ve been using recently.
The following downloads are semi-automatic. You have to manually download and unzip the data and locate the unzipped data in a directory called c:/temp. I think that the manual download and unzipping could be automated within R, but I don’t do it often enough to learn it yet.
My computer is a couple of years old and only has 128 MB of SDRAM. This necessitated that I do this in two parts; otherwise it crashed. However the datasets as assembled can be loaded and used easily. Machines being sold now have much more SDRAM and the division in two parts would not be necessary for such a machine. (I haven’t needed more horsepower for little datasets like proxy datsets, but it’s time for me to upgrade anyway.)
HADCRU2
The Jones data was downloaded from http://www.cru.uea.ac.uk/ftpdata/hadcrut2.dat.gz on August 14, 2005. The August 2005 download produced an R-table of 1791 months from Jan 1856 to Mar 2005, with 2592 columns of 5×5 degreee grdicells, big hand: 36 lat bands N to S; little hand 72 long bands W to E (from dateline). Values of -9999 are changed to NA. Script
CRUTEM2
The Jones data was downloaded from http://www.cru.uea.ac.uk/ftpdata/crutem2.dat.gz on August 14, 2005. The August 2005 download produced an R-table of 1815 months from Jan 1851 to Mar 2005, with 2592 columns of 5×5 degreee grdicells, big hand: 36 lat bands N to S; little hand 72 long bands W to E (from dateline). Values of -9999 are changed to NA. Script
I know that there is quite a bit of interest in temperature datasets. Some of the insights to be gained are through examination of details. Many people who are working on them are using difficult languages like Fortran or time-consuming languages like Excel and I supsect that their productivity is much lower than it could be. One of the best decisions that I ever made was downloading R and getting familiar with it. You can download the R language from this link. While I’m now pretty good at R, I was able to get results right away and the payback time is pretty much instantaneous. R has a vast library of statistical and time series accessories, which provide remarkable power.
I’ve also gotten pretty good at tweaking special purpose scripts to make time series tables from ftp files of disparate formats. I’ll write a little manual some time of the tricks that I look for in downloading. I could also do a script pretty quickly (5 minutes to an hour) for some of the other data sets. so if someone who’s not used to R wants access to GISS or something else and posts up some details of access information, I’ll try to take a look at it.
Utilities
I’m going to start making a file of utilities that I find handy so that others can do this as well. This script (not up yet) will have utilities as follows:
jones – calculates column from latitude and longitude coordinates;
jonesinv- calculates latitude and longitude center of gridbox from jones format column #
plot1 – plots autocorrelation function and time series plot for an individual gridcell
I’ve got lots of other utilities, which I’ll look at see what’s applicable to the temperature dataset.
Climate Audit 
38 Comments
Ok, you finally talked me into downloading the R program. I’ve worked through part of the tutorial and it looks real cool. Now I just need to find something I really want to do with it. I suppose I should grab a couple of your scripts and download some data to play with.
I just edited the crutem2 download. The data ends in March 2005. I had dimensioned the matrix so that it was trying to go to the end of the year and it picked up early data. I fixed this. Sorry for any inconvenience. Update: I had to fix the link so this doesn’t appear to have impacted anyone.
1. Yes, treat yourself to a new computer!
2. I can link to the hadcrut2 script, but get a not found when I try to link to the crutem2 script.Steve: Fixed.
get a grant for a new puter. and some travel money and stipend while you’re at it…
I guess I should play the academic game of applying for grants. They all seem to do pretty well from NSF. I think that you need to be American to apply to NSF. In Canada, the corresponding agency is pretty much on the same wave length as David Suzuki, so I’m not particular optimistic. But all they can do is say no.
I’ve never liked bureaucracies very much; micro-cap companies were fun because of their absence of bureaucracy. My only reluctance is the idea of spending a lot of time on grant paperwork if the prospects are futile. There’s a frustrating asymmetry in this: if I tried to get funding from non-public sources, there would be a hue and cry about tainted research.
Re #5
Set up a blind trust, and then nobody knows who funded it.
Done that, but I had to put the data in directories c:\temp\crutem2 and c\tem\hadcrut2. You might want to change the script or directions to match. Looking forward to the scripts and seeing the plots. There are lots of spots I would like to look at!
I understand your feelings, Steve, yet I think there would be some benefit from applying for a grant and maybe even getting one.
1. You’ll learn a bit about what your “opponents” (“colleagues”?) are doing.
2. You’ll be a bit less of a threat to funders (rather than just invalidating the work that they paid for, it gives them a chance to take a little credit for what you get done.)
3. It might generate a little “shared experience sympathy” with your “opponents” (“colleagues”?)
4. It will give your work more credibility to the press and 3rd parties.
Don’t invest a bunch of time in it, if it is like hitting a brick wall. But take a wack at it. Considering amount of time you are spending on this effort and I would assume your developing interest in it…why not go legit?
P.s. Of course if you ARE an evil shill sophist of the oil companies, this might complicate things. Also, you will need to keep your virtue and not get sucked into becoming a craven tool of the current political fad. 😉
On the gripping hand…
This thread is drifting miles from home but what the hell. I think there would be good support for a project to conduct an audit of scientific data and methods, particularly from older and uglier scientists (like myself http://users.sdsc.edu/~davids/) who wouldn’t win a scientific beauty contest (That was a very funny comment, Steve). Its important to have a positive focus: mention the contribution of audits in other areas, that audits will increase the public’s confidence science, ensure responsible use of public funds, compliment the peer review process. Particularly if the audit had a broad scope and participation, involving the public in an educational way, and increasing transparency of science it could be very exciting. There are ways for foreign nationals to be involved. The problem would be getting it through a panel of peers, and it wouldn’t hurt to get funders on your side prior to the pitch too, as audits could only enhance their effectiveness, but there might be flack from the scientific community about supporting such a thing. I don’t see sources of funding as necessarily problem, the main thing is for the results to pass peer review. An audit will not necessarily support one side ot the other, unless someone is hiding something. You have to be careful you are casting as the villain and who is the hero. I think its better to frame us all as heros in the face of the malicious errors.
Looks like you got your worker-bee, Steve.
Now, Dave, you know any hot sheilas or POMs for me? I met the cutest Essex girl at Surfer’s once. The Brit ladies much more approachable than the Aussies though. Oh…and a gig back in San Diego might be nice. I still have a longboard chached in a friend’s garage there.
(see this stuff is way more fun than ARIMA)
(#11) No, but this site might find some help here http://www.doubleyourdating.com/. Cheers
See the trouble I get into for using business slang like "beauty contest", which has a specific nuance. Maybe Hunter will do point out that none of his friends use the term "beauty contest" and that a google does not show "global warming" and "beauty contest" used together.
For innocent readers, I used the term recently in response to comments by others as to what our "real" criticism of MBH was and that we weren’t trying to arrange a beauty contest of the various flaws, as some were interrelated.
dude, you need to be less pugnacious. Stay cool and adult. Leave the silliness to me. Oh and misuse “sexy” too.
Steve (#13): On another thread ("Gambling Runs"), I’m trying to get your cheerleaders to actually be involved in serious technical issues. If you don’t want me to continue with that, then just carry on with the gratuitous sarcasm.
Steve: Jjohn H., you seem to expect instant gratification sometimes. I sleep sometimes; I’m playing golf today and I have journal work to do urgently. Let me say that I appreciate these recent constructive comments. I intend to look up the texts you mentioned to also reply constructively.
Steve (#13 and #15): I’m not sure if you are purposely missing the point. I was not seeking your constructive replies. I was asking that you not be so insulting. I can see no reason for your sarcasm, other than that it perhaps gets a few cheers from the crowd.
re: #16
Hmmm. How do I say this without being insulting… Hmmm. Ok, I’ve got it.
John, I hear Walmart is having a special on senses of humor this week. Maybe you and a couple of your friends….
Seriously, you have to admit you got into a rather silly complaint about business usage of a term and topped it off with that senseless google search. As far as I could tell, Steve was just lightening the atmosphere a bit. If you can’t laugh at yourself now and again, well, there’s always Walmart.
Come on John. He said he would get back to you. Give him a chance. Let it ride man. p.s. hang in there
John A (#6):
> Set up a blind trust, and then nobody knows who funded it.
That sounds a good open and honest way to do business …..
Dave Dardinger (#17):
You say “you got into a rather silly complaint about business usage of a term and topped it off with that senseless google search.”
I don’t agree that the “complaint” was a “complaint” — it was just an observation about where Steve comes from. I don’t agree that the “complaint” was “silly”. I don’t agree that the google search was “senseless”.
I think you need to get over just dismissing what you don’t understand or don’t agree with as “silly” and “senseless”.
oh sheesh. Thing are getting trivial now. Could we at least say new silly things rather than arguing about whether or not what we said before was silly?
re: #20
Why did I just know you’d defend against silly and senseless? Ah well. De gustabus and all that.
BTW, the reason I said your Google search was senseless is that you were ignoring my point that the term had been widened in scope while you were trying to find a narrow use, i.e. concerning just Global Warming. So whether you’d found one or a million hits wouldn’t have proven anything. And, for that matter, unless you looked at a lot of other terms like “climate”, “CO2”, “AGW”, etc. you’d risk the sort of thing which happened in that journal article which couldn’t find any peer reviewed articles which disagreed with the warmer concensus.
Dave Dardinger (#22): I simply thought that, for a discussion on climate, Steve’s use of the term “after-market support” was remarkable, which was why I remarked upon it. If you can convince me that it wasn’t remarkable, then please do.
It’s ok. I guess it was remarkable to you, because you’re not used to that sort of chatter. I am since I’ve worked in commerce. I will sometimes slip into Navy or sports lingo (as analogy) and I guess for one not used to that it might seem strange. When you realize Steve’s background, hopefully you will not find it so strange. We can have a diversity of backgrounds.
It’s more remarkable that the concept of after market support seems so remarkable to someone in the climate field.
TCO (#24):
> I am since I’ve worked in commerce
> When you realize Steve’s background, hopefully you will not find it so strange
— unlike many, you seem to have got the point I was making!
Steve (#25):
> It’s more remarkable that the concept of after market support
> seems so remarkable to someone in the climate field.
As usual you seem to (purposely?) miss the point — what was remarkable was the terminology, not the concept.
re # 26, 27
I, like Steve and TCO, find it remarkable that commonplace terms from business would be remarkable to you. Most people work for companies at least part of their lives and know at least the basic vocabulary of business. Are you so insulated from the marketplace that you have to regard its terms as rarae aves? I certainly didn’t even blink seeing the term and I’m hardly someone who had reason to use it in the past.
Dave Dardinger (#28): I give up on this one:
1. I KNOW what “after-market support” means. I can even understand what it means when used figuratively.
2. I have never heard a scientist use “after-market support” figuratively. It seems that the use of the term indicates someone coming from a background other than science — which is exactly what I said in my original posting (“IntCal04 – RadioCarbon”, #8) which was:
“phrases like “after-market support” don’t help a scientific discussion “¢’¬? they just suggest where you are coming from”.
So ALL I was saying is that it is unhelpful (because to compare an article presented on a web site, or a scientific discussion, with something which is “marketed” is, to me, unhelpful) and that it illustrates that Steve comes from a different background — which has its good points and it has its bad points. Now can you guys leave it at that and cease the gratuitous insults?
I think it is a case of “John Hunter, give up on this one”. No one apart from you has found Steve’s comment anything other than a perfectly normal use of english. Steve was perhaps being a touch ironic in using it, if I may be so bold as to infer something for him. You see, Steve is not marketing the concept for which support was sought, and the tag is not always implemented correctly in all browsers. Maybe we still haven’t become accustomed to your very abrasive style, but the original remark came over to me, and dare I suggest it, to others, as rather condescending.
It does seem strange that marketing should seem so out of place to you, I would have thought that anyone at all familiar with the IPCC and its methods would have found marketing concepts so much like second nature as to be unnoticeable.
Steve,
I finally tried your CRUTEM2 collation script, but got the error message: Error in parse(file, n = -1, NULL, “?”) : syntax error on line 65
On a hunch I changed the 1851 on line 35 to 1859, and the 1853 on line 161 to 1859, since I’m using a recent, larger, versi\n of crutem2.dat, and the error message changed to: Error in parse(file, n = -1, NULL, “?”) : syntax error on line 129
R version 2.1.1 in Windows XP
Can you offer any hints?
I can’t figure out what you’re doing. There aren’t that many lines in the script. Did you download prior to the edit noted in #2 above?
Can you cut and paste the scripte by “paragraph” into R and tell me which script line yields the diagnostic?
Oops, my brower added redundant blank lines when I saved the script.
I will break it up into seperate pieces, run them in succession, and get back to you.
It appears that the line
v
WordPress will not accept that line. It is line 65 in your script. I will try to show it in a different way.
v=array(v,dim=c(36*72,ncol(v)/36)))
Things are looking better; thank you!
I deleted the rightmost right paren from that line, and the script appears to have worked.
On the other hand, it appears that SpamKarma is about to lock me out for the rest of the day.
Dear Mr. McIntyre,
Would it be possible to get the R scripts you wrote? The links in the article ware broken.
Kind regards,
Jeff Powell
try checking climateaudit.info – substitute .info for .org
One Trackback
[…] https://climateaudit.org/2005/08/17/downloading-cru-data/ […]