Gridding from CRN1-2

In order to assess the 2006 anomaly in John V’s 2006 calculation reported previously, I’ve calculated a gridded anomaly for the U.S. using standard spatial smoothing software in R. This enables the calculation to be done in a few lines as explained below. I’ll also show results which reconcile most of the results to date. I’ve shown the details of calculation step so that there will be no confusion. I also want to show how simple it is to produce gridded data from station data. It only takes a couple of lines of code.

CRN Ratings
First I downloaded the current ratings from Anthony Watts from http://www.surfacestations.org/downloads/USHCN_stationlist.xls and saved it as a tab-separated ASCII file. I first deleted the comment columns and changed the apostrophes to _. This is important for R programs. I then added these ratings to my USHCN info file details as follows:

url=”d:/climate/data/station/watts”
#downloaded from http://www.surfacestations.org/downloads/USHCN_stationlist.xls
#some columns deletedl also replace ‘ with _
loc=file.path(url, “stationlist.txt”)
watts=read.table(loc,sep=”\t”,fill=TRUE,nrow=1221,header=TRUE)
details$watts=NA
test=match(details$id,watts[,"USHCN.ID"]);temp=!is.na(test); sum(temp) #1221
details$watts[temp]=watts[ test[temp],5]
details$watts[(details$watts==0)&!is.na(details$watts)]=NA

I then loaded my collation of USHCN station data and extracted the stations with CRN ratings of 1 or 2 as follows:

load(“d:/climate/data/station/ushcn/ushcn.collation.tab”)
temp=(details$watts< =2)&!is.na(details$watts);sum(temp)
index_good=(1:1221)[temp]
good=ushcn.collation[[2]][,index_good] #2544 53
M=ncol(good)

Anomaly Calculation
I then converted this to an anomaly basis and calculated annual averages. I did a separate calculation for 2006 using available months only. The functions anom and ts.annavg are available in scripts/gridcell/collation.functions.txt.

good.anom=good
for(i in 1:M) good.anom[,i]=anom(good[,i])
good.ann=ts( array(NA,dim=c(nrow(good)/12,M)),start=tsp(good)[1])
for(i in 1:M){ good.ann[,i]=ts.annavg(good.anom[,i])}
temp=(c(time(good.ann))>=1890)
good.ann=ts(good.ann[temp,],start=1890)
time0=c(time(good.ann)) #1890:2006
for(i in 1:M) good.ann[N,i]=mean(good.anom[(nrow(good.anom)-11):nrow(good.anom),i],na.rm=T)

Gridding
Gridding is a type of spatial smoothing for which there are a variety of standard functions and no need for people to ponder over triangles or hexagons (which don’t matter much) or to write new code. If one wants to vary smoothing parameters (in this perspective, GISS smooths much more than CRU), then one should do so in the context of standard software where there is working experience. In this case, I used the function interp from the R-package akima (which I had previously installed.) I used three “masks” to screen out values which otherwise would get interpolated into the ocean or into Canada (tho not much turns on this.) Because each gridpoint represents an equal area, I simply calculated the mean of the values.

library(akima)
N=length(time0)
annual=rep(NA,N)
for (j in 1:N){
X=data.frame(details[index_good,c("long","lat")],good.ann[j,]) #locations of “good” stations
names(X)=c(“long”,”lat”,”anom”)
temp=!is.na(X$anom)
gridded < – interp(X$long[temp], X$lat[temp], X$anom[temp])
gridded$z=gulfmask(gridded$z)
gridded$z=atlmask(gridded$z)
gridded$z=canadamask(gridded$z)
annual[j]=mean(gridded$z,na.rm=T)
}
annual.tobs=ts(annual,start=1890)

The mask functions were not particularly elegant. The akima package with this data yields a 40×40 grid and I simply set values at inappropriate gridpoints to NA as follows.

gulfmask=function(Z){
Z[21:31,1:2]=NA;Z[21:30,3]=NA;Z[22:30,4]=NA;
Z[23:30,5]=NA;Z[c(23:24,27:30),6]=NA; Z[27:28,7]=NA; Z}

atlmask=function(Z){
Z[40,c(1:26,29:31)]=NA; Z[38:39,1:26]=NA; Z[37,1:23]=NA;
Z[36,1:16]=NA;Z[35,1:15]=NA; Z[34,1:13]=NA;Z[33,1:11]=NA;
Z}

canadamask=function(Z){
Z[40,37:40]=NA; Z[37:39,35:40]=NA; Z[36,34:40]=NA;
Z[34:35,32:40]=NA;Z[32:33,29:40]=NA; Z[31,c(28:29,33:40)]=NA;
Z[30,36:40]=NA;Z}

Here’s an illustration of the points that are masked out by these masks. I didn’t bother with masks in other places because the algorithm didn’t interpolate over there, but it could be esily done.

usgrid52.gif

Here’s an example of the type of annual map that is generated by the smoothing function. This was generated by doing a single step and plotting from the object gridded produced in the above routine as follows (using the R package fields). You’ll note that this smoothing function does not calculate gridded values at the edges which are an extrapolation from the stations. (This map also shows the stations used in the calculation.)

par(mar=c(3,3,2,1))
breaks0=seq(-3,3,.25)
image.plot (gridded$x,gridded$y,gridded$z, col=tim.colors(length(breaks0)-1),breaks=breaks0,ylab=””)
contour(gridded, add=TRUE,ylab=””)
points (X$long[temp],X$lat[temp], pch = 19,cex=.7)
US(add=TRUE,lwd=2)
title(main=paste(“USA “, time0[j]))

usgrid53.gif

So it’s pretty easy to do a gridding calculation using standard software. Hansen’s program has very ad hoc statistical methods in which stations are adjusted from other stations and gridcells calculated. At the end of the day, the best approach to this aspect of his program will probably be to see what smoothing parameter best emulates his method and thereby restore it to a known statistical methodology.

COMPARISONS

First here is a comparison of my calculation using CRN1-2 data to NOAA results from 1890-2007. (I re-iterate for the N-th time that analyses on U.S. data are not representative of the ROW.) NOAA has values to Aug 2007 and I’ve estimated the last 4 months as the same as the corresponding 2006 months for now. NOAA has run warmer by about 0.2 deg for the past decade and was about 0.1 deg C cooler in the 1930s.

usgrid54.gif

Second here is a comparison of CRN1-2 data to GISS pre-Y2K results. As you see the GISS with the Y2K error was running warm by about the same as NOAA – which may be one reason why they didn’t spot the error. GISS runs a bit warmer than NOAA in the early and mid portions of the graph due to their greater efforts to do an urban adjustment.

usgrid55.gif

Third, here is a comparison of CRN1-2 data to GISS post-Y2K results (but not including the September change from SHAP to FILNET, which seems much too opportunistic.) Relative to CRN1-2, GISS tends to warm the cool periods (1960s, 1900s) and to cool the warm periods (1930s), reducing the size of the mid-20th century cooling. For the U.S., as noted before, by using the trends from “unlit” stations to establish trends – in a context where there are a lot of decent rural stations with long records – the GISS methodology for USHCN stations seems more appropriate than the NOAA (and probably the CRU methods). Unfortunately, this method is not used in the ROW, where there is nothing equivalent to the USHCN.

usgrid56.gif

Finally, here is a reconciliation to John V’s annual results sent ot me yesterday. His 2006 results run much colder than the ones that I calculate- which is presumably an artifact of the 3 months stub period that he used in his calculation. So the anomaly with his 2006 results can be traced definitely to this (which shows up a “cool” difference in the graph below.)

usgrid57.gif

So what does my calculation using CRN1-2 data look like: see below, which includes a 2007 estimate (using the 2006 difference between the CRN1-2 and NOAA to estimate 2007 CRN1-2 from the YTD NOAA results.) In this calculation, the warmest year remains 1934 by a margin of nearly 0.2 (enough to be “statistically significant” according to things that GISS has said, then 1921, then 1998, then 2006 in fourth place. Values since 1998 have been at uninterrupted high levels. Looking at the graph, it’s definitely noticeable – but is it the sort of change that’s inconsistent with the sort of stochastic behavior in the early part of the series – hard to tell here. If you cut the series off in 1997, you certainly wouldn’t have noticed anything odd about the 1990s. You can sort of see in retrospect why MBH98-99 were so excited about how warm 1998 and in such a rush to include it in their article. The main verification issues right now is why the ROW history is so different from the US – is it climatic or nonclimatic? And can one tell?

usgrid58.gif

106 Comments

  1. Mike B
    Posted Oct 4, 2007 at 1:45 PM | Permalink | Reply

    THanks Steve. What does CRN1,2 TOBS – CRN 1,2 NO TOBS look like?

  2. Posted Oct 4, 2007 at 1:46 PM | Permalink | Reply

    That’s an interesting analysis. A few points come to mind:

    I realized that my 2006 data included 10 months, not 3 months as I previously stated. I made the correction and apologized in the previous thread.

    I think that trendlines would be informative. Eyeballing it, it looks like GISTEMP shows less warming over the century than CRN12. That is consistent with my results. Some sort of smoothing over the single-year data would also make it easier to identify any trends.

    Where did you find USHCN TOBS data for the rest of 2006? I’ve been looking for it.

    My results were for rural CRN12 stations (CRN12R) rather than all CRN12 stations. Your comparison of my results vs your results is therefore skewed. If I recall correctly, CRN12R shows slightly less warming than CRN12 in my analysis.

    By the way, I’d prefer if you used “John V” or “OpenTemp” in showing my results. If you prefer to use my last name, it’s “Van Vliet”.

  3. Posted Oct 4, 2007 at 1:48 PM | Permalink | Reply

    SteveMc said:

    If one wants to vary smoothing parameters (in this perspective, GISS smooths much more than CRU), then one should do so in the context of standard software where there is working experience.

    You’re right. Like most software developers I have been suffering from a bit of a not-invented-here syndrome. I will look around for an open-source toolkit for gridding or smoothing. Can you recommend some search terms to get me started?

  4. Steve McIntyre
    Posted Oct 4, 2007 at 1:52 PM | Permalink | Reply

    #2. John, I have USHCN TOBS only up to Oct 2006 for most of these stations and used the 10-month average of anomalies.

    If you icluded 10 months of 2006 data, I’m really puzzled why your 2006 is so low. Something must have gone awry – maybe with missing data somewhere – leading to your too low 2006.

  5. Posted Oct 4, 2007 at 1:53 PM | Permalink | Reply

    Shouldn’t the corner in the southwest be removed from the grid? Without removing it, the stations in that part of the USA lower 48 will affect more land than they should.

  6. Steve McIntyre
    Posted Oct 4, 2007 at 1:55 PM | Permalink | Reply

    John V, the open source code in R that I’ve posted here is, I think, exactly what you’re trying to do. The R package is akima; you might look at its manual. But I think that this post already does it.

  7. Posted Oct 4, 2007 at 1:58 PM | Permalink | Reply

    #4 Steve McIntyre:
    There was a bug in OpenTemp where it would generate yearly results based on incomplete months, but still divide by 12. It has been fixed. It now excludes any years without data for all 12 months. (There are more details in the previous thread).

  8. Earle Williams
    Posted Oct 4, 2007 at 2:02 PM | Permalink | Reply

    Re #4

    John V.,

    You said

    You’re right. Like most software developers I have been suffering from a bit of a not-invented-here syndrome. I will look around for an open-source toolkit for gridding or smoothing. Can you recommend some search terms to get me started?

    There are entire texts on the subject and no shortage of code developed. Some quick googling didn’t point me to any particular open-source toolkit but I did find a nice write-up that breaks down some of the issues at:
    http://www.epa.gov/oar/oaqps/pm25/docs/dsisurfaces.pdf .
    It may provide a good starting point for terminology to use in more focused searches.

    Regards,
    Earle

  9. Posted Oct 4, 2007 at 2:19 PM | Permalink | Reply

    For irregular sampled data, I recommend kriging (well known in geochemical sampling), this gives you also an error estimate.

    http://en.wikipedia.org/wiki/Kriging

  10. Posted Oct 4, 2007 at 2:22 PM | Permalink | Reply

    Re my #5, I see now that the Akima interpolation did not extend into the Pacific in the southwest (as you said). Never mind.

  11. Posted Oct 4, 2007 at 2:26 PM | Permalink | Reply

    be also aware that a rectangular lat long grid is not equal area

    http://en.wikipedia.org/wiki/Map_projection#Equal-area

  12. Steve Sadlov
    Posted Oct 4, 2007 at 2:28 PM | Permalink | Reply

    As with the many excellent plots rendered earlier by JohnV, this post once again brings me back to the question – just what in the heck happened between the mid/late 1910s and the early 1920s? Yes, it could be nothing more than an odd stochastic jump. Or, it could be something truly wierd. Either something wierd with the weather/climate, or, wierd with how data from prior to the 1920 were gathered, recorded, handled or adjusted. I look at really neat charts every day as part of running a business. If I say something like that on one of my charts, my hackles would be raised.

  13. Steve Sadlov
    Posted Oct 4, 2007 at 2:29 PM | Permalink | Reply

    say s/b saw. Doh!

  14. Posted Oct 4, 2007 at 2:58 PM | Permalink | Reply

    I can highly recommend generic mapping tools (GMT, freeware!) for better handles on cartography and mapprojections

    http://www.soest.hawaii.edu/gmt/

    What is GMT?
    GMT is an open source collection of ~60 tools for manipulating geographic and Cartesian data sets (including filtering, trend fitting, gridding, projecting, etc.) and producing Encapsulated PostScript File (EPS) illustrations ranging from simple x-y plots via contour maps to artificially illuminated surfaces and 3-D perspective views. GMT supports ~30 map projections and transformations and comes with support data such as coastlines, rivers, and political boundaries. GMT is developed and maintained by Paul Wessel and Walter H. F. Smith with help from a global set of volunteers, and is supported by the National Science Foundation. It is released under the GNU General Public License.

  15. Posted Oct 4, 2007 at 3:01 PM | Permalink | Reply

    take a look at these!

    http://www.soest.hawaii.edu/gmt/gmt/gmt_examples.html

  16. Stan Palmer
    Posted Oct 4, 2007 at 3:20 PM | Permalink | Reply

    re 2

    I think that trendlines would be informative. Eyeballing it, it looks like GISTEMP shows less warming over the century than CRN12. That is consistent with my results. Some sort of smoothing over the single-year data would also make it easier to identify any trends.

    Wouldn’t the provision of a trend line be consistent with the assumption that there is a linear trend. What is the justification for this assumption? As someone has pointed out, there appears to a be a step-wise discontinuity around 1920 with the additional appearance of a step down in the 70s and a current step up in 1998. If this appearance has any validity a linear trend would not be of much use.

  17. DocMartyn
    Posted Oct 4, 2007 at 3:23 PM | Permalink | Reply

    How does this correlate with population changes from, say 1930 to 2006. Has not the US population moved from the NE to the SW?

  18. Kristen Byrnes
    Posted Oct 4, 2007 at 3:27 PM | Permalink | Reply

    OK, here is the update for CRN 1,2R stations that shows the most recent move or change in equipment according to MMS (remember it’s NCDC so expect some problems like Woodstock, VA that Douglas Hoyt talked about in the last thread). But we can verify that when we get the B44 forms. I think that no data should be used in its current CRN rating prior to moves or equipment changes. I understand John V has been, ummmm, engaged all day so I don’t know if he had time to make the changes. I haven’t gotten back the same for CRN 5 but a volunteer is working on it.
    John V says he programmed opentemp to adjust geographic distribution each month to account for missing months, I think that was a good thing. I do not think it is necessary to eliminate a whole year of data just because one month is missing.
    I understand that this is a small amount of data but given time we will be able to get enough data to fill each grid cell.
    John V here is a link to satellite data:

    http://vortex.nsstc.uah.edu/public/msu/t2lt/uahncdc.lt

    There is data from a second satellite called RSS which usually runs about a tenth warmer.

    Rural CRN12 Stations:
    42572383001,34.7,-118.43 Fairmont CA (Not Surveyed, aerial photographs only but looks good) / No changes in location or equipment / records back to 1931
    42572694004,44.63,-123.2 Corvallis State University OR / no change in location or equipment / records back to 1948

    42572376006,35.27,-111.73 Fort Valley AZ / Equipment: 2004 – current “other temperature equipment” / 1986 – 2004 MMTS (LIG backup) / before 1986 LIG / Location same throughout / records back to 1948

    42572273004,31.7,-110.05 Tombstone AZ Equipment: MMTS 2002 – current / LIG 1933 – 2002 / Location: same throughout / records back to 1933

    42574501001,38.33,-120.67 Electra ph CA / Station closed 1997 / Equipment LIG throughout / Location: same throughout / records back to 1931

    42572489002,39.97,-120.95 Quincy CA / Equipment: MMTS 1999 – present / LIG before 1999 Location: last change 2004 / records back to 1931

    42572786006,48.35,-116.83 Priest River Experimental Station ID / no location or equipment changes / records back to 1911

    42572533004,41.63,-84.98 Angola IN / Location: Location move reflected in elevation change 1977 / Equipment 1985 – Present: MMTS / LIG before 1985 / records back to 1887

    42572743002,46.33,-86.92 Chatham experimental farm MI / Location: last move (short distance) probably not a problem / Equipment: LIG throughout / 1987: STN ESTABLISHED TO COMPARE WEATHER READING WITH CHATHAM EXP FARM 20-1484-2, 1.1 MI TO THE NNW / This station is a problem, the correct name is Chatham Exp Farm 2. The actual station is Chatham Exp Farm, which was active from 1948 to 1988 with no recorded station moves and LIG equipment throughout station history.

    42574341005,45.13,-95.93 Milan 1 nw MN / Location same throughout / Equipment Nimbus 2006 to current / LIG before that / records back to 1948

    42574341007,45.58,-95.88 Morris wc Experiment Station MN / Location: same throughout / Equip: LIG throughout / records back to 1948

    42572235004,31.97,-91 Port Gibson 1 ne MS / Possible problem with either MMS or Surfacestations.org / MMS exposure & obstruction field do not match photos, MMS says: MMTS 260/10 F&P 90/7 TREE 100/75/29 140/100/25 HOUSE 160-220/90-120/14-7 TREES LINES 165-200/175-125/24-24 200-280/120-200/30-15, photos do not show these obstructions / Location changed 2001 / Equipment: MMTS 1988 to current / LIG before 1988 / records back to 1948

    42572677004,45.92,-108.25 Huntley Experimental Station MT / Location and Equipment (LIG) unchanged / records back to 1951

    42572488003,39.45,-118.78 Fallon Experimental Station NV / Location changed in 1992 / equipment the same (LIG) throughout

    42572662003,43.97,-101.87 Cottonwood 2e SD / Location unchanged / Equipment: MMTS 1986 – current / LIG before 1986 / records to 1948

    42572572005,40.55,-111.5 Snake Creek PH UT / Location: no changes / Equipment: Thermograph 1997 to current / LIG 1997 and before / records to 1948

    42572403002,38.9,-78.47 Woodstock 2 NE VA / Location change 1994 / Equipment MMTS 1993 to current / LIG before 1993

  19. Posted Oct 4, 2007 at 3:30 PM | Permalink | Reply

    I just thought trend lines would be more consistent with some of Steve McIntyre’s previous articles.

    http://www.climateaudit.org/?p=2069

    http://www.climateaudit.org/?p=2107

    http://www.climateaudit.org/?p=2092

    If not trendlines, then maybe a direct comparison of temperature anomalies (vs differences in temperature anomalies), a smoothed plot, trailing 20yr trends, or trends for key periods would be useful. (Those are the plots I typically provide in my results).

  20. MrPete
    Posted Oct 4, 2007 at 4:11 PM | Permalink | Reply

    Hans (#11, 14) and Steve M — there’s a very simple way to achieve equal-area map output.

    X = longitude as usual
    Y = sin(latitude)

    This adjustment to Y creates an authalic (equal-area) latitude coordinate system.

    Handy, accurately equal-area, and very fast.

    (Map projections is one of my long-ago specialties…)

  21. Tim Ball
    Posted Oct 4, 2007 at 4:19 PM | Permalink | Reply

    The last graph (USA Calculated from CRN 1-2 TOBS) illustrates what I have pointed out for many years and is evident in all plots of USA temperatures for this period. The variablity is high from 1850 to 1940, low from 1940 to 1980 (except for the anomalous 1950 spike), then high from 1980 to the present. I believe this pattern is most evident in the middle latitudes (30° to 65°) especially in the US with the most extensive record. It is likely a result of the general pattern of waves in the circumpolar vortex that dominate the weather pattern in these latitudes changing from zonal flow to meridional flow. It suggest zonal flow dominates when global temperatures are declining and meridional when they increase.

  22. Gunnar
    Posted Oct 4, 2007 at 4:24 PM | Permalink | Reply

    >> Wouldn’t the provision of a trend line be consistent with the assumption that there is a linear trend. What is the justification for this assumption? As someone has pointed out, there appears to a be a step-wise discontinuity around 1920 with the additional appearance of a step down in the 70s and a current step up in 1998. If this appearance has any validity a linear trend would not be of much use.

    Thank you Stan. That is a huge assumption. In fact, it is assuming AGW a-priori.

  23. Philip_B
    Posted Oct 4, 2007 at 4:27 PM | Permalink | Reply

    One candidate for what happened in and around 1920 is expansion and mechanization of agriculture, i.e. large scale land use changes. In 1920, US wholesale agricultural prices dropped nearly 50%, indicating a large surge in agricultural supply.

    http://eh.net/encyclopedia/article/Smiley.1920s.final

  24. Steve McIntyre
    Posted Oct 4, 2007 at 4:30 PM | Permalink | Reply

    #20. PEte and Hans, sure, but within the lower 48, the difference is negligible.

  25. Posted Oct 4, 2007 at 4:33 PM | Permalink | Reply

    #20:
    The OpenTemp grid cells are area-weighted for this reason.

  26. Gunnar
    Posted Oct 4, 2007 at 4:40 PM | Permalink | Reply

    >> One candidate for what happened in and around 1920 is expansion and mechanization of agriculture

    Why do you think of man first, when he is dwarfed by his environment, including the sun.

  27. Steve Sadlov
    Posted Oct 4, 2007 at 4:51 PM | Permalink | Reply

    RE: #21 – Tim, that is definitely something to look at. It may be a proxy for PDO phase or some higher order harmonic.

  28. Steve Sadlov
    Posted Oct 4, 2007 at 4:53 PM | Permalink | Reply

    RE: #26 – Another thought that I had was that there is some sort of systemic error in TOBS adjustments. For example, there may be something badly wrong with how TOBS was dealt with for the time prior to 1920. There may be other systemic errors above and beyond this.

  29. steven mosher
    Posted Oct 4, 2007 at 5:02 PM | Permalink | Reply

    JohnV addresses some of these these issue in his inline comments of the code.
    Take no offense at my implication that you have not read the code. hehe.

  30. MrPete
    Posted Oct 4, 2007 at 5:11 PM | Permalink | Reply

    (#24 — I mostly tend to make this adjustment so things “look correct” to people :-D)

  31. steven mosher
    Posted Oct 4, 2007 at 5:12 PM | Permalink | Reply

    here are JohnVs comments in the code, relative to this issue. He is well aware of the issue.
    And plans an improvement. RTFC!

    // TODO: Improve the area calculation for a cell
    // The current calculation works well for a small cell, but will break down for larger cells (> ~1000km on a side)
    float sinLatitude = (float)Math.Sin(latitude * Math.PI / 180.0);
    float degLength = (float)(2.0 * Math.PI * _earthRadius / 360.0); // Length of a degree at the equator
    _area = (width * sinLatitude * degLength) * (height * degLength);
    }

    ///
    /// Get the latitude (in degrees) of the centre of this cell.
    ///
    internal float Latitude
    {
    get { return _latitude; }
    }

    ///
    /// Get the longitude (in degrees) of the centre of this cell.
    ///
    internal float Longitude
    {
    get { return _longitude; }
    }

    ///
    /// Get the area of this cell.
    ///
    /// The area of the cell is calculated from the width and height. Currently the area calculation assumes
    /// small values for width and height. That is, it assumes that the cell is square in cartesian coordinates.
    /// The error for a 1000km cell is approximately 0.2%, increasing to 0.8% for a 2000km cell.

  32. Steve McIntyre
    Posted Oct 4, 2007 at 5:22 PM | Permalink | Reply

    If one line in my code above is changed to:
    annual[j]=weighted.mean(gridded$z,rep(cos(gridded$y*pi/180),40),na.rm=T)

    this allows for latitude change in weights. You don’t need to worry about the radius of the earth since everything is proportional.

  33. Steve McIntyre
    Posted Oct 4, 2007 at 5:36 PM | Permalink | Reply

    The following implements a weighted calculation as a function for an index list of USHCN stations:

    ushcn_wt=function(index) {
    good=ushcn.collation[[2]][,index] #2544 53
    M=ncol(good)
    #anomalies
    good.anom=good
    for(i in 1:M) good.anom[,i]=anom(good[,i])
    good.ann=ts( array(NA,dim=c(nrow(good)/12,M)),start=tsp(good)[1])
    for(i in 1:M){ good.ann[,i]=ts.annavg(good.anom[,i])}
    temp=(c(time(good.ann))>=1890)
    good.ann=ts(good.ann[temp,],start=1890)
    time0=c(time(good.ann)) #1890:2006
    for(i in 1:M) good.ann[N,i]=mean(good.anom[(nrow(good.anom)-11):nrow(good.anom),i],na.rm=T)
    #AREA AVERAGES
    N=length(time0)
    annual=rep(NA,N)
    for (j in 1:N){
    X=data.frame(details[index_good,c("long","lat")],good.ann[j,]) #locations of “good” stations
    names(X)=c(“long”,”lat”,”anom”)
    temp=!is.na(X$anom)
    gridded < – interp(X$long[temp], X$lat[temp], X$anom[temp])
    gridded$z=gulfmask(gridded$z)
    gridded$z=atlmask(gridded$z)
    gridded$z=canadamask(gridded$z)
    annual[j]=weighted.mean(gridded$z,rep(sin(gridded$y*pi/180),40),na.rm=T)
    }
    ushcn_wt=ts(annual,start=1890)
    ushcn_wt
    }

    This yields an annual series and plot in a couple of lines:

    temp=(details$watts< =2)&!is.na(details$watts);sum(temp) #picks 1-2
    index_good=(1:1221)[temp]
    test=ushcn_wt(index_good)
    ts.plot(test)

    Or for rural stations:

    temp=(details$watts< =2)&!is.na(details$watts)&(details$bright=="dark");sum(temp) #picks 1-2
    index_good=(1:1221)[temp]
    test=ushcn_wt(index_good)
    ts.plot(test)

    Simple as pie.

  34. Philip_B
    Posted Oct 4, 2007 at 5:41 PM | Permalink | Reply

    Why do you think of man first, when he is dwarfed by his environment, including the sun.

    I could equally well ask, why posit a global scale process to explain what could well be local effects.

  35. Kenneth Fritsch
    Posted Oct 4, 2007 at 6:33 PM | Permalink | Reply

    MrPete on Post # 75 from the closed thread “2006 and CRN1-2″ said:

    I don’t have links at my fingertips, but I’ve seen statements alluding to even satellite measures being calibrated off the surface station measures.

    I have seen these alluded to statements also, but never seen the links validating it. All I have found is a calibration process whereby the sensing devices from satellites are calibrated using the deep space temperature and a known calibrated temperature standard on the earth’s surface. I have not been able to find the surface temperature station data and satellite connection. Admittedly the satellite measurements have errors and measurements appear to be algorithm related.

    I think this is an important point in these ongoing discussions that needs to be put to bed once and for all.

  36. Posted Oct 4, 2007 at 7:34 PM | Permalink | Reply

    Steve McIntyre:
    I just emailed you with my results for all CRN12 stations, rather than just the rural stations. I would appreciate it if you would update the “Vliet minus CRN1-2 (SM)” plot to use these results to make the comparison valid.

    As I mentioned before, I would also appreciate my results being labelled as “John V”, “OpenTemp”, or at least “Van Vliet”.

    Thanks.

  37. steven mosher
    Posted Oct 4, 2007 at 8:04 PM | Permalink | Reply

    RE 37. JohnV I think I’m gunna start to list.

    1. The version of OpenTemp I Used.
    2. The files and date I used.

    Is there a simple metadata format we could adopt. *cvs is convient.
    .
    Ideally, ( in my perfect world ) clicking on a data point in a chart would magically find the
    code, load the data, and rerun… Hmm. Neat idea.. and so every piece of data would come encoded with
    its own DNA for replicating it… Ok way too too much coffee

  38. BarryW
    Posted Oct 4, 2007 at 8:12 PM | Permalink | Reply

    One thing that stands out to me but I can’t see why it exists is the periodicity in the differences between the CRN1-2 and the other data. I would have thought there would be a random difference in any one year with maybe a superimposed trend (assuming UHI issues) but the sign of the differences appear in clusters. If I’m actually seeing this and not imagining a pattern (quite possible) then what would induce a sinusoidal pattern in one or the other data sets but not the other? Could it be that the pattern is due to the non uniform geographic coverage of the CRN1-2 with a climatic shift over the US on a decadal timeframe? Would it go away if you reduced the NOAA data set to those areas where there is equivalent coverage in CRN1-2?

  39. Posted Oct 4, 2007 at 8:54 PM | Permalink | Reply

    #38 steven mosher:
    I have not changed any of the OpenTemp processing code since the first release (on or around Sept 18). I have only been adding new features and fixing output bugs. To answer your question from earlier today, your old batch files will continue to work.

    I’m going to add a few more features and then I’ll do a stable versioned release:
    – file prefix for output (to reduce the number of folders required)
    – per-station date filtering
    – random selection of N stations from the list

    The gridding and averaging code could use a little work, but changes there will change the output so they can wait for the next version.

  40. Posted Oct 4, 2007 at 9:15 PM | Permalink | Reply

    Steve McIntyre:
    I think there may be a problem with the reference period for the anomalies in your GISS comparison plots above. You did not state which reference period you were using, but from your code it looks like all dates after 1890 (I don’t know R so that’s just a guess). As I understand the anomaly method, the average of difference between two anomaly plots with the same reference period should be zero. The GISS comparison plots seem to be biased positive.

  41. steven mosher
    Posted Oct 4, 2007 at 11:47 PM | Permalink | Reply

    RE 40.. I should be able to make a “batcave” a master.bat with all my other *bats included.

    I’ve sent you a bunch of station lists, perhaps we should compile a test script. So when you
    have a new release you can check against old versions.

  42. Posted Oct 5, 2007 at 6:58 AM | Permalink | Reply

    #32 Steve McIntyre:
    Shouldn’t that be cosine?

    annual[j]=weighted.mean(gridded$z,rep(cos(gridded$y*pi/180),40),na.rm=T)

    Or maybe my brain’s just not working yet. It’s early here.

    Steve: Yes. MY bad. In the US 48, the areal weighting really isn’t a big consideration as the rane is only from 0.87 to 0.65.

  43. Posted Oct 5, 2007 at 7:12 AM | Permalink | Reply

    Where does one find the new USHCN V2 monthly (or daily) station data? V1 is available from CDIAC, either daily or monthly, but only through 2005. I’d like to see how the 2006 spike looks at my local stations, Circleville and Delaware OH. Delaware has been defunct since 1/30/01, but that hasn’t prevented it from providing phantom evidence for AGW up through 2005, according to CDIAC! NCDC says the new V2 was to be released in July of 2007, but I can’t find a link there to the actual data.

  44. Gunnar
    Posted Oct 5, 2007 at 7:26 AM | Permalink | Reply

    >> I have not been able to find the surface temperature station data and satellite connection.

    You are correct. I understand that there is on board calibration with a known sample of oxygen, and readings are also calibrated to “known calibrated temperature standard on the earth’s surface”. Like all instrumentation, there are always error, and all instrumentation requires calibration, including thermometers, thermocouples, etc. We don’t even use thermometers for measuring kids body temperature for fever anymore. It’s archaic.

  45. Posted Oct 5, 2007 at 8:02 AM | Permalink | Reply

    #43 Hu McCulloch:
    USHCN v2 is not available yet. USHCN v1 is available here (to October 2006):
    ftp://ftp.ncdc.noaa.gov/pub/data/ushcn

  46. Kenneth Fritsch
    Posted Oct 5, 2007 at 10:00 AM | Permalink | Reply

    Re: #44

    The link and excerpt below is the closest I have come to any details of the calibration of the microwave instrumentation on satellites used to measurement tropospheric temperatures. Further reading on the subject leads me to conclude that the warm targets are onboard the space craft.

    http://iridl.ldeo.columbia.edu/descriptions/.msuprecip.html

    Once every earth scan, the instrument makes calibration measurements, viewing deep space (2.7K) and high emissivity warm targets. There is one target for the two lower frequencies (channels 1 and 2) and another for the two highest frequencies (channels 3 and 4). The temperature of each target is monitored with redundant platinum resistance thermometers (PRT’s). Calibration of the instrument digital counts into brightness temperatures (Tb) is simply a linear interpolation of the Earth-viewing measurements between the space and warm target measurements (see Spencer et al., 1990). Please refer to Smith et al. (1979) for further details of the MSU.

  47. John Hekman
    Posted Oct 5, 2007 at 10:57 AM | Permalink | Reply

    Steve M

    Hu McCulloch (#43) is a top-notch econometrician, if you have questions about that. (Hi, Hu, long time!)

  48. Alan Bates
    Posted Oct 5, 2007 at 10:58 AM | Permalink | Reply

    Hello from the UK.

    This is my first attempt to comment although I have lurked with great interest for a couple of years or more.

    I am a mature (in age – 60+) student with the Open University studying Earth Sciences. I am way out of my depth on scraping data etc. but there is a technique I have used in my previous employment for plotting spatial data of this sort i.e. Semi-Variograms and Krigging.

    This has been used in the past in geostatistics, for example in Uranium prospecting and in presenting data for air pollution by SO2 in the UK. As I understand it, the semi-variogram takes all the spatial data on an irregularly spaced grid and determines how far away one has to be before the results from one station are no longer correlated with another. The Kiigging then uses this information to carry out the interpolation to produce a contour map of the results.

    The advantage (if I understand it) is that the process is well accepted and uses standard computer packages.

    best wishes

    Alan

  49. Posted Oct 5, 2007 at 12:05 PM | Permalink | Reply

    Earle Williams and Hans Erren, thanks for the links. I will probably stick with inverse-distance-weighting (IDW) in the near-term and move to one of the more robust methods for the next version of OpenTemp.

    Alan Bates, thanks for the additional info. I was leaning towards Kriging based on my reading yesterday — it’s useful to get an independent recommendation for it.

  50. UK John
    Posted Oct 5, 2007 at 12:49 PM | Permalink | Reply

    Now thats better!

    I can understand most of this, unlike the previous thread, which seemed to consist of “my Dick’s bigger than yours”.type statistics

    Please keep up the good work, out here the world has gone enviro”mental”, and you are one of the few places keeping me sane.

  51. steven mosher
    Posted Oct 5, 2007 at 2:36 PM | Permalink | Reply

    Thanks Alan!

  52. Posted Oct 5, 2007 at 11:45 PM | Permalink | Reply

    Regarding Steve’s comment

    “So what does my calculation using CRN1-2 data look like: see below, which includes a 2007 estimate (using the 2006 difference between the CRN1-2 and NOAA to estimate 2007 CRN1-2 from the YTD NOAA results.) In this calculation, the warmest year remains 1934 by a margin of nearly 0.2 (enough to be “statistically significant” according to things that GISS has said, then 1921, then 1998, then 2006 in fourth place. Values since 1998 have been at uninterrupted high levels. Looking at the graph, it’s definitely noticeable – but is it the sort of change that’s inconsistent with the sort of stochastic behavior in the early part of the series – hard to tell here. If you cut the series off in 1997, you certainly wouldn’t have noticed anything odd about the 1990s. You can sort of see in retrospect why MBH98-99 were so excited about how warm 1998 and in such a rush to include it in their article. The main verification issues right now is why the ROW history is so different from the US – is it climatic or nonclimatic? And can one tell?”

    It looks like from 2000 on, there is a 0.5 degree step function, notice none of the temperatures swings go below 0.5 on the graph. Is this another Y2K problem?

  53. Kristen Byrnes
    Posted Oct 6, 2007 at 8:43 AM | Permalink | Reply

    Leon Palmer,

    Or it could have to do with all the stations being asked to change time of observation to 7 AM in the last year or two.

  54. steven mosher
    Posted Oct 6, 2007 at 9:34 AM | Permalink | Reply

    RE 53. ALL the stations? Even the ASOS. Interesting, can you site the reference

  55. Posted Oct 6, 2007 at 10:13 AM | Permalink | Reply

    Steve McIntyre:
    I am interested in looking at the scale of the differences between CRN12 results and GISTEMP/NOAA results. I would appreciate it if you would post the annual temperatures you calculated from CRN12 as well as a link to the NOAA monthly or annual tempertures. Thanks.

    I am also having trouble reconciling a few of your plots. Maybe you can help:

    The “NOAA minus CRN1-2 TOBS (SM)” plot above shows a small warming trend (~0.1C/century by eye).
    The “GISS Post-Y2K minus CRN1-2 TOBS (SN)” shows a small cooling trend (~0.1C/century by eye).

    Based on the this I would guesstimate NOAA minus GISS to be ~0.2C/century.

    The “US48: NOAA – NASA” plot in the Second Look at USHCN article shows a trend of 0.66C/century.

    Why the difference? Are my estimates for the trends above wrong?

    Steve: good question. It doesn’t look like the graphics are consistent. I’ll try to reconcile them this weekend (which is Thanksgiving and I’ll be away for much of it.)

  56. Posted Oct 6, 2007 at 12:36 PM | Permalink | Reply

    RE #21

    Tim,

    Wouldn’t you expect it to be the other way round?

    Aren’t cooler temperatures associated with a wider, more southerly track for the jet stream and a tendency to block, creating meridional flow?

    Don’t warmer temperatures have the reverse effect?

    Also, the variability is just as high 1890-1920 as in 1920-1940 even though the first period is consistently cooler than the latter. If your argument was correct, shouldn’t the degree of variability be different?

    I do not understand why everyone is analyzing such an obviously biased data set to death. Things may look quite different when the rest of the map is filled in. The part which is missing, the interior plains, is an area particularly susceptible to meridional flow and temperature and precipitation extremes under cooler temperature regimes and vice versa under warmer regimes. That is, it is where a great deal of the variability for US temperatures might be expected to be on decadal time scales.

    I find it suspicious that this data set does not show the commonly accepted 1900 to 1940 warming trend, supposedly stronger than the reputed one since 1975. It may be that this trend does not exist, but until Anthony has a more complete data set, it would be unsafe to assume that the sample we have to hand proves that. It could just as easily point to the unacceptable bias in the current sample.

    Based on the meridional versus zonal flow argument as I have laid it out, as applied to this data set, one could argue that, contrary to the accepted opinion, that 1900 to 1940 and after 1975 were cool periods. Moreover, the severe African and Great Plains droughts of those periods are consistent with persistent meridional flow and jet streams closer to the equator. These droughts are considerable counter facts to the global warming hypothesis.

    I think we have a long way to go before we really know what has been going on with surface temperatures.

    Mike

  57. Carl Gullans
    Posted Oct 7, 2007 at 8:57 AM | Permalink | Reply

    #17: The whole point of CRN1,2 stations is that they are of high quality and are not in need of a UHI adjustment; population moves shouldn’t affect the data at all.

  58. Posted Oct 7, 2007 at 9:42 AM | Permalink | Reply

    Carl Gullans:
    The CRN1 and CRN2 stations are relatively free of microsite issues, but the CRN rating says nothing about UHI.

  59. Posted Oct 7, 2007 at 9:45 AM | Permalink | Reply

    Steve McIntyre:
    I realize it’s a long weekend, but I would appreciate it if you could find a few minutes to update the comparison plot above (my results vs your results). I would like to do some new work with OpenTemp and would prefer to wait until a comparison with your results is available.

    If it’s easier, you could also just send me your yearly CRN12 averages and update the graph later.

    Thanks.

  60. steven mosher
    Posted Oct 7, 2007 at 11:01 AM | Permalink | Reply

    RE 58. UHI

    causes of UHI: from Johnsen

    1. Street Canyons and screening the skyline.
    2. Increased heat storage in buildings.
    3. Anthro heat: animal metabolism.
    4. Urban Greenhouse from urban pollution.
    5. Canyon radiation geometry- Basically multipath IR
    6. Evapotranspiration impairment. ( artificial ground cover)
    7. Wind Shelter: Reduced turbulant transfer.

    When you look at the microsite issues you see the same causes ( save urban pollution)

    Very simply. You can have a site in an Urban place (pop 1M) that doesnt have a lot of wind shelter (#7)
    like an airport. AND you can have a site in the Wilderness ( pop ZERO) that does have wind shelter.

    NOAA and GISS use population as a proxy for the details of the site. Since we dont have a parametric
    model of microclimate, as Karl notes, he uses population to stand in for the 7 causes.
    NAmely, if you have a high population chance are the site will be impaired by these factors.
    If you have a LOW population chances are you won’t. We know this to be untrue. The point of visiting
    sites was to dtermine the following

    1. Street Canyons and screening the skyline. Is the skyline screened at the site?
    are there trees that screen it, is the ground sloped?, buildings, etc etc. MICROSITE
    can happen in the rural arena.

    2. Increased heat storage in buildings. Are their building nearby this rural sites.
    A waste water treatment plant. A ranger station with 22 air conditioners. etc etc

    3. Anthro heat: Not such a big problem in rural sites, unless you consider sites located
    at power plants, damns, waste water treatment.

    4. Urban Greenhouse from urban pollution. Aerosols cool.
    5. Canyon radiation geometry. happens in both urban and rural settings.
    6. Evapotranspiration: non natural ground cover. Happens in both urban and rural.
    7. Shelter: Reduced turbulant tansfer. Trees, buildings, all reduce turbulant transfer.

    Lets put this another way. MICROSITE bias is AHI. Anthropomorphic Heat Island. And AHI
    could be more prevalent in urban settings.

    That’s testable,perhaps, Compare CRN12345 ratings versus population.

    Population might amplifiy certain aspects of it.

  61. Posted Oct 8, 2007 at 7:22 AM | Permalink | Reply

    #60 steven mosher:
    I have read (on this site) about UHI being caused by thermal inversion over large urban centres.
    Is that also a recognized cause of UHI?

  62. steven mosher
    Posted Oct 8, 2007 at 8:17 AM | Permalink | Reply

    Thermal Inversion is probably more readily construed as an effect. Simply Windshelter
    prevents turbulent transfer, But it is also referred to cause.

    The causes I listed are in Johnsen. I can get you a proper cite.

    For now, here is a nice little description

    http://www.nagt.org/files/nagt/jge/abstracts/George_v51n2.pdf

  63. Posted Oct 8, 2007 at 9:00 AM | Permalink | Reply

    Could the thermal inversion over a city could be present and affect station readings even if the station has no immediately local influences? For example, a station in a small neighbourhood park could be 100m from trees, buildings, etc (and have a CRN1 rating) but still be under the influence of the larger UHI effects of the city (such as thermal inversion).

    I really should read the references, but you’re making it too easy by just answering me directly.

  64. DeWitt Payne
    Posted Oct 8, 2007 at 9:18 AM | Permalink | Reply

    The need for external temperature data for satellite atmospheric temperature profile reconstructions is a separate issue from the calibration of absolute sensitivity of the microwave sounding units. There are more variables than there are measurements, i.e. the problem is ill-posed. That means there are an infinite number of temperature profiles that will match the measurements. So you have to start the calculation with a “best guess” and hope the process will converge to the correct profile. Represent all possible temperature profiles as a three dimensional surface with the z axis representing goodness of fit. A well-posed problem will have only one peak so you could start anywhere on the surface and find the correct solution. An ill-posed problem has multiple peaks, so different starting points lead to different solutions. A good best guess would be a radiosonde sounding. See this post for more detail.

  65. Kenneth Fritsch
    Posted Oct 8, 2007 at 10:42 AM | Permalink | Reply

    Re: #64

    DeWitt Payne, I may have made an incorrect interpretation of what your comment reference here actually said, but here is my view of it (from Post #303 in the thread “Unthreaded #21):

    Firstly, it states that IR (and not microwave sounding units [MSU]) instrumentation needs to be calibrated in various ways, one of which is with reliable surface station measurements.

    Secondly, it references Grant Petty’s comments on the problems of finding correct tropospheric (and stratospheric) temperature profiles due to the vertical overlap that the MSU instruments senses.

    In effect the subject changed from IR calibration to the measuring problems from “vertical overlap” that MSU instruments have, without addressing the issue of MSU calibration and the use of surface temperature measurements to accomplish this.

  66. steven mosher
    Posted Oct 8, 2007 at 11:30 AM | Permalink | Reply

    RE 63.

    I have a copy of Oke I can send you.

    you asked:

    “Could the thermal inversion over a city could be present and affect station readings even if the station has no immediately local influences? For example, a station in a small neighbourhood park could be 100m from trees, buildings, etc (and have a CRN1 rating) but still be under the influence of the larger UHI effects of the city (such as thermal inversion).”

    I really should read the references, but you’re making it too easy by just answering me directly.

    That’s ok. I find this much more enjoyable than the other crap. I really must stay away from those threads.

    To answer your question It would be possible to have a CRN1 site infected by UHI, primarily through
    the urban canopy effect. It would be modulated by wind speed and wind direction and surface
    roughness. So, you might have 100feet of pristine globe, but if your have a 1000 ft high structure
    upwind, the boundary layer wil change. There are lots of cool studies on this including
    a neat 3D simulation done of a city in the bubble study. One issue is this. On one side
    of the climate argument you have people arguing for green cities. They Cite the Horrible UHI.
    Then, on the other hand they claim that the stations in urban areas ar not biased.

    This is a fascinating question. and there are only two decent studies of the problem.
    Peterson ( which has flaws ) and Parker ( which has flaws ) I look at the claims being made

    A. UHI is horrible. Our cities are baking
    B. UHI doesnt bias the temp record.

    And I think. Dang. That needs explaining. I don’t think the result impinges on AGW theory in
    a significant way ( hey the ice is melting) but the current explanation of A&B is
    unsatisfying in my estimation

    There is also the phenomena of “cool parks” In UHI studies studies they find areas that are cooler
    than rest. Although the distribution of temp differenetial is skewed.

    Stupid example UHI_bias = ACTUAL +5C, -2.6C Skewed figures like that.

  67. SteveSadlov
    Posted Oct 8, 2007 at 11:47 AM | Permalink | Reply

    There seems to be this belief that UHI affects only a few tens or hundreds of feet vertically and horizontally from urbanization structures and artifacts. I question this. Certainly, while this may be true of IR reradiation from surfaces which are incident to insolation, it is not true for direct thermal flux due to electrical dissipation, climate control, and other work performed by humans and their machines.

  68. steven mosher
    Posted Oct 8, 2007 at 3:30 PM | Permalink | Reply

    RE 67. This disconnect is fascinating.

    I’ll illustrate the insanity by grabbing the other Horn.

    UHI is not real. Why are you envirowackos talking about green roofs, and urban parks,
    and measures to control the UHI that doesnt exist?

  69. SteveSadlov
    Posted Oct 8, 2007 at 3:45 PM | Permalink | Reply

    RE: #68 – Yup. Also, consider this. IR radiometric images show “cool” parks, lawns, etc. But that tells me nothing about the air temperature. Of course the emissivity and H2O content of lawns, shrubs, pools, well watered dirt, etc in the IR band is much lower than concrete. BFD. All the emitted IR from the stuff that does emit highly, plus all the other thermal flux, warms the general air parcel over a city (or even over a place like Happy Camp) and if you measure that air’s temperature, it’s going to be higher than what it would be in the true bona fide boonies. This is not rocket science and frankly, I am tiring of the fact that it is debated at all. Talk about denialism …. those who seek to discredit UHI and other anthropogenically delivered or influenced direct thermal flux effects are the real denialists.

  70. DeWitt Payne
    Posted Oct 8, 2007 at 3:48 PM | Permalink | Reply

    Re: #65

    Ken,

    I’ll to be clearer this time.

    Firstly, it states that IR (and not microwave sounding units [MSU]) instrumentation needs to be calibrated in various ways, one of which is with reliable surface station measurements.

    The original comment I replied to, #296 in UT #21, referred to the calibration of IR sensors on the satellites. While there are IR cameras and spectrometers on satellites, the atmospheric temperature (LS, MT and LT) from UAH and RSS are determined by analyzing the data from multi-channel microwave sounding units. The absolute sensitivity of those units is checked by a two point calibration, empty space for the low point and a controlled temperature warm panel on the satellite for the high point. So far no need for a ground based calibration.

    The question then is how do you convert observed microwave intensity to temperature? That is not a trivial problem. There is no unique solution. You have to start out with a guess and the guess has to be close. Where does the guess come from? Independent temperature measurements, i.e. radiosonde soundings.

    Secondly, it references Grant Petty’s comments on the problems of finding correct tropospheric (and stratospheric) temperature profiles due to the vertical overlap that the MSU instruments senses.

    In effect the subject changed from IR calibration to the measuring problems from “vertical overlap” that MSU instruments have, without addressing the issue of MSU calibration and the use of surface temperature measurements to accomplish this.

    Overlap increases the difficulty of the solution by effectively reducing the number of independent observations, it is not the source of the problem. The problem would still be ill-posed even if there were no overlap. In a sense it’s like climate modeling. The surface temperature record is not an input to the models, but the models must be tuned to reproduce the surface record (even if the modelers won’t admit it in public). It’s the same with the temperature profiles derived from the MSU data except the data being tuned to are radiosonde soundings.

    In general, it takes far more variables to accurately describe an arbitrary temperature profile T(z) than there are channels on a typical satellite sounding unit. This means that you have fewer measurements than unknowns, and the retrieval problem is underdetermined (or ill-posed). The problem is therefore not just that of finding any temperature profile that is consistent with the measurements; the real problem is of choosing the most plausible one out of an infinity of physically admissible candidates.

    You can’t just choose any profile. There must be some reference to independent measurements.

    Is that better?

  71. DeWitt Payne
    Posted Oct 8, 2007 at 3:49 PM | Permalink | Reply

    That’s try to be clearer… An edit function sure would be nice.

  72. Kenneth Fritsch
    Posted Oct 8, 2007 at 4:12 PM | Permalink | Reply

    Re: #70

    You can’t just choose any profile. There must be some reference to independent measurements.

    Is that better?

    I believe it is. If I may summarize your point, you are saying that since the MSU calibration/profile measurements have many choices for the “retrival” process that the surface temperatures are taken into account, however indirectly that may occur, in selecting a “realistic” profile. I follow your thinking on this issue, but can you link me to a specific paper or other scientific publication that makes this case?

  73. DeWitt Payne
    Posted Oct 8, 2007 at 8:29 PM | Permalink | Reply

    Re: #72

    Unfortunately, Petty doesn’t cite much. The entire bibliography consist of 8 books. Since he and Gerald Browning seem to agree, I assumed they were correct. I think you will have to go to the original literature. Here’s an abstract of a recent paper published by Christy, Norris, Spencer and Hnilo in J.Geophys.Res. Here’s the paper cited at the beginning (bottom) of the readme file below. It contains a discussion on the calibration of the sounding unit in the lab before launch. From what I could tell, it doesn’t seem to agree with Petty and Browning. I could easily be missing a lot, though. They do spend a lot of effort comparing their data to the radiosonde data.

    Or you could just ask Christy. His email address is at the bottom of the readme file.

  74. Posted Oct 9, 2007 at 6:07 AM | Permalink | Reply

    Steve McIntyre:
    I realize you’re busy but I hope you can find some time to clean up some loose ends that I have mentioned above. I will repeat them here for convenience:

    1. Update the “Vliet minus CRN1-2″ (see post #36 above);

    2. Post or email your yearly results used for the plots above (monthly results would be nice too if you have them);

    3. Confirm the reference period used for your difference plots (see post #40 above)

    4. Reconcile the “NOAA minus CRN1-2 TOBS (SM)” and “GISS Post-Y2K minus CRN1-2 TOBS (SN)” plots from above with the “US48: NOAA – NASA” plot in the Second Look at USHCN (see post #55 above);

    5. Consider adding trendlines (see post #19 above)

    Thanks.

  75. Kenneth Fritsch
    Posted Oct 9, 2007 at 9:13 AM | Permalink | Reply

    Re: #73

    DeWitt, I have done a rather extensive online search for information along these lines and have many of these same references. What I wanted was a reference(s) that uses something similar to your scenario in linking (or not) surface temperatures to the MSU satellite calibration. An email to Christy may be the best choice at this time.

    Anyway, I appreciate your thoughts on this subject as I tend also to look at these indirect connections in climate science as more important than they often appear on the surface of things. I do think that an important first phase of our discussions here of the surface station temperature trends and data should be whether we can independently compare it to other measurement methods — with independence being the key consideration.

  76. DeWitt Payne
    Posted Oct 9, 2007 at 9:56 AM | Permalink | Reply

    Re:#75

    Ken, This may be apples and oranges. The primary purpose of the satellite observations is weather forecasting. They had been up ten years before Christy and Spencer first published. So the question is, what algorithm is used to determine temperatures on a daily basis in more or less real time? That may be what Petty and Browning are talking about. I haven’t tried looking for that yet.

  77. Kenneth Fritsch
    Posted Oct 9, 2007 at 10:55 AM | Permalink | Reply

    Re: #76

    That the satellite temperature measurements were for short term forecasting is often stated in papers attempting to make adjustments for using the data for calculating long term temperature trends — the same problem that exists with the surface temperature measurements.

  78. DeWitt Payne
    Posted Oct 9, 2007 at 2:32 PM | Permalink | Reply

    Re:#77

    I think it is safe to say that the long term satellite anomaly measurements are not completely independent of the radiosonde measurements. There seems to have been considerable tweaking of the satellite data extraction method to get them to agree. The question then is: are the radiosonde measurements independent of surface temperature measurements? I don’t know. If radiosondes are calibrated to the local temperature before launch, then they aren’t completely independent. If they are just cross-checked without adjustment to see if they are working correctly, then they still might be considered independent because the trace back to NIST could be a separate path. Of course, the next question is: Are the radiosonde measurements traceable to NIST? I don’t know that either.

  79. Kenneth Fritsch
    Posted Oct 9, 2007 at 4:50 PM | Permalink | Reply

    Re: #78

    The recent articles that I have read on the radiosonde and MSU satellite measurements use the MSU measurements to correct radiosonde data sets due to the radiosondes being considered less stable and error prone than MSU. I could not find a Christy article that did a reanalysis of the radiosonde data based on breaks in the time series that corresponded to instrument and procedural changes in the radiosonde measurements and I thus link to the article below.

    http://www.gfdl.noaa.gov/aboutus/milestones/JRL_FY01_Q2_Milestone.html

    I think the “meant for use in short term weather predictions” and the limitations this places on use for longer term temperature trends is even more of problem for radiosondes than it is for MSU and surface temperature measurements.

  80. DeWitt Payne
    Posted Oct 9, 2007 at 6:11 PM | Permalink | Reply

    Re: #79

    Nice work by the NOAA guys. I’m assuming that’s the paper describing the RATPAC radiosonde database. If only Jones and/or Hansen were that open.

    I don’t see anywhere in the Christy et.al. 2000 paper linked above anything that would imply the use of a ‘best guess’ approach to calculating atmospheric temperatures from the microwave intensities. However, given the huge overlap in the T2 and T2LT channel weighting functions, and the significant overlap of T4 and T2, something like that had to be done at least once. Maybe it was done in the original laboratory calibration. You don’t by any chance have a copy of or have access to Mo, T., 1995: A study of the Microwave Sounding Unit on the NOAA-12 satellite. IEEE Trans. Geosci. Remote Sens., 33, 1141–1152. $35 is more than I want to spend when I’m not sure the article is what I’m looking for.

  81. DeWitt Payne
    Posted Oct 9, 2007 at 6:25 PM | Permalink | Reply

    OTOH, maybe Grody et.al. is the better choice. It’s only $9 and is the most up-to-date. If you already have it, I don’t see the need to buy it myself.

  82. DeWitt Payne
    Posted Oct 9, 2007 at 6:54 PM | Permalink | Reply

    Re: #75

    I found it (I think)! Better yet, it’s free. Conrath, B.J., Vertical Resolution of Temperature Profiles Obtained From Remote Radiation Measurements, J.Atmos.Sci., 1972.

    For a finite number of measurements m, the set of linear integral equations (2) will possess either no solution or an infinite number of solutions for deltaT(x). The effects of this non-uniqueness must be considered in examining the vertical resolution obtainable with a given set of measurements.

    He then proceeds to finesse this. I’m not sure I can (or want to) follow the math all the way.

  83. Posted Oct 10, 2007 at 10:00 AM | Permalink | Reply

    Steve McIntyre:
    I assume you have just been busy, but I have to ask you again to clean up some of the loose ends regarding this article. See my post #74 above.

  84. Sam Urbinto
    Posted Oct 10, 2007 at 6:13 PM | Permalink | Reply

    He’s been very busy with the Juckes topic. I can only hope I’ve lessened the workload and not added to it.

  85. Posted Oct 10, 2007 at 8:27 PM | Permalink | Reply

    I realize he’s busy with Juckes. That’s not an area that interests me so I’ve been trying to wait patiently.
    SteveMc, the loose ends (ie errors) in this article really should be fixed.

  86. Mike B
    Posted Oct 11, 2007 at 3:41 PM | Permalink | Reply

    Give it a rest, John. It’s been barely a week, and Steve was on vacation for some of that time.

    The only “error” that you’ve referred to is the spelling of your name. The other requests you’ve made are suggestions on how the analysis should be performed, which Steve may or may not think are appropriate.

  87. Posted Oct 11, 2007 at 4:20 PM | Permalink | Reply

    Mike B:
    Thanks for your input. The spelling of my name is not an error. The errors are:

    1. The final difference plot compares SteveMc’s results with *all* CRN12 stations versus mine with *only rural* CRN12 stations;
    2. The reference periods do not appear to be consistent since the average of the differences is not zero for the apparent reference period of 1890-2006;
    3. The “NOAA minus CRN1-2 TOBS (SM)” and “GISS Post-Y2K minus CRN1-2 TOBS (SN)” plots from above do not reconcile with the “US48: NOAA – NASA” plot in the Second Look at USHCN thread;

    I also made a simple request for his data last Friday. It should only take a minute to send the data file.

    Steve McIntyre — any chance of getting that data file soon?

  88. Dennis Wingo
    Posted Oct 11, 2007 at 8:27 PM | Permalink | Reply

    Steve

    In your Google targeted ad thingy on the lower right of your homepage there is this offer for a 100 year archive of weather data!

    http://weather-warehouse.com/?gclid=CN2wl5apiI8CFQacggodaGXOvA

    Just thought it was funny.

  89. cce
    Posted Oct 13, 2007 at 9:41 PM | Permalink | Reply

    He’s apparently too busy demanding data, corrections, and explanations to provide data, corrections, or explanations.

  90. Willis Eschenbach
    Posted Oct 14, 2007 at 2:16 AM | Permalink | Reply

    cce, I’m actually quite happy that Steve M. runs on his own schedule, and not yours. Instead of doing what you want him to do, he’s been updating the Graybill bristlecone series, and simultaneously verifying the Starbucks Hypothesis. You don’t like the speed with which he’s fulfilling your requests? Live with it. John V. has made it clear what he thinks the errata are, and they will get addressed when Steve gets around to it. Until then, I offer you three options:

    1) Lead

    2) Follow

    3) Get out of the way

    Note that “Get snarky and whine because the world doesn’t run on cce’s schedule” is not on the list of options. If I were Steve, all your comment would do is make me go off and work on something else for a while … but I’m a reformed cowboy, and Steve is a gentleman, so I suspect your comment won’t affect him either way.

    w.

  91. cce
    Posted Oct 15, 2007 at 1:07 AM | Permalink | Reply

    I like option 4:

    1) Don’t care one way or the other since nothing will come of this anyway.

    I just like pointing out irony. I wish him luck finding Waldo. Be sure to check Starbucks.

  92. Posted Oct 16, 2007 at 6:46 AM | Permalink | Reply

    Steve McIntyre:
    Thanks for updating the “NOAA minus NASA” plot in A Second Look at USHCN Classification.

    I have to ask you again to update your “Vliet minus CRN12 TOBS (SM)” plot above. I sent you the data on October 5th. I have two reasons to ask for the change:

    1. It is not a valid comparison (rural CRN12 versus all CRN12) of my results to yours and therefore leaves an incorrect impression about the match;
    2. Clayton B looked at the effect of OpenTemp’s averaging radius (here) — I would like to see how my OpenTemp results with a 1000km radius compare to your results using Akima for gridding

    If you can’t update the “Vliet” graph above, please take it down and send me your data so I can do the comparison myself. Thanks.

    Steve: Fair enough. Will do some time soon.

  93. Posted Oct 22, 2007 at 8:54 PM | Permalink | Reply

    I download the excellent “GetData Graph Digitizer” today:

    http://getdata-graph-digitizer.com/index.php

    I tried it out on Steve McIntyre’s “USA Calculated from CRN1-2 TOBS” to compare against my OpenTemp CRN12 TOBS results. The digitization was not perfect and any digitization errors are my own. Here’s a graphical comparison:

    I overlaid a five-year trailing average and a linear trend on the complete data set. Unforunately, I could not easily do a year-by-year difference because my digitized years did not always line up with the actual years. (Misalignment is visible in the graph above).

    Overall I’d say the correlation is quite good. It doesn’t raise any concerns about the OpenTemp algorithm in my mind.

  94. Steve McIntyre
    Posted Oct 22, 2007 at 9:38 PM | Permalink | Reply

    #93. John, a digital version (which looks similar to this graphic and should be the same) is at CA/data/station/watts/recon.crn12.dat

    The point in this post is that there is perfectly good smoothing software readily available in standard packages. There’s nothing particular magical about any of these smooths, which simply produce varying weights of the underlying proxies, and, as shown in the script above, the calculations can be done in a few lines. I don’t understand why you wouldn’t just use this sort of standard software where it is available.

    As I’ve said endlessly, my concern is with the ROW data and methods where one does not see the same relationship between the 1930s and 2000s. IT seems to me that your time would be spent more fruitfully trying to come to grips with some of the ROW issues.

  95. Posted Oct 22, 2007 at 9:48 PM | Permalink | Reply

    Steve McIntyre:
    Thanks.

    The point in this post is that there is perfectly good smoothing software readily available in standard packages.

    Absolutely. The only problem is the overhead and technical skill required to install and learn these packages. My goal is to make a very simple program that anyone can use.

    As I’ve said endlessly, my concern is with the ROW data and methods where one does not see the same relationship between the 1930s and 2000s. IT seems to me that your time would be spent more fruitfully trying to come to grips with some of the ROW issues.

    I do want to get to the ROW data, but not prematurely. There is a wealth of information in the USA48 data and much knowledge can be gained from that. The site quality ratings permit a number of new analyses with one variable (micro-site issues) at least partially constrained. It is possible to differentiate and quantify micro-site and UHI issues. Using stations that are known to be of high quality, the regional variation can be estimated for areas with few stations. Tmin and Tmax trends can be investigated. Etc.

  96. Posted Oct 22, 2007 at 9:49 PM | Permalink | Reply

    I should add that I also wanted some independent verification of OpenTemp before going much further with it. A comparison against your results will help.

  97. Steve McIntyre
    Posted Oct 22, 2007 at 11:38 PM | Permalink | Reply

    The only problem is the overhead and technical skill required to install and learn these packages. My goal is to make a very simple program that anyone can use.

    Installation of the akima package took about 3 seconds and one command. Yes, there is a little overhead involved in learning R, but it is a very common statistical language that is designed for use by anyone for multiple purposes.

  98. Willis Eschenbach
    Posted Oct 23, 2007 at 12:29 AM | Permalink | Reply

    I’d like to support Steve M’s call for the use of the R computer language. It is fairly fast to learn, immensely powerful for dealing with large datasets, and specifically designed for statistics (it’s built on the core of “S”, a language made for statistics from the bottom up). I resisted learning it for a long time, but I’m glad I finally put forth the effort.

    Perhaps the best part of it is that you rarely need to re-invent any wheels. There are standard, tested functions for virtually any statistical manipulation you want to undertake. It gives very good control over the graphical representation of data. I’m not very good with it yet, but it’s a jewel. I still use Excel for lots of quick-and-dirty stuff, but when it gets big and complex, I turn to R.

    Finally, if you want to develop something like OpenTemp, you can write it up and test it as an R function, and its principles, strengths, and weaknesses will be totally transparent.

    Just my $0.02 worth …

    w.

  99. Posted Oct 23, 2007 at 4:55 AM | Permalink | Reply

    Could someone summarize how the problem of geographical distribution of CRN12 sites was addressed? Thanks

  100. Posted Oct 23, 2007 at 8:05 AM | Permalink | Reply

    #99 David Smith:
    The problem with a small number of stations (and thus poor geographic coverage) was primarily an issue with *rural* CRN12 stations. There are quite a few total CRN12 stations.

    I did some work with rural CRN12 vs rural CRN123 here:

    http://www.climateaudit.org/?p=2124#comment-147568

    The coverage is much better when the CRN3 stations are included.

  101. JerryB
    Posted Oct 23, 2007 at 8:53 AM | Permalink | Reply

    FWIW, a discussion of grid box sizes, and shapes, may be found here.

  102. stan palmer
    Posted Oct 23, 2007 at 10:39 AM | Permalink | Reply

    re 93:

    I overlaid a five-year trailing average and a linear trend on the complete data set. Unforunately, I could not easily do a year-by-year difference because my digitized years did not always line up with the actual years. (Misalignment is visible in the graph above).

    I’ve always wondered what the trend line in such diagrams is supposed to mean. What phsyical interpretation should it be given? My own prejudice is that a trend line is useless. It presupposes a linear trend without justification.

  103. Steve McIntyre
    Posted Oct 23, 2007 at 11:37 AM | Permalink | Reply

    If you go to the online script for this, I simplified the above calculations into a function that calculates the geog weighted average (using the akima smooth) from a set of USHCN identifications index_good and a USHCN collated version. Doesn’t get any simpler. You can adapt the script to get the contoured maps for individual years if you want.

    recon.crn12=ushcn_wt(index_good,ushcn=ushcn.tobs)
    ts.plot(recon.crn12)

  104. Clayton B.
    Posted Oct 23, 2007 at 3:54 PM | Permalink | Reply

    Finally, if you want to develop something like OpenTemp, you can write it up and test it as an R function, and its principles, strengths, and weaknesses will be totally transparent

    Please explain how this is different than other code?

  105. Conard
    Posted Oct 23, 2007 at 10:46 PM | Permalink | Reply

    re: 94

    Like JohnV I was interested in the Hansen reverse-engineering project and thought that this area (software development) might be where I could contribute. In this vein I have the following questions:

    1) Is there work than can be done by the non-statistician to enable the statistician to focus more on the design and interpretation of the analysis. If there is this type of work, and there is a will to support it, then software “friendlier” than R may have advantages.

    2) Are there tasks that occupy the time of principal researchers where purpose built, or integrated software can play a significant role? For example I have seen many requests for data cut this way and that, data sets to be correlated, etc.

    3) JohnV has certainly demonstrated that there is sufficient good will and expertise that can be harnessed towards these ends.

    c

  106. Steve McIntyre
    Posted Oct 23, 2007 at 10:56 PM | Permalink | Reply

    #105. Analysis of ROW station data has been barely scratched.

2 Trackbacks

  1. [...] McIntyre, Steve (October 4, 2007). “Gridding from CRN1-2″. Climate Audit. http://www.climateaudit.org/?p=2145. Retrieved [...]

  2. By Surface Stations « Climate Audit on Jul 31, 2012 at 9:05 AM

    [...] I had done my own initial assessment of this a few years ago, I had used TOBS versions and am annoyed with myself for missing this. [...]

Post a Comment

Required fields are marked *

*
*

Follow

Get every new post delivered to your Inbox.

Join 3,184 other followers

%d bloggers like this: