A Second Look at USHCN Classification #2

Continuation

306 Comments

  1. Anthony Watts
    Posted Sep 30, 2007 at 9:38 AM | Permalink

    OK then, now where were we?

    OK Mosh in 624, you can’t place direct links to these forms, as they are “order” links. The are transient. You have to give the station name and the month.

    But I’m not surprised at the edit. On one of the Marysville forms, I saw a hand written note “data from the airport” when apparently they had some equipment trouble.

    And just looking at random months, it appears as much as 25-33% of data is missing for the last few years at Marysville…lots of FILNET goin on.

    Yes USHCN, “a high quality” network.

  2. Anthony Watts
    Posted Sep 30, 2007 at 9:46 AM | Permalink

    For this thread continuation, here is the link at NCDC that Kristen Byrnes found that gives access to PDF files for B91 observer forms with original COOP data as written by the observer.

    http://www7.ncdc.noaa.gov/IPS/getcoopstates.html

    Links that appear with the PDF’s at NCDC are transient, generated as “orders” so be sure to save and PDF file you download, and then upload it to the appropriate station at

    http://gallery.surfacestations.org/main.php

    by navigating to the station first then uploading the file(s).

  3. Anthony Watts
    Posted Sep 30, 2007 at 9:58 AM | Permalink

    Figured I’d look in my own back yard where it all started with paint first. Here is Chico University Farm

    http://gallery.surfacestations.org/main.php?g2_view=core.DownloadItem&g2_itemId=28535

    13 days missing for July…thats 13/31*100 = 41.9% of data missing!

    With my knowing the schedule there you can bet that every weekend is missing in every record, so best case for Chico is 8/31*100 = 25.8% missing data.

    Lots and lots of FILNET going on.

  4. Anthony Watts
    Posted Sep 30, 2007 at 10:18 AM | Permalink

    And now let’s do a spot check of our two favorite station pairs, Orland and Marysville:

    Orland July 2007 31 days of data
    http://gallery.surfacestations.org/main.php?g2_view=core.DownloadItem&g2_itemId=28549

    Marysville July 2007 14 days of data
    http://gallery.surfacestations.org/main.php?g2_view=core.DownloadItem&g2_itemId=28542

    The difference? The guy at Orland I talked to takes his job as observer seriously, as do many amateur observers.

    We may need to create a new quality classification for stations based on how much missing data there is on a regular basis.

  5. steven mosher
    Posted Sep 30, 2007 at 10:35 AM | Permalink

    RE 1-4..

    I was thinking that the B91s would make a great check for the TOBS issue of
    Double counting min/max issues and I found on on my first try.. If I remember
    it was Orland august18 1991.

    Ever wonder what the records from 1934 look like.

    it’s turtles all the way down.

    I finished a write up of my Class5 study. Somebody needs to double check it.

    Short version: Warming with ALL 1221 stations. .5-.6C
    Remove 60 ( 58 is the exact number i think) class5s. Warming drops by .05-.06C
    For every year since 1880. So, if USHCN can adjust for MMTS, then can drop these
    60 stations. Plus, I find that the UNSURVEYED sites will be warmer than those already
    surveyed..

  6. Chris Harrison
    Posted Sep 30, 2007 at 11:02 AM | Permalink

    With my knowing the schedule there you can bet that every weekend is missing in every record, so best case for Chico is 8/31*100 = 25.8% missing data.

    From my days working on control systems for Gas distribution I know that power consumption is much higher on work days than at the weekend. If weekends are being interpolated from week days we might have yet another slight warming bias were the stronger week day UHI effect is polluting the weekend data.

    I wonder if a comparison of weekend/holiday vs. weekday energy usage against weekend/holiday vs. weekday temperatures might be an interesting way to quantify UHI.

  7. steven mosher
    Posted Sep 30, 2007 at 11:29 AM | Permalink

    Anthony, I think Peterson,Karl easterling will probably make Filnet code available.

    recal when we discussed the sine-like nature of the Tmax and the damped Tmin.
    I wonder how they infilled dailies?

  8. JerryB
    Posted Sep 30, 2007 at 12:10 PM | Permalink

    FWIW, with max/min equipment, a two day gap followed by an observation would
    make that observation a 72 hour max/min observation, so it would be a three day
    problem, not just a two day gap.

  9. IanH
    Posted Sep 30, 2007 at 12:23 PM | Permalink

    This supports my assumption over in the TOBS thread. In almost all cases the temperature at the OBS time is higher than the min when the thermometer is reset, and so there is no need for any adjustment to 24hr min/max to account for TOBS. I accept that others were considering the MMTS hourly measurement but that is a relatively trivial determination for a daily min/max. The only TOBS for the historical record would be around days when OBS time changed, when the record could simply be dropped.

  10. JerryB
    Posted Sep 30, 2007 at 12:39 PM | Permalink

    Re #9,

    IanH,

    Wrong thread for that comment. You might post it in the TOBs thread.

  11. bmcburney
    Posted Sep 30, 2007 at 12:47 PM | Permalink

    If it is shown that observations are less likely to
    take place on weekends, I wonder if they are also less
    likely to take place on bad weather days.

  12. Murray Duffin
    Posted Sep 30, 2007 at 1:04 PM | Permalink

    I think looking only at the 1900 to 2006 trend in the various differences plots obscures something stranger going on. It seems to me that if the methods of adjustment were consistently different, then the the trend should be consistent, but that is not what we see.
    CRN5-CRN1 TOBS Trend 1900 to 1940+ down, 1940+ to 1980 up, then flat.
    CRN5-CRN1 TOBS E of 95W practically no trend 1900-1970+, then steeply upwards. Big hole in difference 1990+-
    CRN5-CRN1,2 TOBS E of 95 W Same trends, but not as pronounced, and hole in 1990+-, as might be expected.
    CRN5-CRN1 TOBS W of 95W 1900-late 1930s down, late 1930s-late 1960s up, then flat.
    CRN5-CRN1,2 TOBS W of 95W 1900-1950+ down, 1950+ to 1970+ up, then flat.
    US48 NASA-CRN1,2 1900-1950+ flat, 1950+ – 1979 down, then up sharply.
    US48 MOAA – NASA 1900-late 1930s up, late 1930s to 1979 down, then up sharply.
    There is some tendency for the difference trend to follow the temperature trend. Ca 1980 to present is flat in 3 cases, up sharply in 3 cases. Seems very strange to me.
    1980 to present flats are in total USA or W of 95W. 1980 to present upwards are E of 95W or NASA-CRN1,2, or NOAA-NASA. What happened ca 1980 that tilted NASA up vs CRN1,2, and tilted NOAA up vs NASA??? Murray

  13. Posted Sep 30, 2007 at 1:18 PM | Permalink

    Here is the Port Gibson, MS temperature history per GISS, as stored by me at Anthony’s surfacestations.org site on August 30, 2007.

    Here is the GISS plot currently available from the GISS website.

    What happened to the 1885-1957 portion of the record shown in the August 30, 2007 image?

    Also, it looks like several temperatures have changed in the 1960s and 1970s. It looks the new version is slightly cooler than the August 30 version.

  14. steven mosher
    Posted Sep 30, 2007 at 1:33 PM | Permalink

    RE 10.

    Don’t know if you caught that Jerry but on my first try I found a B91 that showed the
    temp at Observation time at 8AM as being lower than the Tmin. Kinda cool. Anyway, the observer
    struck through the Tmin and put the Observed temp in the box. If you like I can go find it
    again.

  15. Posted Sep 30, 2007 at 1:48 PM | Permalink

    Saint Joseph, LA looks like a CRN1 candidate.

  16. Phil
    Posted Sep 30, 2007 at 2:45 PM | Permalink

    #4 Anthony Watts says on September 30th, 2007 at 10:18 am:

    (snip)…We may need to create a new quality classification for stations based on how much missing data there is on a regular basis.

    There is some data quality historical info at the following urls:

    http://cdiac.esd.ornl.gov/ftp/ndp019r3/qa94mean.asc
    http://cdiac.esd.ornl.gov/ftp/ndp019r3/qa94mxmn.asc

    It may not have been updated since 1994.

  17. Clayton B.
    Posted Sep 30, 2007 at 3:17 PM | Permalink

    RE: #13 Port Gibson,

    It looks like all data before 1958 is gone??

    The following image was posted (by me) in the original thread. It was based on JohnV stations.csv (GHCNv2).

    Same chart with only 1958 to present:

    Here is the station history (USHCN):

  18. steven mosher
    Posted Sep 30, 2007 at 3:26 PM | Permalink

    A while back I mentioned some issues with the USHCN daily records. Files I had downloaded before
    are Now are missing huge chucks of data and/or the data looks FUBAR. I have not been back since that
    posting.

  19. JerryB
    Posted Sep 30, 2007 at 3:43 PM | Permalink

    Re #14.

    steven,

    I would not ask you to dig for it, but if you happen to come accross it, or
    other similar instances, it might be good to mention a few of them in the
    TObs thread with the specific temps involved.

  20. steven mosher
    Posted Sep 30, 2007 at 3:53 PM | Permalink

    Didnt Hansen promise a simplified version in a couple weeks?

  21. Kenneth Fritsch
    Posted Sep 30, 2007 at 5:56 PM | Permalink

    I have concluded some rework of random groups and CRN12 and CRN5 trends for the period 1960-2005 using the USCHN Urban data set.

    I included a geographical weighting by calculating trends for all 1200+ stations from the USCHN Urban data set to determine a state by state temperature trend over the period 1960-2005. The normalizing or weight factor was then determined by dividing each of all the state trends by the average state trend. For each of the stations in the random and CRN groups I found the state in which it resided and than took the corresponding state weight factors of all the stations in the groups and calculated an average weight for that group.

    Using this weighting process did not significantly affect the previously measured difference between the CR12 and CR5 groups, but did noticeably reduce the variations in the random groups.

    For the random groups, I randomly selected a 53 station (corresponding to the CR12 group number of stations) and a 58 station (corresponding to the CR5 group number of stations) from the population of stations included in the CRN1 through CRN5 categories. I then took the differences of 34 sets of paired random groups and determined a distribution for both the unadjusted sets and the sets adjusted using the geographic weighting. The results are presented here:

    Without Geographic Weighting:
    0.0000 to 0.0030 = 0.44
    0.0031 to 0.0060 = 0.35
    0.0061 to 0.0090 = 0.09
    0.0091 to 0.0120 = 0.09
    Greater than 0.01201 = 0.03

    With Geographic Weighting:
    0.0000 to 0.0030 = 0.53
    0.0031 to 0.0060 = 0.29
    0.0061 to 0.0090 = 0.12
    Greater than 0.0091 = 0.06

    The calculated trends and differences for CR12 and CRN5 were as follows:

    Without Geographic Weighting:
    CRN12 = 0.0377
    CRN5 = 0.0477
    CRN5 – CRN12 = 0.0100

    With Geographic Weighting:
    CRN12 = 0.0378
    CRN5 = 0.0480
    CRN5 – CRN12 = 0.0102

    If we include the geographical weighting and use the random groups to estimate statistical significance, the difference between the CRN12 and CRN5 shows a significant or near significant difference using a rejection level of 5% or less. I would continue to warn against using long term differences in comparing snapshot shot quality differences and suggest that nearer term differences are probably more important – if than can be accurately measured.

    When I earlier reported that when I did the same comparison between CRN12 and CRN5 for the same period using the John V derived data set I found no difference, John V suggested that using a geographical average in place of the arithmetic average might show the difference. If I redo that calculation using my state by state geographical weighting I will continue to see no difference. This development needs more attention and work.

  22. steven mosher
    Posted Sep 30, 2007 at 6:31 PM | Permalink

    RE 21. So you are reporting out a difference in trend of About .01C/year over the
    period of 1960 to 2005? Am I Understanding that correctly Kenneth?

    I think I am starting to get the grasp of your approach.

  23. Kenneth Fritsch
    Posted Sep 30, 2007 at 6:46 PM | Permalink

    Re: #22

    I am reporting the trends in degrees F per year (which I had previously noted but should have done so again in the above post. The differences are in degrees F — between trends. That 0.01 degree F is approx. 0.0056 C difference per year.

  24. Posted Sep 30, 2007 at 6:51 PM | Permalink

    Chris Harrison says:
    September 30th, 2007 at 11:02 am,

    Weekend decline in electrical power consumption is also a fact.

  25. steven mosher
    Posted Sep 30, 2007 at 7:15 PM | Permalink

    Whew! that’s more reasonable so, .0056C per year over 45 years.

    JohnV hows that line up with your expectation model?

  26. Posted Sep 30, 2007 at 9:05 PM | Permalink

    I notice that Monticello, Mississippi also lost the 1905-1953 portion of its time series. This happened at some point in the last four weeks.

    Also, some of its 1960s and 1970s data appears to be changed, in what appears to be slight cooling of that period.

    It’s like we’re witnessing history in the making.

  27. Clayton B.
    Posted Sep 30, 2007 at 9:28 PM | Permalink

    #26,

    I believe Steve Mc mentioned a data change a few weeks ago. Not sure this is same thing or not.

    From here:

    September 2007: The year 2000 version of USHCN data was replaced by the current version (with data through 2005). In this newer version, NOAA removed or corrected a number of station records before year 2000. Since these changes included most of the records that failed our quality control checks, we no longer remove any USHCN records. The effect of station removal on analyzed global temperature is very small, as shown by graphs and maps available here.

  28. Posted Sep 30, 2007 at 10:35 PM | Permalink

    Kristen Byrnes said…

    I started with your and Steve’s favorite and the first one on John V’s CRN 5 list: Univ. of Tucson #1. John V’s data goes up to March 2006 so that’s where I started (it was off by 3 tenths). Then I went back a year to March 2005, John’s spreadsheet was blank for that month but the report is there, no days missing (same with Wickenburg, AZ in Jan 1999). Then I went back another year to March 2004, it was off by 5 tenths.

    This really got me interested, so I started looking at the data myself. The first thing I did was build a spreadsheet for calculating the average temperature. It might be useful to others so here’s a link:

    http://www.opentemp.org/_tools/ushcn/USHCNObserverRecord.xls

    Note:
    The spreadsheet calculates the average with different levels of rounding, because I thought the differences found by Kristen may have been caused by rounding. (It turns out the rounding wasn’t necessary).

    Next I started looking at the observer records. On my first attempt I used USHCN station 028817 and nothing was matching. I then went back to Anthony Watts’ spreadsheet of stations and realized that the Tucson UofA site is 028815. Working with 028815 I got much better results. The list below shows the GHCN Unadjusted, USHCN Raw (Areal), and Observer Record temperatures:

    Mar2006: 15.6 / 15.6 / 15.6
    Feb2006: 15.2 / 15.2 / 15.2
    Mar2005: ???? / ???? / 15.2
    Mar2004: 19.9 / 19.9 / 19.9

    I stopped checking after this.
    For every month with USHCN values, the GHCN value is exactly the same.
    For some reason March 2005 does not have a value in either USHCN or GHCN.

    From this limited set of data, it looks like GHCN Unadjusted is the same as USHCN Raw (Areal). I can not explain why March 2005 is missing from USHCN and GHCN.

    Kristen, please double-check these results as they are different than yours.
    The missing stations from USHCN are a mystery but I am still comfortable stating that GHCN Unadjusted and USHCN Raw (Areal) match.

    The observer records and my working spreadsheets are available here:
    http://www.opentemp.org/_data/USHCNObserverRecords.zip

  29. Posted Sep 30, 2007 at 10:39 PM | Permalink

    Chris Harrison (#622 in original thread) said:

    John V,
    Instead of gridding an area with regular sized rectangles or triangles, why not do the following.

    1. Imagine the stations are the vertices of an irregular polyhedron with triangular sides. i.e. draw lines between the stations until the area is split up into triangles, each triangle having one station for each of its three corners.
    2. Now compute the temperature for each triangle by taking the average of the three temperatures for the stations at the three corners.

    3. Divide the temperature by the area of the triangle (remembering that the triangle is on the surface of a sphere when computing the area).
    3. Sum the weighted temperatures and multiply by the total area to get an average temperature.

    The polyhedron can be created computationally by starting at an arbitrary station, finding the closest two stations to form the first triangle, then expanding out recursively from each of the three edges of this triangle adding a point to make a new triangle.

    Chris, I did consider this approach. I had to reject it because every month has a different subset of stations reporting. The stations would need to be tesselated differently every month, and identifying trends would become very difficult.

    I believe that a suitably dense mesh of triangles will give the same results with a lot less confusion.

  30. Posted Sep 30, 2007 at 11:31 PM | Permalink

    steven mosher #25:

    Whew! that’s more reasonable so, .0056C per year over 45 years.
    JohnV hows that line up with your expectation model?

    My simple expectation model (posts #380 and #540 in the original thread) does not give a slope . It gives a total expected temperature change if a CRN5 station started as a CRN1 station. It’s a worst-case bias *if* my model assumptions are correct.

    The range I came up with was 0.32 to 0.49 degC. The observed difference in total warming between CRN5 and CRN12 (rural) in my analysis was ~0.35C. Kenneth’s value of 0.0056C/year for 45 years works out to 0.25C total (the remaining 0.10C is probably in the other 50+ years).

  31. Chris Harrison
    Posted Oct 1, 2007 at 4:10 AM | Permalink

    John V

    I believe that a suitably dense mesh of triangles will give the same results with a lot less confusion.

    How do you plan on dealing with empty triangles (triangles with no stations in them)?

  32. JerryB
    Posted Oct 1, 2007 at 4:41 AM | Permalink

    Re #28,

    “… I am still comfortable stating that GHCN Unadjusted and USHCN Raw (Areal) match.”

    FWIW: on converting USHCN “raw” temps from F to C, and rounding to
    tenths, and comparing the monthly numbers to GHCN v2.mean “raw” numbers,
    omitting the partial year data for 2006, I found exact matches except for
    one month at one station: January 2003, at Goldsboro. 3.6 C vs 4.3 C.

  33. Steve McIntyre
    Posted Oct 1, 2007 at 6:51 AM | Permalink

    You get these sorts of areal average calculations in ore reserve calculations in mining. “Kriging”, a term now used in spatial statistics, comes from mining. My own views on spatial average are very much influenced by own practical experience.

    Ore reserve calculations have, over the years, developed techniques that have been applied in “robust” statistics – e.g. trimmed means, nugget effects; where you “cut” extreme values to (aay) 2 sigma. This has the effect of shrinking the area of influence of extreme values.

  34. Posted Oct 1, 2007 at 7:39 AM | Permalink

    #31 Chris Harrison:
    My plan is to nest the triangles into a hierarchical structure. At the largest level (level 0) there are the 20 triangles of an icosahedron. Each of those 20 triangles is sub-divided into 4 triangles to make level 1 with 80 triangles. And so on. I will probably go as deep as level 6 with 81920 triangles. Each will have an area of 6226 square kilometers.

    To calculate an average temperature for a given month, I will:

    1. Calculate the average temperature for all level 6 cells with reporting stations
    2. Combine level 6 cells to get averages for level 5 cells (each contains four level 6 cells)
    3. Combine level 5 cells to get averages for level 4 cells (each contains four level 5 cells)

    7. Combine level 1 cells to get averages for level 0 cells (each contains four level 1 cells)
    8. At this stage, every level 6 cell has a direct temp or an inherited temp (from higher levels)
    9. Calculate the overall (or regional) average from the level 6 direct or inherited temperatures

    It’s early — I hope that makes sense.

  35. Posted Oct 1, 2007 at 7:44 AM | Permalink

    #32 JerryB:
    You probably said that before. Thanks for confirming again.
    There is still the mystery of the missing months — any ideas Kristen?

  36. Kenneth Fritsch
    Posted Oct 1, 2007 at 8:59 AM | Permalink

    Re: #34

    John V you are entering land that needs to be explored but where I dare not go with my limited abilities with Excel. I hope you do a study whereby you determine what is gained by using the more complicated algorithms.

  37. SteveSadlov
    Posted Oct 1, 2007 at 9:08 AM | Permalink

    RE: #12 – It is indeed strange. Also, look at the big shifts – late 1910s, early 1950s. Something odd there too.

  38. Posted Oct 1, 2007 at 9:20 AM | Permalink

    #36 Kenneth Fritsch:
    Although the algorithm seems complex, it can be easily automated and that’s the key for me. I don’t want to spend days partitioning stations from around the world (that is the eventual goal) into grid cells. The simple rectangular grids are ok, but have two disadvantages:
    – irregular size and/or shape
    – all aligned in the same direction

    A high-resolution triangular grid overcomes station-clustering problems since the cells are small. The hierarchical structure overcomes problems with sparse coverage by averaging (but only where necessary).

    The method is similar to your state-by-state geographic weighting, but every “state” is a triangle.

  39. SteveSadlov
    Posted Oct 1, 2007 at 9:28 AM | Permalink

    Who here trusts US temperature data (both raw and TOB adjusted) prior to 1925? For example, who would hang their hat on “there was x deg F / decade warming during the 20th century?”

    Something very unusual seems to have happened sometime between 1915 and 1925. 0.1 deg F of it may be attributed to a TOBs adjustment shift. But not all of it. Prior to this radical move upward in apparent temperature, there was a very cold year, then after it a radially warm year. There is no such drastic contrast anywhere else in the record. It essentially defines the range of the record. I personally don’t trust this to have been a natural occurrence.

  40. Kenneth Fritsch
    Posted Oct 1, 2007 at 9:29 AM | Permalink

    Re: #34

    John V, if 20 triangles makes an icosahedron what do we call a figure made up with 81920 triangles. That graphic is very appropriate to the point you are making. A picture worth 81920 triangles is worth as many words — or something to that effect.

  41. Posted Oct 1, 2007 at 9:44 AM | Permalink

    #39 SteveSadlov:
    If you are referring to the analyses I’ve done with CRN5 and CRN12R stations, I think part of the problem in the early years is the very small number of stations. The GISTEMP results do not show such wild temperature swings in the early years:

    (This is not a defense of GISTEMP — I just knew where to find a GISTEMP graph)

    The error bars will definitely be larger in the early years.

  42. Stan Palmer
    Posted Oct 1, 2007 at 10:35 AM | Permalink

    re 38

    A high-resolution triangular grid overcomes station-clustering problems since the cells are small. The hierarchical structure overcomes problems with sparse coverage by averaging (but only where necessary).

    However, in effect, it would still give more weigting to stations in sparse areas as opposed to those in more dense areas. So the relatively fewer stations in the US west would be given more weight than the stations in the more densely served US south-east.

    Is this an issue that needs to be addressed.

  43. Posted Oct 1, 2007 at 10:53 AM | Permalink

    Stan Palmer:
    Unfortunately nothing can be done about a lack of stations in some areas (and thus stations in sparse areas being given more “weight”). New stations can not be added.

    In the analysis, areas with few stations will have larger error bars. My plan is for every reading to have one or more error distributions that are carried through all calculations.

  44. steven mosher
    Posted Oct 1, 2007 at 11:24 AM | Permalink

    If you had one station in the us where would you put it?

    How well would it measure the “average US temp”?
    How well would it measure the trend?

    If you added a second station, where would you add it? 3rd? 4th, 5th?

    That’s if you PLanned the network…

    Consider this: you dont want stations close to each other. You want to MAXIMIZE the number stations,
    while simulataneously Normalizing the distince beween stations and the closest neighbors.

    Total area coverage only matters if you are concerned about the Absolute average. I’ll play with this
    thought and suggest something

  45. Anthony Watts
    Posted Oct 1, 2007 at 11:25 AM | Permalink

    RE39 Steve “I personally don’t trust this to have been a natural occurrence.”

    Don’t be so sure. I have seen quasi-stationary patterns persist in the circumpolar vortex on occassions that can act very much like a climate switch for north America. It’s not out of the realm of plausibility at all to have two back to back years where such a pattern changes state.

  46. Anthony Watts
    Posted Oct 1, 2007 at 11:34 AM | Permalink

    Re41 The GISTEMP results do not show such wild temperature swings in the early years:

    Again I’d point out that standard thermometer exposures/shelters weren’t well implemented until around 1900. Yes GISS doesn’t show it, but remember it has been heavily filtered and homogenized.

    A microsite bias such as a thermometer going from hanging under a tree to a shelter is not something that can be adequately adjusted for, IMO without some historical records and some application of empircal analysis to the measurement environment change. Much of those changes don’t even have a record or the occurance, but you can often see the effect when the data “settles down” and the swings are minimized.

    Thus I think it is prudent to use data from 1900 forward since the hit/miss shelter issue of the 1800’s cannot be quantified.

  47. Steve McIntyre
    Posted Oct 1, 2007 at 11:36 AM | Permalink

    #44. you use a hexagonal pattern.

    In drilling ore bodies, you are faced with an identical situation because you want to get the maximum coverage of the ore body for the minimum number of holes.

    Applying that analogy, mining engineers typically drill holes along sections so they can plot transects and level plans. The sections are perpendicular to the ore body.

    So thinking out loud here, what I’d suggest in connection with weather patterns is choosing a grid that is perpendicular to the isochrons or isotemps. I’d try to find about 20-30 really GOOD stations more or less arranged in a grid. Then I’d like to examine things along the transects.

  48. Posted Oct 1, 2007 at 11:53 AM | Permalink

    “If you had one station in the us where would you put it?”

    Midway between Omaha and Kansas City – say Auburn, NE, swing a 600KM radius and you would cover an area with minimal UHI “effect”. ‘Course, the ‘Blue Norther’ effect might make for cooler winters but irrigation and other land use change effects would be minimal.

    I specialize in answers to hypotheticals.

  49. Dennis Wingo
    Posted Oct 1, 2007 at 12:20 PM | Permalink

    Steve

    I just found some interesting temperature data taken by the USG while researching other things. It is from the Hanford nuclear facility in Washington.

    http://www.pnl.gov/main/publications/external/technical_reports/PNNL-14616.pdf

    Dunno if it is part of the other weather station networks.

  50. Posted Oct 1, 2007 at 12:20 PM | Permalink

    Steve Mc:
    Rather than use triangles for OpenTemp, would you suggest using hexagons? I considered that route but rejected it because:

    a. To completely cover a sphere you need to use some pentagons
    b. Smaller hexagons are not contained in parent large hexagons

    Triangles also have the advantage of simple geometry for determining which stations lie inside each cell.

    (I realize that you were suggesting hexagons for ideal station placement, and I agree that hexagons are the best option for that).

    Rick Ballard:
    “I specialize in answers to hypotheticals.” — LOL

  51. Michael Jankowski
    Posted Oct 1, 2007 at 12:28 PM | Permalink

    “If you had one station in the us where would you put it?”

    Where the sun don’t shine.

  52. steven mosher
    Posted Oct 1, 2007 at 12:29 PM | Permalink

    RE 47. What SteveMc said.

    The difference is this. When you are planning sampling to set up sites in a regular way.

    Our problem is that we have an IRREGULAR network.

    The question is how do you “regularize it” You dont regularize it by picking
    two stations 10 km apart. You dont regularize it by having ten thermometers in
    one grid and 1 in another.

  53. Chris Harrison
    Posted Oct 1, 2007 at 12:38 PM | Permalink

    John V,
    I think your system of triangles is better than hexagons or squares because you can completely cover a sphere with identically sized objects. I had to think about this for a bit but all of the triangles are proper equilateral triangles whose edges follow great circles.

    I also like the averaging scheme for coping with empty triangles as well. This is much better than some GISS style interpolation scheme to conjure up artificial values for empty cells.

  54. Kristen Byrnes
    Posted Oct 1, 2007 at 12:41 PM | Permalink

    John V. #28 and 35,

    I retraced my steps and was able to find the problem.

    42572274002 (NASA GISS #) is the number you have on the first line of your spreadsheet. I put that into the NASA GISS identifier and it gave me Univ. of Arizona. I then went to the original observers report for Univ. of Arizona #1. But that is not the correct station name. The correct name for the station on your CRN = 5 list is Tucson WFO, not Tucson Univ. of Arizona #1. So what have we learned here? NASA GISS can’t even get the station names right. (I did compare the correct station to the monthly value on your spreadsheet and they are definitely within the possibility of rounding inaccuracy)

    Second issue is rounding inaccuracies vs. changed values due to QC corrections (or probably a combination of both). The only way we are going to be able to find this out for sure is to compare GHCN daily values to the reported values on the original station reports. I do not have the software to display the station dailies from GHCN, is there anyone who can put those on an excel spreadsheet and post them somewhere?

    Third issue is why are GHCN months missing when there are observers reports filed with no missing days. The USHCN QC says that a station month will be thrown out if more than 9 days are missing and/or flagged as suspicious or erroneous.
    So let’s look at Tucson WFO aka GISS # 42572274002 and NCDC # 028817 (this is confirmed at MMS as being the one in the parking lot of the physics building)
    Missing Months are:
    11-05
    06-05
    03-05
    None of these months are missing any days in the original observers reports. Therefore there are two possibilities:
    1) Someone at NCDC goofed up and forgot to add the station for those months, or;
    2) 9 or more of the daily values were flagged as suspicious or erroneous and the months were thrown out.

    Go back farther to 1997 where there are no values for the same station in GHCN:
    12-97
    11-97
    04-97
    In these months the observer’s reports are not even listed at NCDC so someone somewhere might have forgotten to turn them in or lost them.

    Now let’s look at Chico Univ. Farm, CA (GISS # 42572231006 also NCDC # 41715) where the GHCN values are blank for 03-06 and 11-99. These months are missing at least 9 daily values so they obviously have been excluded in accordance with the QC rules.

    Last issue and not really just for John V (something I noticed while looking at the Univ. of AZ sites) but also for Anthony and Steve. A while back there was a big issue about the University of Arizona Station (Tucson WFO) that is in the parking lot of the campus there, mainly between Anthony and Steve Mc on one side and Nate Johnson (who writes the ATMOZ blog and is also a student there) and a couple of people at the university (I think professors) on the other side.
    The two stations at Univ. of AZ are about 2 miles apart. Let’s start with March 2006. The MAXIMUM temperatures between the two stations are usually different with the one in the parking lot usually being cooler BY AS MUCH AS 11 DEGREES! But the MINIMUM temperatures for the same month are usually warmer at the parking lot station.

    Now lets go to July 2006 where the MAXIMUM temperatures also are different almost every single day except this time it’s the other way around. In July the MAXIMUM temperatures at the station in the parking lot are warmer, usually by a few degrees but on one day by 8 degrees. MINIMUM temperatures are also usually warmer in the parking lot, on one day by 12 degrees. However, there were 5 days that month where the MINIMUM temperatures were cooler in the parking lot by 7, 8, 4, 7 and 1 degree.

    The idea for John V is that there are many possible outliers here.

  55. steven mosher
    Posted Oct 1, 2007 at 12:47 PM | Permalink

    JohnV.. I’m thinking that warming bias at class 5s might look like shot noise..
    Does this change your modelling of the expectation? I sent you and ANthony and steveMC
    an excell sheet of all my OpenTemp analysis. Lots of data.

    Your suggestion that I look at CRN12345-CRN1234 Was Brilliant. One reason I looked at
    (ALL_SITES-ALL_SITES_NO_5s), was I thought the 5s would be extra noisy, and comparing
    58 noisy time series ( especially if the Warming bias is shot noise) to 53 CRN12
    ( not so noisy) was going to be tough. ( Side note, Kenneth appears to have overcome
    this with his method)

    Anyways, CRN12345 – CRN1234 is always positive, around .14C as you can see from the excell.
    No zero crossing since 1880. ( no crossings on a smoothed anomaly )

    For interest readers. If you calculate US temps using OpenTemp and use the 400 stations
    that ANthony has surveyed ( 70 times the number of stations used for brazil) you will
    get a temperature trend. The anomaly is about .5 to .6C ( 2005) over the 1950-81 mean.
    If you DROP the 58 class five stations, this Anomaly drops by .14C

    Simply, 25% of the warming is due to 15% of the stations.

    Caveat: Uniformity of geographical distribution.

  56. steven mosher
    Posted Oct 1, 2007 at 1:15 PM | Permalink

    RE 54. Kristen I think One cool thing you could do for the whole community
    is to draw the Data Flow Diagram for Climate data.

    A DATA FLOW DIAGRAM ( you can google this) will show you two things.

    1. Data sources.
    2. Data processes.

    So, draw the data flow diagram. What are the sources? Then, what processes get appllied to
    that data. and what is the OUTPUT of that process.

    I think this task will help you going forward in your education since you have a nose
    for details. Every team needs a person who digs this stuff out and documents it.

    Here is a link on DFDs: http://en.wikipedia.org/wiki/Data_flow_diagram

    A free resource is here: http://en.wikipedia.org/wiki/Dia_%28software%29

    You should do this! you have a good grasp of all the sources. It’s time
    to put that knowledge into a form that OTHERS can benifit from.

    PS. Have you talked to Dr Curry about her offer? I thought it most kind.

  57. SteveSadlov
    Posted Oct 1, 2007 at 1:51 PM | Permalink

    RE: #44 – Out at the furthest western point of the Olympic Peninsula.

  58. Anthony Watts
    Posted Oct 1, 2007 at 2:22 PM | Permalink

    RE56, Steve spot on, I’ve been thinking we need the same thing. A flowchart does not exist to follow the data provenance and methodology.

    This would be a worthwhile project, and I volunteer to help.

  59. Anthony Watts
    Posted Oct 1, 2007 at 2:26 PM | Permalink

    RE55, Steve Thanks so much for doing this. What’s needed now is some independent verification of your findings. I encourage JohnV and SteveMc to have a look. I will as well and we’ll compare notes.

  60. steven mosher
    Posted Oct 1, 2007 at 2:35 PM | Permalink

    RE 54. Kristen you are being a bit unfair to JohnV

    The sheet that we are are working from is the Surfacestations Excell sheet.

    On that sheet we have the following for the first class 5.

    1. USHCN ID: 28815.
    2: Lat: 32.23 Lon: -10.95.
    3. Name U Az.
    4. GHCN: 72274002.

    To Access the GHCNv2 File ( which GISS also uses ) We appended the US country code ‘425’ to the
    GHCN number.

    This is not a GISS ISSUE or a JOHNV issue.

  61. Anthony Watts
    Posted Oct 1, 2007 at 2:47 PM | Permalink

    RE54, 60 Kristen, the Tucson station has a plaque on the fence, see it here:

    http://gallery.surfacestations.org/main.php?g2_itemId=13029

    28815 is in fact the correct station number for U of A in the parking lot there, not 28817, but it is an easy mistake to make, I had to triple check this one when I first setup the Tucson entry because the way it is presented in MMS database makes it easy to conclude that 28817 is the station number that is the USHCN station.

  62. Posted Oct 1, 2007 at 3:02 PM | Permalink

    #55, #59:
    I haven’t checked steven mosher’s results but they seem consistent with mine (see my post #376 in the first part of this thread). I found a difference of 0.35C between CRN5 and CRN12R and he found a difference of 0.14C between CRN12345 and CRN1234.

  63. Kenneth Fritsch
    Posted Oct 1, 2007 at 3:06 PM | Permalink

    I applied my state by state geographic weighting to adjust the trends for the CRN1, CRN2, CRN3, CRN4, CRN5 and CRN12 categories using the Urban (most adjusted) USCHN data set from 1960-2005 in degrees F.

    The adjusted and unadjusted results are listed below for comparison.

    While the sample sizes are small, the adjusted trends show CRN2 having significantly smaller trend then the other groups and CRN1 having a surprisingly larger trend than might be expected from the audit evidence.
    The unadjusted trends:

    CRN1 = 0.0428; CRN2 = 0.0353; CRN3 = 0.0422; CRN4 = 0.0431; CRN5 = 0.0477; CRN12 = 0.0377.

    The geographically adjusted trends:

    CRN1 = 0.0465; CRN2 = 0.0340; CRN3 = 0.0442; CRN4 = 0.0441; CRN5 = 0.0480; CRN12 = 0.0378.

    The CRN1 and CRN2 difference is bothersome to me since we do not have an a prior or an after the fact explanation for it at this point other than the small sample sizes. It adds to the unexplained difference I see for the same time period for CRN12 and CRN5 when I make the comparison with the John V derived data set (Areal adjusted USCHN only) and when I use the fully adjusted USCHN Urban data set.

  64. JerryB
    Posted Oct 1, 2007 at 3:13 PM | Permalink

    Regarding stations 028815 and 028817 and their names: the name of 028815 changed
    in 1998. That change was not reflected in the USHCN station inventory, or in
    the GHCN V2 station inventory. GISS presumably got the names from the GHCN
    station inventory.

    028815 TUCSON UNIV OF ARIZONA 19480701 19550701 32 14 00 -110 57 00 2421
    028815 TUCSON UNIV OF ARIZONA 19550701 19610501 32 14 00 -110 57 00 2411
    028815 TUCSON UNIV OF ARIZONA 19610501 19680901 32 15 00 -110 57 00 2431
    028815 TUCSON UNIV OF ARIZONA 19680901 19860223 32 15 00 -110 57 00 2444
    028815 TUCSON UNIV OF ARIZONA 19860223 19921007 32 15 00 -110 57 00 2444
    028815 TUCSON UNIV OF ARIZONA 19921007 19970528 32 14 00 -110 57 00 2435
    028815 TUCSON UNIV OF ARIZONA 19970528 19980101 32 13 47 -110 57 17 2435
    028815 TUCSON NWSO __________ 19980101 20010802 32 13 47 -110 57 14 2478
    028815 TUCSON NWSO __________ 20010802 20021218 32 13 45 -110 57 13 2435
    028815 TUCSON WFO ___________ 20021218 99991231 32 13 45 -110 57 13 2435
    028817 TUCSON U OF ARIZ #1 __ 19820501 19860709 32 16 00 -111 00 00 2300
    028817 TUCSON U OF ARIZ #1 __ 19860709 19960625 32 16 00 -111 00 00 2300
    028817 TUCSON U OF ARIZ #1 __ 19960625 20010802 32 15 29 -111 00 19 2315
    028817 TUSCON U OF A #1 _____ 20010802 99991231 32 15 29 -111 00 19 2315

  65. Pat Keating
    Posted Oct 1, 2007 at 3:23 PM | Permalink

    Simply, 25% of the warming is due to 15% of the stations.

    Not only that, but is due to the 15% lowest-quality stations.

  66. Kristen Byrnes
    Posted Oct 1, 2007 at 3:35 PM | Permalink

    Steven Mosher # 56,

    A data flow diagram for this junk? LOL NO! It would end up looking like a bowl of spaghetti LOL! Do you have any idea how messed up this stuff is? It’s as bad as the stations. I’ll ask someone to try to help but no promises.

    As for Dr. Curry (yes I thought that was very nice of her) I told her that I’ll start to consider colleges at the end of next year. But it is a possibility because my step-grandmother has her masters from there and if I get to live in her very nice (and big) house with her 5 FAT (I’m talking enormous) cats that is a possible maybe. Then there is N. Carolina, UC Santa Barbara, Bates, Bowdoin, MIT, and about 50 others that asked me to talk to them too (I’m still holding out for something in Greece LOL).

    Steven Mosher # 60

    That was nothing bad about John V. I used his data and the only way I could identify his data was the number he has on the top of each column. I c-&-p’d it from his spreadsheet into the GISS locator just to find the station name. Then I went to the NCDC page where the original reports are. GISS has the wrong name on that station.

    Anthony Watts # 61,

    Correct! The correct name for 28815 (the one in the parking lot at the college) is Tucson WFO (GISS has it as University of Arizona).
    The correct name for 28817 is University of Arizona #1.
    I’m pretty sure that is correct because I have the original station reports (for the months I talked about in 54 above) printed out and right in front of me.

  67. SteveSadlov
    Posted Oct 1, 2007 at 3:56 PM | Permalink

    (I would concur that a data flow diagram would be a good tool. I’d add to that state machines depicting the Hansen Code and other competitors to it, if time allowed. I digress.)

    Anyone familiar with something by Soon et al, 2004, which looked at Arctic Wide anomalies?

    I seem to recall one of the charts showing a very similar profile to CRN1-2R. Notably, I recall a major, almost synthetic looking, step up somewhere between 1915 and 1925.

  68. Kristen Byrnes
    Posted Oct 1, 2007 at 4:06 PM | Permalink

    Correction to 54, 3rd paragraph, it should be 28815 not 28817

  69. SteveSadlov
    Posted Oct 1, 2007 at 4:07 PM | Permalink

    RE: 41 – My usual shorthand has once again resulted in a lack of clarity. Let me try it this way. 5 year mean of both GISS and CRN1-2R take a step up, in unison, between 1917 0r 1918, and, 1920. In fact, it is during that step up that GISS and CRN1-2R demonstrate their best agreement, better than any other time frame. You cannot attribute such a result to a small sample size.

  70. Posted Oct 1, 2007 at 4:13 PM | Permalink

    RE64, 67 University of Arizona #1 USHCN #28817 is across town, at the Agricultural experiment farm.

    From the MMS “location” tab:
    [2001-08-02] 9999-12-31 32.258060 (32°15’29″N) -111.005280 (111°00’19″W) GROUND: 2315 FEET — PIMA 07 – SOUTHEAST MOUNTAIN (+7)
    Location Description: U OF A EXP FARM WITHIN AND 5 MILES NW OF PO AT TUCSON AZ

    #28815 is the one on campus in the parking lot. The reason it is called Tucson WFO is the fact that its just a block away. The NWS Tucson office is located on campus at the University of Arizona. They jointly operate the 28815 station.

    There has been no renaming of 28815 to 28817 in 1998, nor relocation of either station.

  71. Posted Oct 1, 2007 at 4:19 PM | Permalink

    #71 SteveSadlov:
    I understand you now. I was looking at the single-year means, which have larger swings in CRN12R than in GISTEMP.

  72. JerryB
    Posted Oct 1, 2007 at 4:22 PM | Permalink

    The renaming in 1998 was of 028815 from “Tucson Univ of Arizona” to “Tucson NWSO”.
    It was later renamed to “Tucson WFO”.

  73. Kristen Byrnes
    Posted Oct 1, 2007 at 4:27 PM | Permalink

    LOL Anthony, see my correction in 70, I saw what you were talking about.

  74. steven mosher
    Posted Oct 1, 2007 at 4:30 PM | Permalink

    re 67. Kristen

    I will always remain a booster and promoter of the begineers. Primarily because smarter people than I
    blessed me with the opportunity to chat with them, take their consul, and form my own sense of things.

    Let me recap. I asked you to look at ANTHONYS spread sheet and look at the Class1 instruments.
    11 of 17 were ASOS. YOU ASKED ME WHAT I MEANT? Then you asked me to do the work for you.

    You took MORE TIME in framing a response to me, than the orginal task would take.

    A data flow diagram OF THIS JUNK would be hugely valuable. If you dont do it, I’ll merely
    question your dedication. I have fixed many junky systems ( YOU USE THEM TODAY!) by
    merely focusing on the DFD.

    Throwing stones is easy kid. I’m a david from years long gone. Slain many goliaths.
    That is why I like your spirit!. Still I need to paint a picture of what is beyond
    this. ( Sorry, its the dad/prof in me)

    Pick up a stone: What comes to mind? build a house? or slay a giant?

    Just a question.

    DO the DFD.

  75. Posted Oct 1, 2007 at 4:31 PM | Permalink

    RE72, 73 Well like I said, with Tucson it’s confusing. ;-) JerryB, apologies. The name changed, the ID did not. But what you had in your post contained the names and both the 28815 and 28817 ID, and implied a morph of some sort.

    Thank you all for playing “Name That Station”, now Don Pardo, please tell us what lovely parting gifts we have for our players…

  76. Sam Urbinto
    Posted Oct 1, 2007 at 4:36 PM | Permalink

    I want a set of steak knives.

  77. Kenneth Fritsch
    Posted Oct 1, 2007 at 4:49 PM | Permalink

    Re: #74

    You took MORE TIME in framing a response to me, than the orginal task would take.

    Obviously then, Steven, you have taken more time with your reprimand that it would have taken you to do the chore. Please leave lectures such as this one to old guys like me, since those being lectured have the option of noting that giving lectures is what old guys do.

  78. JerryB
    Posted Oct 1, 2007 at 4:58 PM | Permalink

    USHCN station data somehow get from the NWS to USHCN people at NCDC.
    They do their thing and forward “raw”, and adjusted data to GHCN. They
    also publish their results in the USHCN directory on the NCDC FTP server.
    Sometimes they do both, sometimes one or the other.

    Some of what USHCN data currently reside on the NCDC FTP server also reside
    on the CDIAC FTP server (minus the 2006 data), and a portion of a
    previous version of USHCN data which had not been posted on the NCDC FTP
    server has been forwarded to GHCN.

    GISS recently began using the CDIAC set of USHCN data through 2005, the
    GHCN “raw” USHCN data for 2006, and the GISS “patch” modifies the
    pre-2006 data making it incompatible with USHCN adjustments.

    I trust that “flow” is sufficiently obscure to fit a “climate science”
    paradigm.

  79. Sam Urbinto
    Posted Oct 1, 2007 at 5:01 PM | Permalink

    If teaching is the goal of a task, doing it yourself is rather counterproductive. (And of course, I’d think it’s entirely possible it’s actually already been done by somebody.)

  80. Mike
    Posted Oct 1, 2007 at 5:10 PM | Permalink

    #74 Steven Mosher
    She slayed Al Gore, isn’t that giant enough?

  81. Kristen Byrnes
    Posted Oct 1, 2007 at 5:20 PM | Permalink

    LOL Steven

    “You took MORE TIME in framing a response to me, than the orginal task would take.”

    Was that the night Sick Puppies were in town?

  82. Steve McIntyre
    Posted Oct 1, 2007 at 5:23 PM | Permalink

    C’mon – please stop nattering at one another. I’m going to do some deleting.

  83. Kristen Byrnes
    Posted Oct 1, 2007 at 5:28 PM | Permalink

    How about this Steven, Breaking Benjamin is coming up, say, two tickets?

  84. steven mosher
    Posted Oct 1, 2007 at 5:47 PM | Permalink

    RE 77.

    Kenneth If you persist in screwing up my Tom Sawyer Whitewash the picket fence moments
    I will cut your achilles tendon and you will never golf again

  85. steven mosher
    Posted Oct 1, 2007 at 5:55 PM | Permalink

    RE 82.. SteveMC.. I think its good humour amongst friends. Your call of course

  86. Kenneth Fritsch
    Posted Oct 1, 2007 at 6:35 PM | Permalink

    Steven, your motives were obvious and well intended. My big mouth gets me trouble with the parents of my grandkids. That achilles tendon thing has been become a Greek myth: witness my tearing one apart as middle-ager playing basketball and it mending in a matter of months. Now they can mend them even faster. I think breaking fingers would be effective in the age of the internet.

  87. Kenneth Fritsch
    Posted Oct 1, 2007 at 6:48 PM | Permalink

    Re: #78

    JerryB, I took your post to mean that flow charting in this case would not be an exact science. In that case, I ask your permission to copy your post to my computer room wall. Information such as that you provided does help me understand the flow better.

  88. steven mosher
    Posted Oct 1, 2007 at 8:02 PM | Permalink

    RE 87.

    Not to worry. Sometimes I’m Thinking… Hey Kristin, stop bugging me with that important stuff
    Me, Kenneth and JohnV are discussin METHODOLOGY. ( she won’t read this right?)
    Anyway, I sent a bunch of out put from OpenTemp to Anthony, clayton, JohnV and St.mac.

    But I am still puzzling on your method. Not questioning it, just trying to place it and
    understand it. I was talking to JohnV the other day and mentioned how refreshing I thought
    your approach was.. started to rip the cobwebs away beneath my grey hair. So, If its not
    too much of an imposition can you walk me through it kindergarten style?

    I just figured out what the heck I’m doing and why, so I’ll go next. JohnV is off tesselating
    globes.

  89. steven mosher
    Posted Oct 1, 2007 at 8:08 PM | Permalink

    78. MilStd 2167A just had a seizure.

    I know you guys are expecting me to do this dang DFD. NO WAY. NO HOW. Homie don’t
    play that no mo. Where’s sadlov?, he can spell DFD.

  90. SteveSadlov
    Posted Oct 2, 2007 at 9:23 AM | Permalink

    RE: #89 – Earlier someone mentioned would be nice to have an intern (or army of them). I am thinking, similarly, what a great little project, possibly leading to a publication, by some young, up and coming masters or PhD candidate. There have got to be a few such folks who read and post here.

  91. Posted Oct 2, 2007 at 10:53 AM | Permalink

    I finally wrote the USHCN parser this morning, and spent a little time running it against the USHCN temperature data…

    =====
    USHCN vs GHCN
    My first step was to compare the USHCN Raw (Areal) temperatures to the GHCN Unadjusted temperatures. The plots below show the differences in the 5 year trend:

    The very small differences are due to rounding when converting from Fahrenheit to Celsius. The data in the GHCN files is rounded to 0.1C when converting, but my parser converts without rounding.

    =====
    USHCN TOBS vs RAW
    My next step was to compare the USHCN TOBS to Raw (Areal). The plots below show the differences in the 5 year trend (shifted to 1996-2005 reference period):

    No significant surprises there.

    =====
    USHCN TOBS vs GISTEMP
    The last step was to re-do my previous analyses comparing CRN12R, CRN5, and GISTEMP.

    The first plot shows the 5yr trend for each series, shifted to the 1951-1980 average (per GISTEMP):

    The second plot is the same as above, but shifted to the 1996-2006 average to make trends easier to identify:

    The third plot is the difference between GISTEMP and both CRN12R and CRN5. The sign is reversed from my previous analyses (GISTEMP minus CRN instead of CRN minus GISTEMP):

    The fourth plot shows trailing 20yr trends for each of the series:

    The fifth plot shows trends over key periods of the last 90 years (based on the general shape of the temperature history):

    I think these results are pretty important. However, I’ve spent way too much time on this today, so I will leave any conclusions to others. I will post my own conclusions after I’ve had some time to think about the results.

    I can’t forget the disclaimers though:
    – there are geographical biases in the stations
    – these results apply only to the USA lower 48
    – etc

    =====
    My data files, source code, output files, and summary spreadsheets are available here:
    http://www.opentemp.org/_results/_20071002_CRN12R_CRN5_TOBS.zip

  92. steven mosher
    Posted Oct 2, 2007 at 11:25 AM | Permalink

    Thanks JohnV I’ll download in a couple days.

    1. When comparing to GISS do you account for the different definition that GISSuse for the year?
    2. GISS Moving average is a 5 year centered plot. Do you have the actual figure for the 1950-1981
    Average? And I thought you did a Trailing MA?

  93. SteveSadlov
    Posted Oct 2, 2007 at 11:25 AM | Permalink

    TOBS, late 1910s. What up?

  94. Posted Oct 2, 2007 at 12:31 PM | Permalink

    #92 steven mosher:
    1. I believe that GISS uses calendar years for the averages that I’m using. I seem to recall checking that last month but I can’t find the spreadsheet I used:
    http://data.giss.nasa.gov/gistemp/graphs/
    http://data.giss.nasa.gov/gistemp/graphs/Fig.D.txt

    2. I calculate the 5yr plots using a 5yr centred average for both my results and GISTEMP (I wrote “5yr trend” in my post by mistake). The 20yr trends are trailing trends.

  95. Posted Oct 2, 2007 at 12:43 PM | Permalink

    #93 SteveSadlov:
    The same sort of TOBS jump occurs in the NOAA plot of net adjustments for the entire USHCN network. There’s a similar bump around 1950 that appears in the results above and the NOAA net adjustments.

    That’s not an explanation, but at least my results (using station NOAA adjustments) are internally consistent with the net NOAA adjustment.

  96. Kenneth Fritsch
    Posted Oct 2, 2007 at 12:51 PM | Permalink

    Using the USCHN Urban data set, I will attempt to mirror the calculations being made using the derived John V data set. Using the 1960-2005 time period, I compared the trends in degrees F between all the stations contained in the CRN1 through CRN5 categories with the USCHN stations remaining to be categorized. The CRN1 through CRN5 stations in this calculation numbered 395, while those in the remainder group numbered 826. I list the trends below in unadjusted form and again after using my state by state geographical adjustment.

    Unadjusted Trends:
    CRN1 through CRN5 = 0.0431
    Remainder = 0.0378

    Geographically Adjusted Trends:
    CRN1 through CRN5 = 0.0440
    Remainder = 0.0466

    The adjusted comparison indicates that what has been analyzed is not significantly different from that which remains to be analyzed at this point, at least, in terms of temperature trends for the period 1960-2005.

  97. steven mosher
    Posted Oct 2, 2007 at 1:11 PM | Permalink

    JohnV, One other thing to consider. GISS is not a gold standard. Tin comes to mind.

    My goal is to Answer the question about the bias of Class5 sites. Simple: Do they warm more
    than everything else as predicted? The appropriate test for this is: ALL – CRN1234.

    TWO FORKS ON TESTING:

    A. We could do this with GISSTEMP. if we could actually get it to COMPILE. OR Hansen could take
    the station lists and contribute to our understanding in 15 minutes. He could run GISS with the class
    5s and without the class 5s. How easy would that be. jeeesh. I’m dumb founded at his obdurocity. ( hehe)

    B. We can use OpenTemp and never talk about GISSTEMP again. I’m of this open mind. What sealed it for me
    was this: Eli and Gavin hopped on the first results like two dumb bunnies in spring.
    They finished, in less than a minute, concluding that OpenTemp was valid. A happy ending they enjoyed
    in onanistic fashion.

    I thunk on that awhile…. Tick tock.

    200 milliseconds latter I concluded the following: OpenTemp’s first release and first data run
    ( crn12) resulted in Gavin-climate godling-Schmidt concluding that the model was VALID.
    He saw a post on rabbett.
    AND HE DREW CONCLUSIONS. WTF?
    Blew me away. JohnV hasn’t made these claims of validity
    ( why, because he has actually
    BUILT STUFF that needs to work or people get hurt )
    Gavin made this claim because it met his expectation.
    Observation bias.
    No sense of humour.
    Arrogant People who always expect to have their bias
    confirmed., never laugh. Laughter involves upsetting expectations, and always requires a willingness to
    admit that your expections might be wrong.

    mosh pit phenomenolgy of guffaws and nose enemas.

    Where was I?

    Ahh yes, when you matched GISS Folks said OpenTemp is valid. It matches the gold standard.

    Hmmm. I wondered for a milli second about how OpenTemp would have been recieved if it didnt
    match? Personally, since I could actually review your code, understand it, compile it, run it,
    I would trust YOU over NASA!! Gavin indicates that if OpenTemp didnt match GISS, that Opentemp would
    be invalid. this strikes me as unscientific. This strikes me as GISS can do no wrong. Which we
    know, is wrong.

    ANYWAY. I decided that I dont care what GISSTEMP says or does. It cant be compiled. Cant be checked.
    cant be tested. the term Goose s … comes to mind. Don’t know why. perhaps trader Joes is having a
    special this week on the foods of goddards.

    So, my sense of things is just to focus on using OpenTemp to address questions that people have.
    Forget the GISS comparisons for now, until they publsih their new paper.

  98. SteveSadlov
    Posted Oct 2, 2007 at 1:15 PM | Permalink

    RE: #95 – It is quite interesting that both the 1910s and 1950s adjustments have the effect of “cooling” the preceding period.

  99. Kristen Byrnes
    Posted Oct 2, 2007 at 2:23 PM | Permalink

    John V,
    I posted historical pictures of Portland Maine on surfacestations.org and there looks like some CRN 5 data that is early in the century that you can use.

  100. Posted Oct 2, 2007 at 2:39 PM | Permalink

    #97 steven mosher:
    I concur that GISTEMP is not a gold standard. It is however the current standard and should be given some credit. The algorithms in GISTEMP manage to take a whole pile of crappy stations and spit out a result that closely matches the best stations (CRN12R) — for the USA lower 48 of course.

    The CRN12R results from OpenTemp have two outstanding issues: lack of stations and suboptimal geographic weighting. Once these are resolved I will be comfortable calling CRN12R the standard for the USA lower 48.

    I agree that it’s time to leave GISTEMP behind, and that we should be looking at quantifying micro-site issues.

    My approach for determining the effect of station biases is a little different than what’s been done so far:

    1. We need to distinguish UHI from micro-site. The best way to do this is to run Rural, Airport, Small, and Urban sets for each of the categories (CRN1 and CRN2 must be grouped because of the very small numbers of CRN1 stations):
    – CRN12R, CRN12A, CRN12S, CRN12U
    – CRN3R, CRN3A, CRN3S, CRN3U
    – CRN4R, CRN4A, CRN4S, CRN4U
    – CRN5R, CRN5A, CRN5S, CRN5U
    Compare the trends in each of the categories to the rural trend for the same category to get the UHI effect.

    2. The results above could potentially have some major problems with geographic distribution. Do some additional runs with more stations to validate results:
    – CRN123R, CRN123A, CRN123S, CRN123U (should be midway between CRN12 and CRN3 results)
    – CRN45R, CRN45A, CRN45S, CRN45U (should be between CRN4 and CRN5 but closer to CRN4)

    3. Compare each run to its counterpart CRN12 run (eg. CRN5R to CRN12R) to get the effect of station bias.

    4. Try to determine the error bars due to geographic distribution. I suggest running many random sets of CRN123R with ~20 stations in each set (CRN12R is too small for this). A little Monte Carlo magic and we’ve got error bars.

    For any set of results we need appropriate metrics for comparison. Kenneth has been using 1960 to 2005. SteveSadlov likes 1930 to 2005. I prefer splitting the trend into key periods (1915-1935, 1935-1975, 1975-2005).

    Thoughts?

  101. Anthony Watts
    Posted Oct 2, 2007 at 2:43 PM | Permalink

    John V Thank you for your continued diligence and hard work. There will be new stations coming online soon, and hopefully we’ll be able to add to the database of station ratings for this program.

    Mosh I too was surprised that Gavin embraced the 1st run from this and posted it on RC. Any engineer knows better that to trust the first results from any invention, as first assumptions in design and methodology don’t always pan out for a correct result. Of course, Gavin isn’t an engineer.

  102. Posted Oct 2, 2007 at 3:09 PM | Permalink

    #99 Kristen Byrnes:
    I’m not sure what I should do with CRN5 data from early in the century.
    What did you have in mind?

  103. Posted Oct 2, 2007 at 3:17 PM | Permalink

    For UHI-check I suggest to use US Census population data.

  104. Anthony Watts
    Posted Oct 2, 2007 at 3:25 PM | Permalink

    RE 99,102, Kristen/JohnV what I hope to do is to get NCDC to release B44 forms, which contain site sketches and descriptions. I have all of California’s and they are extremely useful in determining past micro-site issues. If we can get those, perhaps redacted to remove observer names/addresses then we can look at creating a timeline of CRN1-5 ratings

  105. Kenneth Fritsch
    Posted Oct 2, 2007 at 4:57 PM | Permalink

    I think we are combining a couple of projects that really should not be combined at this point. John V is looking at the validity of adjustments used by GISS and using their basic coded approach to reconstruct the raw measurements with his own choice of adjustment algorithms. Evidently the goal there is to determine how well his adjustments compare with GISS. In another project we have the Watts and team CRN categories that we are attempting to analyze (probably prematurely and without a good deal of forethought at this point) to determine whether the categories show statistically significant differences between any of the groups or combinations of the groups.

    My question is why are we attempting to use the untested John V data set to test the Watts and team categories. I think there is much to learn about temperature data set adjustments from the step by step process that John V is showing us. I think we would be better served by looking at the CRN categories at this point in an independent mode.

    There are, of course, not only the GISS data set but one from USCHN and CRU that are also fully adjusted and applicable to measuring the CRN group differences. Why make the process of analysis so complicated when it can be performed in parts.

    Also we seem to stick to be looking at the CRN categories under the street light of long term series (my 1960-2005 is used simply as a constant frame of reference and not because I judge we should be making comparisons that far back in time) and not attempting to find better ways to analyze these categories given their snapshot time reference.

  106. Kristen Byrnes
    Posted Oct 2, 2007 at 4:58 PM | Permalink

    John V # 102,

    The station in Portland was on roofs or in windows from Dec. 1874 to Dec. 1940. That’s 66 years of data that Anthony usually makes CRN-5. If you program your opentemp to adjust the geographic distribution every month in order to account for missing data (there’s a lot of missing data in 2006 for example) then you will not have geographic distribution 100 years ago based on geographic distribution today. The same with CRN rating. Your CRN rating is flawed going back in time because the micro site biases changed. Portland is CRN 2 today but was CRN 5 back in 1940. The same with many other stations.

    Anthony 104,

    I can get the sketches from NCDC (65 cents ea) for the temp stations already surveyed and accepted for opentemp but if you can get them for free, that would be better.

  107. Clayton B.
    Posted Oct 2, 2007 at 5:09 PM | Permalink

    JohnV,

    Have you thought about how to get opentemp program to select stations based on certain things in a database (airport, elevation, CRN rating, etc.) in the code?

    Anthony,

    Can the surfacestations.org table be modified to quickly include additional columns such as groundcover, waterplant?, shading, etc. Or would this require a re-look at all of the observation reports?

  108. Posted Oct 2, 2007 at 5:13 PM | Permalink

    #105 Kenneth Fritsch:

    Evidently the goal there is to determine how well his adjustments compare with GISS.

    That seems to be a common misconception. I have been comparing to GISTEMP for a couple of reasons:
    1. It’s the only “accepted” temperature trend for the USA lower 48 that I could find;
    2. There has been interest here on problems with the GISTEMP algorithm

    As I said above, I am ready to move on from GISTEMP. We can evaluate the effect of micro-site and UHI independently using OpenTemp (or manual processing) alone.

  109. Posted Oct 2, 2007 at 5:18 PM | Permalink

    #106 Kristen Byrnes:

    If you program your opentemp to adjust the geographic distribution every month in order to account for missing data

    It is actually programmed that way. Every month is processed independently.

    I agree that the CRN ratings are snapshots, but that’s all we have right now. I hope it is safe to assume that in *most* cases the station quality gets worse over time.

    One of the items on the OpenTemp roadmap is restricting station data to a date range. When this is implemented it will be possible to split the Portland Maine record:
    – 1874 to 1940 in the CRN5 category
    – 1940+ in the CRN2 category

  110. steven mosher
    Posted Oct 2, 2007 at 5:21 PM | Permalink

    Hi JohnV.

    I’ve really enjoyed using OpenTemp. I wish had this tool before. So, Glad to see that USHCN stuff
    has been added. When I first started this stuff I never looked at GISS since GISS gets_ushcn();

    Anyhow, I proceed to your comments..

    “I concur that GISTEMP is not a gold standard.
    It is however the current standard and should be given some credit.”

    NOPE. When Hansen and Gavin run ModelE ( there GCM) which global temperature dataset do they
    use to “calibrate?” HADCRU. We went down this path with Gavin. We cricticized GISSTEMP and his
    response was “we dont use it.” So, TIN, not gold. Credit is earned in my book. OpenTemp showed
    that microsite bias is real. We already knew that. GISSTEMP could do the same thing. If somebody outside
    NASA could get the heap to compile. Now, I’ve asked for the ABSOLUTE TEMPS fom gavin on several occasions
    so that Smoothing operations ( trailing, centered, gaussian etc etc ) become more transparent. do you have this.
    The reason I ask is that Hadcru is WRT 1960-91. So anytime one complains about comparing HADCRU
    to GISS you get this rejoiner about difference reference periods… etc etc etc. EVEN WHEN they share data
    the refuse to calculate anomalies to the same reference period.

    “The algorithms in GISTEMP manage to take a whole pile of crappy stations
    and spit out a result that closely matches the best stations (CRN12R) — for the USA lower 48 of course. ”

    At first blush yes. But when we say “closely match” we must say so with authority. The issue is
    comparing 53 sites to 1221 and concluding “they are they same” On one hand you have a big sample
    on the other a small sample. One cant conclude they are the “same” with any confidence. On the other hand
    one approach is to test whether Class5 sites suck. This is NOT a AGW issue. This is a NOAA data quality
    issue. So, I compared ALL 1221 sites to the 1160 sites that have not been rated a class5. And
    what did I find? The 1160 sites were .05C cooler for the past 120 years. I think USHCN1221 is not
    carved in Stone. Did god ordain 1221 perfect sites in the US? Nope. You have a site survey. You have a
    piece of temperature analysis code that is solid. You have an argument to drop those 60 sites from the
    USHCN1221.

    “The CRN12R results from OpenTemp have two outstanding issues:
    lack of stations and suboptimal geographic weighting.
    Once these are resolved I will be comfortable calling CRN12R the standard for the USA lower 48.”

    That’s fine. But that’s not my central issue. My issue is the difference between non standard sites
    and sites meeting guidelines. That’s a scientific issue that I think is neat. Its relationship
    to AGW antagonizes a bunch of people ( this amuses me so I play with it)

    “I agree that it’s time to leave GISTEMP behind, and that we should be looking at quantifying micro-site issues.

    My approach for determining the effect of station biases is a little different than what’s been done so far:

    1. We need to distinguish UHI from micro-site. The best way to do this is to run Rural, Airport, Small, and Urban sets for each of the categories (CRN1 and CRN2 must be grouped because of the very small numbers of CRN1 stations):
    – CRN12R, CRN12A, CRN12S, CRN12U
    – CRN3R, CRN3A, CRN3S, CRN3U
    – CRN4R, CRN4A, CRN4S, CRN4U
    – CRN5R, CRN5A, CRN5S, CRN5U
    Compare the trends in each of the categories to the rural trend for the same category to get the UHI effect.”

    You can go down this path. That is the COOLEST thing about OpenTemp.

    Let me arm wave about UHI for a paragrah or 56. If you go ack through all the literature about UHI
    you will see that it is attributed to several causes. I’ll mention a few.

    1. CANYON Effects: The geometrical configuration of the physical structures and the material properties
    of those structures Combine to focus or intentsify radiation through multipath reflections.
    Think TWT, traveling wave tube. Think corner reflector. However, the Canyon effect also can
    inhibit reradiation as the “skyview” is occluded or vinetted. Note, canyon effects can also
    shade a site from insolation
    2. Wind Shelter. When the wind blows chances are you get vertical mixing and cooling of the surface.
    So, the issue in Cities is you have very high barrier to wind. The structures are ALL bluff structures
    ( have a look at UHI wind studies and basic areo) So the structure impact the flow at the ground in
    dramatic ways.

    3. Evapotranspiration. Wet ground is cooler than hot pavement. Especially at night. So, asphalt
    concrete, stone, rock etc etc, all give up heat with a time constant that differs from soil.
    ( Hey dad why do the snakes come out on the road when the sun go down)

    4. Artificial heat: building heat, car heat, human body heat, human activity.

    those are the basic causes of UHI. It’s been measured.In situ. From satillite.
    It’s a tenet of Enviromentalism. The number
    of studies documenting it are astounding. I can tell you the tempterature of the soil 6 inches from
    a parking lot in phoenix with no tree cover.

    MICROSITE BIAS is a sub species of UHI. All the causes are the same. The question is, in my mind,
    how it manefests itself at the small scale. Do we see it all the time, most of the time, some of the time, rarely?
    And based on that supposition.. what’s the best method to find it. Maybe we do not find it and upset years
    and years of climate science near the ground. Was that last clear? The standard accepted consensus
    understanding ( gieger) is that these issues matter near the ground. Finding that they don’t would be a challenge to
    one of the einsteins of climate science.

    “2. The results above could potentially have some major problems with geographic distribution. Do some additional runs with more stations to validate results:”

    I’m more an more convinced that chopping the data sets into smaller peices is a fundamental error.
    1. With every cut you increase your probability of finding a spurious correlation.
    2. The warming of microsite should have a Poisson distribution. Basically, as you pointed out the 5C
    bias will only happen rarely. The temp signal is a continous signal, but the sites quantize that
    signal into a max and min on a daily basis. As the wind blows, as the clouds come and go, as the rain
    comes and goes, the impact of MICRO_UHI shows up like shot noise. Now, if you compare SITE A to SITE B
    ( see OKE studies and the CRN studies) it is possible to detect these excursions as bias rather than noise.

    However, if you mash 30 days together and then 365 days together, the bias just looks like noise.
    Simply. Class5s will tend to look noisier. We have 53 Class1&2 ( urban and rural) and 58 Class 5.
    It’s a noise fest. And I’m not to sure slicing and dicing move the ball forward. But Hey, I’ll slice
    and dice the data till the cows come home. The Urban/rural distiction is problematic. The Population
    is referenced to 1980!. The night lights is reference to 1995. The extent of a night lights pixel is 2.7km
    Anecdotally Hansen rates Orland Ca as Urban. It’s Not. Peterson, on the other hand rates mineral california as a
    small city, not rural. Population: 140. and these are seasonl folks. Its a ghost town.

    The point I guess Is that I’m not prepared to go down the Urban/rural slice. Now, the equipment slice
    and the ASOS slice are more interesting.

    – CRN123R,
    CRN123A,
    CRN123S,
    CRN123U (should be midway between CRN12 and CRN3 results)
    – CRN45R,
    CRN45A, CRN45S, CRN45U (should be between CRN4 and CRN5 but closer to CRN4)

    3. Compare each run to its counterpart CRN12 run (eg. CRN5R to CRN12R) to get the effect of station bias.”

    Ok. I’ll look at this stuff. HEY, when I ran the study you suggested.

    CRN12345 VERSUS CRN1234. I found a .15C difference . That is, CLASS5 are warming.

    SO. engineering hats on. Hypothesis: class 5s are warmer than class1234. True or false using OpenTemp
    as the tool?

    “4. Try to determine the error bars due to geographic distribution. I suggest running many random sets of CRN123R with ~20 stations in each set (CRN12R is too small for this). A little Monte Carlo magic and we’ve got error bars.”

    I’m gunna leave the monte carlo magic to Kenneth. However, I would lobby to lump the class3 in with the class1&2.

    On the assumption that Microsite bias is bursty. At class3 neither one of our “models” would suggest that
    the bias could be discriminated from noise. ( Purists will scream )

    “For any set of results we need appropriate metrics for comparison. Kenneth has been using 1960 to 2005. SteveSadlov likes 1930 to 2005. I prefer splitting the trend into key periods (1915-1935, 1935-1975, 1975-2005).”

    You know what? I was an proponent of this regime thing, but I’m getting nervous and jerky about it.
    I saw tamino do it ” it’s only natural to see these periods” And I looked back at my own thought process.
    And I had thought ” pick the linear period.” this decision, of course, destroys the statitics to follow.
    Since you dont have a random sample you have a sample cherry picked for its apparent linearity. There are techiques for
    identifying regimes and regime changes. I would go that route. The other route I have taken is the following:
    Here is the data from 1880 to present! I take what I think is an intersting approach and throw the red meat in the octagon.

    “Thoughts?”

    At some point this becomes a zen moment for me. So I will look at your cuts at things and ponder.
    THAT SAID, class 3 is the weird one. I have to check my data but when I ranked anomaly curves from low to high
    I think I got 3,12,4,5. Understand? Had I got 12,3,4,5.. anyway I was thinking about weird way of looking
    at the problem.. you have 4 series CRN12, CRN3, CRN4, CRN5. And the hypothesis is that CRN5>CRN4>CRN3>CRN12
    ….. and then the thought petered out.. But I thought It would odd if the order came out CRN12,CRN3,CRN4,CRN5
    Anyhoo.

    I’ll look at your slices…

    PS. GISS Anomaly? Do you have te absolute for 1950-81? How did you adjust for the differnce in MA approaches

  111. Kenneth Fritsch
    Posted Oct 2, 2007 at 5:46 PM | Permalink

    Re: #108

    John V, I have trouble following your reply in this post.

    Is my observation that you are comparing your adjustments to GISS a misconception? If so what are you doing?

    Since the USCHN data set is fully adjusted (primarily due to the work of Karl) I would assumed it also an “accepted” data set. I think part of the GISS Y2K error was due to their equivocation on using their own adjustment to the USCHN raw data or using USCHN data adjusted through Filenet (minus only a small UHI adjustement to be completed using the Hansen satellite light proxy).

    I have continued to pose the question whether the several temperature data sets agree with one another within the limits of their acknowledged uncertainties over the time period in they cover.

    Also I must ask why we have not seen your comparison to GISS using all the the station data and not confining it to CRN12 CRN12R and CRN5 or am I overlooking something here?

  112. Posted Oct 2, 2007 at 5:47 PM | Permalink

    110 “On the assumption that Microsite bias is bursty. ” yes and no…it depends on the type of bias, cooling biases related to plant/tree growth are subtle and long period.

    Heat sink type microbiases, such as concrete/brick masses near the observing tend to bias Tmin more than Tmax. Biases like nearby traffic indeed can be bursty.

    A big case was made by two presenters at Pielke’s recent conference to do away with Tmin and only look at Tmax due to Tmin having so many possible microsite biases compared to Tmax. Perhaps running opentemp on Tmax only with CRN12345 – CRN1234 would be a good test.

  113. steven mosher
    Posted Oct 2, 2007 at 6:01 PM | Permalink

    RE 101. The sad thing for Me Anthony were the other things he wrote, Basically,

    Since OpenTemp (53 sites) matches GISS both are valid. If you get different results Then
    YOU have a problem. That sealed it for me. I read H87, H99, H2001. I plowed through the code.
    The gizard of wads.

    Simply, if you wanted to write a paper on microsite bias you have no choice
    but to use OpenTemp. GISS wont compile. Hansen promised simplified code. HA! JohnV delivered the
    simplified code from scratch faster than Ruedy could simplify existing code. The whole point of tools
    is to allow other to do work.

    Like JohnV I’m just gunna focus on the microsite issue. I think there are big challenges there
    ( Kenneth has done some intersting work )

    Going forward you could do us a favor by having an excell that adds the following

    RURAL/URBAN/Small town
    Nightlights.
    BIAS: ( warming, cooling, mixed)

    That way we can salami the data some more.

  114. Sam Urbinto
    Posted Oct 2, 2007 at 6:23 PM | Permalink

    Just because it matches once, (or however many times) doesn’t mean it always will. Seems a bit pre-mature, especially if you’re only checking 53 vs 1000+

    That said, it would be nice if we could get a GISSTEMP “All minus class1-4″ and see what it says….

    Patience is in order I would suppose….

  115. Clayton B.
    Posted Oct 2, 2007 at 6:36 PM | Permalink

    Good grief steven mosher!

    We are we not just plotting the absolutes instead of looking for a certain period to reference? Also, (for JohnV) are you referencing the same value for all series or are you referencing the average of each series for that series? confused?

  116. David Smith
    Posted Oct 2, 2007 at 6:37 PM | Permalink

    It’s an error to assume that vegetation necessarily cools a site.

    Cooling effects of vegetation include transpiration and blocking of direct sunlight

    Warming effects of vegetation include blocking of outbound IR (esp. nighttime), lower albedo and less air mixing.

    Vegetation type, height and proximity all play a role in determining the relative importance of the cooling and warming effects. It’s a complicated situation.

    Concerning another microsite topic, it is conceivable that different sites respond differently to changes in cloud cover. Arizona asphalt may respond differently than Arizona soil to decadal-scale changes in cloud cover. Regional cloud cover does change on decadal scales, just as temperature and precipitation do.

  117. Anthony Watts
    Posted Oct 2, 2007 at 7:30 PM | Permalink

    Re116, David Smith I agree, and there’s cases for both +/- related to plant/tree growth…it was a short post and I didn’t go into all the possible angles, and there are many.

  118. Posted Oct 2, 2007 at 9:18 PM | Permalink

    Lots to respond to this time. It’s late but I’ll do my best…

    steven mosher:
    I do not have the absolute temperatures. An absolute temperature has little meaning. It’s the trends that matter. It is trivial to shift from a 1951-1980 reference to a 1961-1990 reference.

    But when we say “closely match” we must say so with authority. The issue is
    comparing 53 sites to 1221 and concluding “they are they same”

    Sample size is not the issue. A few weeks ago there was pretty good consensus here that the rural CRN12 stations were the best stations and that they would give the most accurate temperature trend. Now that GISTEMP is found to match the CRN12R trend there is less confidence in the CRN12R trend. Can you honestly say that you would not be promoting the differences between CRN12R and GISTEMP if they were large? (You were promoting Hans Erren’s plot showing the divergence between CRN12R and GISTEMP a little while ago).

    We do need to figure out the confidence bounds on the CRN12R trend. It’s possible that it’s just a coincidence that it matches GISTEMP. But right now, with the best available data we have, it appears that GISTEMP gives an accurate temperature trend for the USA lower 48.

    You have an argument to drop those 60 sites from the USHCN1221.

    I agree that the bad stations should be dropped. That will improve the accuracy of the temperature trend. The problem is that the ultimate goal is not the USA lower 48 trend, it’s the worldwide trend. We do not have the data to selectively remove stations outside the USA.

    I’m more an more convinced that chopping the data sets into smaller peices is a fundamental error.

    We have multiple influences on station trends. To separate them (without statistical wizardry that few would trust) we must analyze them independently. I can see grouping CRN123 against CRN45 (instead of the single cRN level groupings).

    Your preferred method is removing a small number of stations from the large sample. I see no statistical difference between removing 60 stations from a set of 400 and adding 60 stations to a set of 0. The advantage of analyzing the 60 in isolation is that you can compare them to another set of 60.

    The point I guess Is that I’m not prepared to go down the Urban/rural slice.

    You have also been arguing that UHI is very real. I don’t see how you can quantify microsite issues if they are mixed in with UHI.

    SO. engineering hats on. Hypothesis: class 5s are warmer than class1234. True or false using OpenTemp as the tool?

    Absolutely true, as I have been saying for a while. My results from earlier today show CRN5 warming ~0.4C more than CRN12R from 1900 to 2005. Some of this is due to UHI but nobody knows how much.

    Since you dont have a random sample you have a sample cherry picked for its apparent linearity. There are techiques for identifying regimes and regime changes.

    The problem is that to reach any meaningful conclusions the data needs to be summarized in some way. My 20 year trailing trends were an attempt at that. My key period trend were another attempt. Looking at the entire range of data is just another way of cherry picking.

    PS. GISS Anomaly? Do you have te absolute for 1950-81? How did you adjust for the differnce in MA approaches

    I don’t have the absolute for 1951-1980. You can move the reference period by taking the average for the period of interest and subtracting. It doesn’t matter if the trend is already referenced to a different period. What are MA approaches?

  119. Posted Oct 2, 2007 at 9:34 PM | Permalink

    Kenneth Fritsch:

    Is my observation that you are comparing your adjustments to GISS a misconception? If so what are you doing?

    A couple of things:

    First, I’m not making any “adjustments”. I’m using OpenTemp to calculate average temperatures and temperature trends using data from GHCN or USHCN.

    Second, I have been comparing the temperature trend from CRN12R (the best stations) against GISTEMP because GISTEMP has been getting a lot of attention. Your statement was that my *goal* was to get OpenTemp to match GISTEMP and that was completely wrong. My goal is to create an independent and open temperature analysis.

    Also I must ask why we have not seen your comparison to GISS using all the the station data and not confining it to CRN12 CRN12R and CRN5 or am I overlooking something here?

    My original analyses were done using all CRN1 and CRN2 stations (CRN12) and CRN5 stations. It was a first attempt at comparing the best stations (CRN12) to the worst stations (CRN5). GISTEMP was added because it is well known, often referenced, and frequently criticized.

    The comments following that analysis indicated that the rural CRN12 stations should be looked at as they would be the best-of-the-best. They have the least microsite issues and no UHI issues. Based on that feedback (which makes sense to me) I switched to CRN12R and CRN5.

    I see little reason for comparing GISTEMP to more analyses made with stations that are known to be bad. I have compared it to the best stations (CRN12R) and the worst stations (CRN5). What would a comparison to CRN12345 add? I think it would be more interesting to start comparing CRN5R to CRN12R and all the other comparisons I discussed above. Am I missing something?

  120. Posted Oct 2, 2007 at 9:37 PM | Permalink

    Clayton B:

    Also, (for JohnV) are you referencing the same value for all series or are you referencing the average of each series for that series? confused?

    I subtract the average temperature for the entire region (USA lower 48) for the reference period from each yearly average. It’s just a shift up or down the vertical axis. It doesn’t do anything to the trends or the shape of the curve.

  121. Posted Oct 2, 2007 at 9:41 PM | Permalink

    Sam Urbinto:

    Just because it matches once, (or however many times) doesn’t mean it always will. Seems a bit pre-mature, especially if you’re only checking 53 vs 1000+

    You can only play the coincidence card so many times. Nothing can truly ever be said with 100% certainty, but you have to go with the best available results until something better comes along.

    If there are only 53 (or 17) good stations available, then the results from the 53 (or 17) are the best you can get. As more stations are surveyed the results will be updated to decrease the uncertainty.

  122. Joe Bowles
    Posted Oct 2, 2007 at 10:16 PM | Permalink

    I am thoroughly enjoying the discussion on this thread. The discussion started with focus on the difference of a CRN 1,2 to a CRN 5 site. As far as I can tell, we only know the current condition of the sites, not their status in the past. So it seems to me that looking at the long term records for those sites as classified may be introducing noise into the analysis. It might be helpful to focus on a shorter term during which it would be reasonable to conclude that the sites were in their present condition. That would probably shorter the time period to sometime in the late 80’s to maybe the early 90’s. It might be efficacious to use the year immediately after the automatic systems went into service in the 90s.

    If a significant difference shows up, then, I can see taking it back further in the records to see if the same result occurs, but it seems to me that starting with the full time series is counter-productive. It should be easier to spot effects, to the extent they occur, in the most recent period when at least the measurements are being made in a consistent fashion, right, wrong or indifferent.

    I am particularly impressed with Kristen. I love the way she stands her ground. The issue of missing days she spotted seems of paramount importance to me, since the filled data is adding an artificial bias to the database. I suspect that checking the database by site for multiple days of the same value might provide an indication of the level of impact this is causing.

    I wish I shared the computer expertise to make a contribution, but I take my hat off to all of you. Until the basic data is sorted out, the science isn’t going anywhere. It is a bit hard to know where to attribute the warming if we don’t even understand the variability of our measurements, much less the variability of nature.

  123. Kristen Byrnes
    Posted Oct 3, 2007 at 6:11 AM | Permalink

    John V,
    It looks like you are using different spreadsheets for different CRN ratings. So why not just use the same station number in the CRN 5 spreadsheet then move the data for those dates there. You can even identify the station and CRN rating by adding the following to the station ID in the top line of your spreadsheet: –A for CRN 1, –B for CRN 2, -C for CRN 3, -D for CRN 4, -E for CRN 5 and –F for CRN unknown. That way you can eliminate the “snapshot problem” without going back and rewriting your program.
    It would also lend your efforts more credibility because you are not making the incorrect assumption that stations usually get worse over time (many urban stations actually got better in the 40’s because many of them moved to airports).

  124. Posted Oct 3, 2007 at 7:12 AM | Permalink

    #107 Clayton B:

    Have you thought about how to get opentemp program to select stations based on certain things in a database (airport, elevation, CRN rating, etc.) in the code?

    I have thought about a front-end for OpenTemp that could query the database to select stations, setup analysis parameters, and view results. Realistically though, I have been spending way too much time on this and won’t get to the front-end for a while. If there is enough information in the database, I can help with SQL queries for extracting stations.

  125. Posted Oct 3, 2007 at 7:19 AM | Permalink

    #123 Kristen Byrnes:
    I assume you are talking about the station.csv file (the list of all temperatures for all stations in an analysis). That file is *output* only — it is created from the USHCN or GHCN file as an intermediate step in the analysis. So, it was a good idea but I will need to add to the program to filter stations by date ranges.

    (many urban stations actually got better in the 40’s because many of them moved to airports).

    I think this is another reason to break the century into key periods. We can look for differences in the trends between periods. For example, in my results above the delta between CRN5 and CRN12 trends is largest from 1935 to 1975. Why? I don’t know. Is it statistically significant? I don’t know.

  126. Posted Oct 3, 2007 at 7:22 AM | Permalink

    #123 Joe Bowles:
    I like the idea of using recent data, but I will be accused of cherry picking (again) if I go that route.

    A major problem with using a short time frame for comparing CRN12 to CRN5 is that we are looking at trends, and trends take time. The CRN5 bias has to change for a trend to be observable. If they consistently read warm there will be no difference in the trend.

  127. Clayton B.
    Posted Oct 3, 2007 at 7:25 AM | Permalink

    124 JohnV;

    I plan on getting into your code when I get the chance and seeing how to structure the queries.

  128. steven mosher
    Posted Oct 3, 2007 at 7:30 AM | Permalink

    Wow. Lots to respond to. I won’t get to them all

    For me the big issue is quantifying the effect of Microsite. ( a form of UHI)

    When I analyzed all 1221 sites and then took out the CRN, the anomaly went down by .05C
    across the board. since 1880. So, OpenTemp had an anomaly of around .6C ending in 2005.
    removing 5% of the sites dropped this by .05C.

    When I analyzed 400 surveyed sites CRN12345 and took out the CRN, the anomaly dropped .15C
    consistently.

    On the other hand, Like JOHNV points out, the CRN12 track the anomaly preety well. So I took
    out CRN12 from all 1221. Sure enough, you get a small difference and you get zero crossing.

    Puzzling.

    I would like to do some urban/rural cuts, but I want better info than Nightlights, and better
    info than 1980 population. Orginallly USHCN was going to do the classification by decade,
    but they dropped the approach.

    I’ll do more later, but good questions all around. I don’t have answers for them all.

  129. steven mosher
    Posted Oct 3, 2007 at 7:54 AM | Permalink

    OK One more bit ( whats the best way to post charts?)

    anyway. here is what I did. I went to GISS. I got the tabular data for Anomaly in the US ( is it only
    lower 48?)

    I Ran OpenTemp on all 1221 sites. I got the average for 1950-1981.
    I ran CRN12. I substracted out the 1950-1981 Mean. Giving me an anomaly.
    I smoothed both with a 5Year moving average. Trailing.
    Then I did CRN12-GISS.

    The result: 1884 to 2005

    0.0841875
    0.0241875
    -0.0738125
    -0.1098125
    -0.0658125
    -0.0278125
    0.0381875
    0.0441875
    0.0201875
    0.0221875
    0.0081875
    -0.0598125
    0.0281875
    0.0221875
    -0.0698125
    -0.1098125
    -0.0858125
    -0.0638125
    -0.0558125
    -0.0078125
    0.1261875
    0.1241875
    0.0641875
    0.1301875
    0.1521875
    0.0881875
    0.1381875
    0.1741875
    0.1001875
    0.0621875
    0.1021875
    0.0421875
    0.0441875
    0.0981875
    0.1421875
    0.0701875
    0.0701875
    0.0361875
    0.0001875
    -0.0058125
    0.0041875
    0.0161875
    0.0541875
    0.0341875
    0.0181875
    -0.0318125
    -0.0658125
    -0.1338125
    -0.1578125
    -0.1478125
    -0.0298125
    0.0061875
    0.0381875
    0.0541875
    0.0681875
    0.0441875
    0.0761875
    0.0821875
    0.1101875
    0.1321875
    0.1061875
    0.0741875
    0.1021875
    0.1181875
    0.0781875
    0.0101875
    0.0121875
    -0.0358125
    -0.0278125
    -0.0118125
    0.1121875
    0.1081875
    0.1761875
    0.1481875
    0.1761875
    0.0941875
    0.1141875
    0.0541875
    0.1061875
    0.1301875
    0.1521875
    0.1701875
    0.1941875
    0.1781875
    0.0961875
    0.0981875
    0.0681875
    0.0521875
    0.0261875
    -0.0098125
    -0.0058125
    -0.0338125
    0.0021875
    0.0341875
    0.1061875
    0.0681875
    0.1381875
    0.1681875
    0.1361875
    0.1121875
    0.1561875
    0.0881875
    0.0721875
    0.0901875
    0.1141875
    0.1201875
    0.1361875
    0.0681875
    0.0661875
    0.0381875
    0.0481875
    0.0461875
    0.0841875
    0.0881875
    0.0981875
    0.0841875
    0.1381875
    0.1361875
    0.1041875
    0.1621875
    0.1621875
    0.1361875

  130. Kenneth Fritsch
    Posted Oct 3, 2007 at 8:10 AM | Permalink

    Steven, I post my constructed graphics using imageshack, a free online service that you transfer images to and then use that link to post here.

    http://imageshack.us/

  131. Posted Oct 3, 2007 at 8:15 AM | Permalink

    Clayton B:
    OpenTemp just needs a list of station IDs (USHCN or GHCN), latitudes, and longitudes. They should be listed one per line, with commas between each value.

    steven mosher:
    The 1951-1980 average for your time series above is not zero. There may be a problem with your normalization, or maybe I misunderstand what you did. Did you subtract the 1951-1980 mean of all stations from the CRN12 results? Why not subtract the 1951-1980 mean of CRN12 stations?

  132. Kenneth Fritsch
    Posted Oct 3, 2007 at 8:47 AM | Permalink

    Re: # 119

    First, I’m not making any “adjustments”. I’m using OpenTemp to calculate average temperatures and temperature trends using data from GHCN or USHCN.

    John V, I was confused as I had you ticketed for a much more ambitious project. It appears that you are saying that OpenTemp is a convenient program and means of doing trends and the like using GISS data sets. I would have to ask then why you have concentrated on the “raw” data set for the CRN category analyses?

    I suppose your reference to geographical averaging and adjusting may have confused me.
    Was that recent discussion confined to geographical adjustment for calculating trends and not to compare with some of the adjustments that are fundamental to GISS?

  133. Joe Bowles
    Posted Oct 3, 2007 at 9:00 AM | Permalink

    John V said:

    I like the idea of using recent data, but I will be accused of cherry picking (again) if I go that route.

    A major problem with using a short time frame for comparing CRN12 to CRN5 is that we are looking at trends, and trends take time. The CRN5 bias has to change for a trend to be observable. If they consistently read warm there will be no difference in the trend.

    I think you misconstued my point. The way I see it, we know the status of the sites at a point in time, but we don’t know their status at any earlier point in time. This adds some confounding issues to the analysis:

    1) The sites may have been higher quality at earlier points in time, and consequently the differences would be expected to be smaller
    2) The adjustment process seems to become more suspect the further back we go
    3) We don’t necessarily have the same measurement base within site going back
    4) Sites change over time regardless of their status, so the further back you go, the more changes that are showing up in the data

    I would contend that the charge of “cherry picking” doesn’t apply here. The analysis is interested in asking the question as to whether the difference in site quality affects the indications of GISS temperatures. In essence, we are making a gargantuan assumption when we extend the analysis beyond what we know. The easy way of avoiding any argument is to use the date that the classification went into effect as the start date.

    I’d also like to point out that most of the analytical methods assume the underlying data is ratio data…and temperature data is actually interval data. The difference from a measurement standpoint is that the distances between different points on the scale are not equal as they would be in measuring length. In essence the midpoint between the Tmax and Tmin has a different meaning than a straight mean. The underlying properties are different. It hit me that the form of analysis being done on the data is based on ratio scales.

    In order to find the effect of the site classification, we may have to move to a different level of data analysis, probably focusing on robust statistics.

    One of the problems that I see is that the variability of measurements may not be Gassian. For instance, a specific upward bias in measurement in electronic devices may tend to be unidirectional due to the equipment design. That may be further affected by a normal distribution of variance in quality. My point is that we may not have satisfied the underlying assumptions of the regression models and consequently, we may be getting distortion of the results.

    That isn’t to say that the form of analysis is not useful, but rather that we might want to look at some other ways of viewing the data to test the result. In the case of interval data, it might be helpful to run some “stem and leaf” plots on the sites and calculate the median, quartiles and mid-spreads of the distributions. The median would provide a more appropriate indication of the central tendency for interval data; comparison of the midspreads would suggest the relative variance of the data; and the stem and leaf plot might provide clues as to the shape of the distribution. In essence, I think the differences would become more apparent using relatively short terms of say, 15 years or so.

    I understand your point that it is easier to see the differences when a longer term is used, but that actually doesn’t justify extending the term beyond the period for which we know, or can reasonably support the site classification. My take is that we actually don’t know much about the earlier data. We don’t have confidence in its quality. One might conjecture that a CRN5 site has always been a CRN5 site…but that is strictly conjecture. Anthony has been careful to make that point in some of the earlier posts. It would seem just as logical to conjecture that sites were higher quality early on and moved down as the sites around them improved. But that is conjecture, too. We know their current status and that is all we really know. Moving outside the parameters of our verified information is no different than what is being done with the computer models in making predictions/projections/scenarios about the future when their models have not been validated.

    Unless we can show significant differences in the time period for which our classifications are valid, nothing is gained by going back to the 20s or the 1890s. Beyond that, validating the model in the short term may offer clues as to what issues may be showing up in the earlier data. Again, I see nothing wrong with using the full data sets in subsequent sets, but I think the case has to be made based on what we can validate and that such extensions may actually be obfuscating what we are looking for.

    You are doing some terrific work here. I just thought I would throw in my two cents for consideration.

    My training was in mathematical psychology and now that I am getting old (ancient according to my grandchildren), I have to go back and revisit some of the issues to make sure I am thinking about it correctly.

    Joe

  134. Posted Oct 3, 2007 at 9:21 AM | Permalink

    #132 Kenneth Fritsch:

    It appears that you are saying that OpenTemp is a convenient program and means of doing trends and the like using GISS data sets.

    That’s essentially right. It is an automated method of ingesting USHCN or GHCN data, processing it, averaging it across regions, and spitting out results.

    I would have to ask then why you have concentrated on the “raw” data set for the CRN category analyses?

    Initially I used GHCN raw because I had it available and I wanted to get results quickly. Now that I have added the USHCN parser I think it is best to use TOBS adjusted data because TOBS bias is a large effect that is not (too) controversial.

    The USHCN urban adjustments fundamentally change the trends in the raw data to correct for UHI. My preferred method of understanding UHI and microsite issues is to isolate them without any fancy processing.

    I suppose your reference to geographical averaging and adjusting may have confused me.
    Was that recent discussion confined to geographical adjustment for calculating trends and not to compare with some of the adjustments that are fundamental to GISS?

    That’s right.

  135. Posted Oct 3, 2007 at 9:29 AM | Permalink

    #133 Joe Bowles:
    I basically agree with everything in your post. We need to eventually do most of the analyses that you describe. As each analysis is done it should be discussed here as a quick form of peer review before moving on. Down here in the comments (as opposed to the main article), I am a big proponent of “releasing early and releasing often” so that the analyses can improve based on feedback.

    I can modify OpenTemp to extract only a range of dates for each station. Can anybody (Anthony? Kristen?) get me a list of when each station was given its current rating?

    As far as rural CRN1 and CRN2 sites, do we have any reason to believe that many of them were previously CRN4, CRN5, or urban?

  136. Kenneth Fritsch
    Posted Oct 3, 2007 at 9:58 AM | Permalink

    I am having problems understanding the choice of temperature regime period selections. When I use the entire USCHN Urban data set from 1960-2005 I calculate: Trend = 0.039 and R^2 = 0.39. When I do the same calculation for the period 1975-2005 I calculate Trend = 0.057 and R^2 = 0.33.

    Now I was led to believe that the regimes were selected based on visual linearity considerations, but the actual calculations seem to provide better linearity periods. This is only a minor point for me as I would prefer to be able to measure something closer to the snapshot time than extended time periods.

  137. Posted Oct 3, 2007 at 10:23 AM | Permalink

    Kenneth Fritsch:
    Hmmm, you’re right. The R-squared value is lower for 1975-2005 than for 1960-2005 in my results too. That’ll teach me to trust my eyes to identify trends and patterns. (I need to re-learn that lesson every couple of weeks it seems).

    I can take some consolation in the fact that R-squared for 1935 to 2005 is about 0.03.

    So, back to the drawing board. What metrics should we use to compare CRN categories? Should we start at 2005 and work backwards until we reach a maximum R-squared? (With a constraint of requiring at least N years). We could repeat the procedure to find prior key periods.

  138. steven mosher
    Posted Oct 3, 2007 at 10:38 AM | Permalink

    RE 131. JohnV I think I solved the puzzle and found the flaw.

    All we have from GISS is an ANOMALY from the GISS 1950-1981 Mean.

    When you compared CRN12 to GISS in anomaly form what MEAN did you use?

    1. You cannot have used the GISS mean. They dont publish it. They publish
    Temp-X. Nobodoy knows what X is.

    2. I assume you created the CRN12 anomaly using the CRN12 average from the 1950-81 period.
    Temp-Y.

    IF, the average of CRN12 ( WRT 1950-1981) is equal to the average of GISS ( WRT to 1950-1981)
    then X is equal to Y then the anomalies are comparable.

    Simply: ANOMALY GISS is equal to (Yearly Giss) – (Giss average for 1950-1981)
    ANOMALY CRN as you calculated it is equal to ( Yearly Crn12) – (CRN12 average for 1950-1981)

    You subtracted different amounts.

    Solving this is simple.

    1. What is the AVERAGE TEMPERATURE you subtracted from the CRN12 series to create the anomaly chart?
    A. The average of GISS temp 1950-1981
    B. The average of CRN12.
    C. something else.
    2. What is the average temp GISS subtracted from its series?

    HERE is what I did.

    1. Calculate the temp using OpenTemp for all 1221 sites.
    2. Calculate a MEAN for 1950-1981. This figure was 11.2021875C
    3. Calculate the Temp for CRN12.
    4. Create an anomaly for CRN12 using OpenTemps 1950-1981 Anomaly. ( CRN12-11.2021875)
    5. Download GISS data. 1880-2005. This is in anomaly form WRT to 1950-1981.
    6. Smooth the two using the same smoothing. Trailing MA.
    7. Difference them.

    To compare OpenTemp anomaly for CRN12 with Gisstemp anomaly, you have to reference the
    anomaly to the same mean. You cannot construct a CRN12 anonomaly referenced to its
    1950-1981 mean and compare it to a GISS anonmaly that is referenced to ITS 1950-1981
    Unless you believe the means are the same.

  139. steven mosher
    Posted Oct 3, 2007 at 10:52 AM | Permalink

    RE 136.

    Selecting a regime. I select the last two years to fit to a linear model because it seems “natural”

    “picking” a regime because it looks linear and then fitting it is not random sampling. I pick these two
    points, they look linear! oh wait, throw in the third one there… you get the drift.

    You can pick a linear regime and “fit the data”

    Here is an iteresting paper: http://www.beringclimate.noaa.gov/regimes/Red_noise_paper_v3_with_figures.pdf:

    Due to inertia in red noise processes determined by the value of ρ, they are characterized by extended intervals or “runs,” when the time series remains above or below its mean value. These runs can be misinterpreted as different “regimes.” Figure 1b shows a realization of AR1 process with ρ = 0.7. The regime shift at i = 28 could be statistically significant at the 0.05 level based on the t-test, if the data points were independent. Therefore, it is necessary to either recalculate the significance level by taking into account the serial correlation or use the so-called “prewhitening” procedure, which consists of removing red noise by using the difference (Xt – Xρˆt-1). Both cases require the estimate of the AR1 coefficient, which can be obtained using the entire series of observations.

  140. steven mosher
    Posted Oct 3, 2007 at 11:14 AM | Permalink

    RE 122 Joe Bowles. Thanks. JohnV , ClaytonB and Kristin have all been great additions.

  141. Joe Bowles
    Posted Oct 3, 2007 at 1:24 PM | Permalink

    Gentlemen:

    I have been thinking about this all day and have come to the conclusion that we might see more using the median of the average temperature by day as the starting point for identifying trends. The median is an unbiased central tendency descriptor for non-normal distributions while the mean is a biased central tendency for non-normal distributions. Since the mean is influenced by extremes, it may be hiding some of the information. If nothing else, it would give us a clue as to which direction the data is skewed (direction of the mean versus the median) and might suggest just how badly skewed the data is.

    My guess is that each station is relatively independent and we probably have different distributions by location. Logically, we should compare within the same general geographic parameters to tease the behavior of the CRN 1,2 with respect to the CRN 5. I also like the process of doing it with the full data set and then, just excluding the CRN 5s.

    If the distribution is normal, 1 1/2 times the interquartile range will approximate a 90% confidence interval, though I have no reason to believe that the data would be Gaussian. My thought is that comparison on the basis of medians is consistent with the kind of data (interval) and that the differences might show up more cleanly. Using medians should get rid of unusual extremes in the data and provide a more consistent basis for examining the differences.

    My suspicion is that if the differences are significant, it will be more obvious using medians rather than means. Other than calculating the medians of the daily averages, the rest of the analysis could be performed as you are already doing.

    Joe
    Joe

  142. Posted Oct 3, 2007 at 1:25 PM | Permalink

    steven mosher said:
    “JohnV I think I solved the puzzle and found the flaw.”
    What flaw? Is the close agreement between GISTEMP and CRN12R (for the USA lower 48) a flaw?

    Shifting the temperature plots to a reference period is merely a convenience to make it easier to compare trends. It has no effect on the trends, and the trends are what matter. The absolute temperature has little meaning (as Larry has told me repeatedly) but the trends determine if there is warming or cooling and that’s what it’s all about in the end.

  143. Posted Oct 3, 2007 at 1:31 PM | Permalink

    #139 steven mosher:
    Ok, ok, I get the point. :)
    Using “key periods” for trends is probably a bad idea.
    Can we all agree on something like my trailing 20yr trend plot? (Second plot from the bottom here). I prefer it over the anomaly plot because it’s easier to spell. :) Oh, and also because trends can be compared for any year regardless of the net divergence between sets of results.

    Excuse my uncharacteristic smiliness – I finally finished an overdue contract this morning.

  144. Steve Sadlov
    Posted Oct 3, 2007 at 1:58 PM | Permalink

    RE: “SteveSadlov likes 1930 to 2005.”

    Actually, what I would really like would be something like 1500 – 2005, or even better, 0 – 2005. But I am a pragmatist.

  145. Steve Sadlov
    Posted Oct 3, 2007 at 2:05 PM | Permalink

    RE: “Sample size is not the issue.”

    OMG! We had a failure! SHUT DOWN THE LINE!

    And the converse …..

    Wellllll …. you only tested 5 and got one failure. This means precisely what? ….. SHIP IT!

  146. Kristen Byrnes
    Posted Oct 3, 2007 at 2:25 PM | Permalink

    John V 135,
    The CRN 1,2 stations are already done. I have a ton of homework tonight and will ask someone to do the CRN 5’s.

  147. Joe Bowles
    Posted Oct 3, 2007 at 2:25 PM | Permalink

    I gave it a try just looking at the medians for years 1991-2005 (twleve months ended December 31). The data came from John V’s monthly tables for CRN 1,2 and CRN 5. The slope differences show up, but the R^2 for all of them are lousy. The regression accounts for about 7.4% of the variance for CRN 1,2; 19% for CNR 5; and 19.8% for the difference.

    Median, slope and RSQ were calculated using Excel 2007 formulas.
    CRN 1,2 CRN 5 Diff

    1991 12.42 12.53 0.11
    1992 11.51 11.99 0.48
    1993 10.92 11.16 0.24
    1994 12.08 12.24 0.16
    1995 11.44 11.6 0.16
    1996 11.09 11.63 0.54
    1997 10.65 11.05 0.4
    1998 11.8 12.23 0.43
    1999 11.65 12.23 0.58
    2000 11.83 12.45 0.62
    2001 12 12.51 0.51
    2002 11.57 11.84 0.27
    2003 11.74 12.53 0.79
    2004 12.17 12.55 0.38
    2005 12.17 12.48 0.31

    Min 10.65 11.05 0.11
    Q1 11.475 11.735 0.255
    Median 11.74 12.23 0.40
    Q3 12.04 12.495 0.525
    Max 12.42 12.55 0.79

    Mean 11.669 12.068 0.399
    Stdev 0.477 0.489 0.187

    slope 0.0301 0.0493 0.0192
    R^2 7.4% 19.0% 19.8%

    It looks like CRN5 has a bias of about 0.0192 per year.

  148. Kristen Byrnes
    Posted Oct 3, 2007 at 2:30 PM | Permalink

    Rural CRN12 Stations:
    42572383001,34.7,-118.43 Fairmont CA (Not Surveyed, aerial photographs only but looks good) / No changes in location or equipment / records back to 1931
    42572694004,44.63,-123.2 Corvallis State University OR / no change in location or equipment / records back to 1948

    42572376006,35.27,-111.73 Fort Valley AZ / Equipment: 2004 – current “other temperature equipment” / 1986 – 2004 MMTS (LIG backup) / before 1986 LIG / Location same throughout / records back to 1948

    42572273004,31.7,-110.05 Tombstone AZ Equipment: MMTS 2002 – current / LIG 1933 – 2002 / Location: same throughout / records back to 1933

    42574501001,38.33,-120.67 Electra ph CA / Station closed 1997 / Equipment LIG throughout / Location: same throughout / records back to 1931

    42572489002,39.97,-120.95 Quincy CA / Equipment: MMTS 1999 – present / LIG before 1999 Location: last change 2004 / records back to 1931

    42572786006,48.35,-116.83 Priest River Experimental Station ID / no location or equipment changes / records back to 1911

    42572533004,41.63,-84.98 Angola IN / Location: Location move reflected in elevation change 1977 / Equipment 1985 – Present: MMTS / LIG before 1985 / records back to 1887

    42572743002,46.33,-86.92 Chatham experimental farm MI / Location: last move (short distance) probably not a problem / Equipment: LIG throughout / 1987: STN ESTABLISHED TO COMPARE WEATHER READING WITH CHATHAM EXP FARM 20-1484-2, 1.1 MI TO THE NNW / This station is a problem, the correct name is Chatham Exp Farm 2. The actual station is Chatham Exp Farm, which was active from 1948 to 1988 with no recorded station moves and LIG equipment throughout station history.

    42574341005,45.13,-95.93 Milan 1 nw MN / Location same throughout / Equipment Nimbus 2006 to current / LIG before that / records back to 1948

    42574341007,45.58,-95.88 Morris wc Experiment Station MN / Location: same throughout / Equip: LIG throughout / records back to 1948

    42572235004,31.97,-91 Port Gibson 1 ne MS / Possible problem with either MMS or Surfacestations.org / MMS exposure & obstruction field do not match photos, MMS says: MMTS 260/10 F&P 90/7 TREE 100/75/29 140/100/25 HOUSE 160-220/90-120/14-7 TREES LINES 165-200/175-125/24-24 200-280/120-200/30-15, photos do not show these obstructions / Location changed 2001 / Equipment: MMTS 1988 to current / LIG before 1988 / records back to 1948

    42572677004,45.92,-108.25 Huntley Experimental Station MT / Location and Equipment (LIG) unchanged / records back to 1951

    42572488003,39.45,-118.78 Fallon Experimental Station NV / Location changed in 1992 / equipment the same (LIG) throughout

    42572662003,43.97,-101.87 Cottonwood 2e SD / Location unchanged / Equipment: MMTS 1986 – current / LIG before 1986 / records to 1948

    42572572005,40.55,-111.5 Snake Creek PH UT / Location: no changes / Equipment: Thermograph 1997 to current / LIG 1997 and before / records to 1948

    42572403002,38.9,-78.47 Woodstock 2 NE VA / Location change 1994 / Equipment MMTS 1993 to current / LIG before 1993

  149. Kenneth Fritsch
    Posted Oct 3, 2007 at 6:22 PM | Permalink

    The concept of temperature regimes (which I believe usually leads to bad statistical approaches) got me to thinking harder about the snapshot quality assessments versus the time period of trend differences that might show maximum differences. At face value we can only state something about differences as we see them in the present even though we probably know that the quality differences involving features like pavement and parking lot surfaces have been in place for a while. Unfortunately we also know that the changes in quality probably happened sporadically over a period of time and over the population of stations.

    The proposition I propose is where would we look if we thought the changes we see in the snapshots currently took place, in effect, a few decades ago and then for the following decades continued with a higher absolute temperature but at the same time showing a relatively constant measuring environment under which the trend over these decades was similar to the higher quality stations. We would expect the trend to be captured most efficiently in the period during which the quality was going bad for most of the currently designated poorer quality stations and that would not necessarily be in the recent time.

    My question would be, before I reveal my preliminary snooping results, whether any one here can make a case for the scenario that I outlined above. I do not want to dendro this analysis after the fact, but perhaps I already have.

  150. steven mosher
    Posted Oct 3, 2007 at 7:28 PM | Permalink

    RE 149.

    Well, Kenneth, I’ve Pondered this for some time. My sense is this. If microsite Bias came in like
    a step function of 5C it would a simple matter to detect.

    If it came in gradually since the 1950s, buildngs here and there, some ashphalt, more runways more planes

    it wuld be harder to detect.

    Then, realize that some of these warming and cooling processes are mitigated and modulated by oter processes.

    The temperature is a continuous function. It gets quantized into a daily high and daily low. During that period
    you might get a microsite shock. The microsite bias, while real, while grounded in accepted climate science,
    might be very hard to find, especially after monthly, yearly, averaging..

    That’s one way I see of reconciling JohnVs finding and yours and mine.

    Make sense? John Sees no substantial difference between GISS and CRN12 and I’m see effects when 5s are dropped.

    So, is the effect of microsite REAL, but hard to pull out from the noise? And if we do drop 60 bad sites
    can we actually improve the S/N.

    Sorry, long day

  151. Joe Bowles
    Posted Oct 3, 2007 at 7:31 PM | Permalink

    Kenneth:

    My prejudice is that we should have some careful definitions to discern what constitutes the data being bad…as well as some specific reason to set the time frame. The issue of “cherry picking” basically boils down to selection of time periods which optimize the result. We basically have a hypothesis that because of siting problems and non-conformance with the standards, a specific class, i.e. CRN5, is adding a bias to the temperature record.

    The selection of the time frame needs to have an a priori basis. That might be the time frame that encompasses the point at which the majority of stations changed location; changed equipment; or whatever. We need to think of it as an experiment with the null case being no difference.

    At some point, it might be helpful to run a factor analysis on the source data to see what the relative contribution of the different factors identified on this site are. We have a variety of factors which may be affecting the data. Equipment, lights or no lights, altitude, locational proximity to other sites, and TOBS are just the start of the factors. Unfortunately, I don’t have the program at the moment or I would give it a shot.

    I think the more attention we give to construct validity in designing the experiment, the better. There is always a tendency to go fishing and then, trying to build a case based on ad hoc information uncovered in the analysis. I suspect that is the problem with a lot of the stuff coming out of NASA et al. We need to do better.

    I have the same problem with just arbitrarily running the data in longer time frames to reach the level of significance we would like to see. For one thing, I am not convinced that micro-site problems account for a significant portion of the variance, though I still think it is important and needs to be carefully examined.

    On the other hand, I think John V has the right idea to try things and release the results as they become available. There are so many factors in play that it may be the combined affect rather than any individual factor.

    I have been involved in business valuation for many years. I have seen many cases in which the individual assumptions looked reasonable, but the combined effect of the combination of factors resulted in unrealistic values. My guess is that the same thing is happening in the temperature record. We have to examine each assumption individually…and also in combination. Sadly, in the temperature record we don’t have an independent source to test the reasonableness like we do in valuation where we can look at the market for guidance.

    So go ahead and show your cards…but think about how we can structure an a priori case for why we are looking at it in the way we are.

  152. steven mosher
    Posted Oct 3, 2007 at 8:02 PM | Permalink

    RE 126.

    JohnV I just settled the matter of cherry picking for my self by only
    sending out data from 1880 to 2005. Gave me peace of mind. I think I posted this to Anthony once who chided me about going
    back to 1880. If somebody else wants to slice times my conscience
    is clear. Here’s the data. here’s what I did. Here’s how to repeat. As soon as I can figure out how to
    post charts I’ll just post charts, Instructions, and leave the commentary to others.

    Now, I am starting to use your tool for individual site analysis.. I wont put that stuff out until
    I run it by you… ok?

  153. Kenneth Fritsch
    Posted Oct 4, 2007 at 6:01 AM | Permalink

    Re: #151

    Joe Bowles, you frame my dilemma near perfectly and make suggestions with which I can agree. I need to do some more work to firm up my mining (snooping) of the data. In the meantime, I was hoping to hear from a CA participant with a firm a prior explanation. I would certainly caution against my analysis having much statistical validity at this point.

  154. Geoff Sherrington
    Posted Oct 4, 2007 at 6:39 AM | Permalink

    Re Steve # 47

    In the mining analogy, one can choose the interval down the drill core to be as small as 0.1 of a m (practically). This allows estimations of range from very short inetervals up to large intervals (from one end of the ore deposit to the other). So, it is easier to work out how far away a point can be before it loses its ability to predict for another point, weighted which way or whatever. I have been looking, without success, for a climate paper which systematically takes a cluster of stations, some close together, some well apart, and does the maths we were using in the 1970s. The methods work in 3D so time can be a variable as well, or altitude, as well as lats/longs. I remain totally unconvinced from the papers I have read that stations 1200 km apart have predictive relevance to each other. If close stations were properly treated my gut feeling is that prediction ends in well under 200 km. And as you know, contouring methods using triangular grids were also studied intensively in the 1970s. References to David, Matheron et al at Fontainebleau, France, then follow the trail.

  155. steven mosher
    Posted Oct 4, 2007 at 7:03 AM | Permalink

    re 131 johnV.

    here is vector of “giss” temps: 1,2,3,4,{5,6,7},8,8,8,10
    Here is vector of “crn12″ temps:0,1,2,3,{4,5,6},7,7,7,9

    Now, lets take the average of the center, 5,6,7 =6
    Create the aaomaly: -5,-4,-3,-2,-1,0,1,2,2,2,4: WRT 6
    Now anomaly of sample:-6,-5,-4,-3,-2,-1,0,1,1,1,3: WRT 6
    Now anomaly WRT 5:-5,-4,-3,-2,-1,0,1,2,2,2,4: ( essentially the sample mean)

    What it does is shift the anamalies in Y. The slope does chnage as you note, But it can
    distort perceptions of the “match” between the anomalies. ESpecially when people do
    Delta Charts: In case 1, where we subtract the mean of all “giss” from both series you get
    a delta that looks like, -1,-1,-1,-1,-1,-1,-1 etc and some would comclude CRN was colder. It was
    In case 2 the difderence is 0,0,0,0,0, and people conclude that CRN wasnt warmer or colder. That
    would be wrong. It doesnt go to the trend, I get that, BUT if CRN12 is colder during the 1950-1981
    period, colder than the GISS mean of that period ( say its colder by .1) then by subtracting the
    CRN12 mean from its series you mask the difference in temps. Not the trend of course. Same thing
    happens for the 5s. If the 5s are warmer in that period, Then by sutracting that HIGHER mean from
    the sample it makesthe 5s look warmer than giss.

    Make sense

  156. Joe Bowles
    Posted Oct 4, 2007 at 9:09 AM | Permalink

    Re 153

    Kenneth:

    I think Kirsten’s work may provide an a priori basis for time selection. I keep thinking about the upward bias of the MMTS systems. Based on what Anthony posted on his web site, those systems seem to set a discrete upward bias in the temperature, and seem to get worse with time. We have a lot of those in even the best sites, so maybe we need to figure out the adjustment for them, adjust the data for the MMTS contamination, then, try again to see what the microsite effects are.

    At this point, I wonder if we should do comparisons within class between MMTS and LIG to see if there is a discernable difference. I would gather that the bias of the instruments would be similar across sites and classification, but I would still look within class. If that bears fruit, then, one of the major contaminents can be removed.

    If we set a rule…even if it is arbitrary…for selection of a time frame, I think it gives us more power. The cherry picking complaint is that the selection of the endpoints is based on the end result, whether to increase or decrease the temperature trend, rather than on some other basis.

  157. Joe Bowles
    Posted Oct 4, 2007 at 9:11 AM | Permalink

    Kristen:

    My apologies for misspelling your name. My fingers don’t always hit in the order I intend…and my proofing skills obviously leave something to be desired.

  158. Joe Bowles
    Posted Oct 4, 2007 at 10:13 AM | Permalink

    I wonder if we could better understand the effects of rating bias by taking a cluster of stations within a fairly close geographic dispersion and do pair comparisons. We know that CRN5 yields some upward bias in temperatures and seems to affect the trend, but we haven’t sorted out the effects as you move upward or downward through the classes.

    Hansen used 40 stations in his adjustment process, but it isn’t intuitively clear how the adjustments were made on this basis. My guess is that he was trying to use a large sample set under the assumption that it would cover a multitude of sins. On the other hand, using a large sample also hides a lot of information.

    In essence, I think we might pick up a lot more insight based on direct pair comparision within a specific area and then see how the implied adjustments would affect the adjustments in other geographic areas.

    I see no logical reason that the adjustments would be the same since weather patterns are local/regional. I am thinking that once we figure how how to make the adjustments on the micro-level, we could extend the methodology to the macro. The number of pairs done across all stations would me massive, but doing them in geographically defined areas would be practical.

    I think we can only get limited information based on CRN class alone since we have the compounding problems of equipment, TOBS, and UHI effects at a minimum.

  159. steven mosher
    Posted Oct 4, 2007 at 10:24 AM | Permalink

    RE 158. JOE B. That is what I am presently doing.

    The discussions over CRN12 versus GISS are getting to dang heated for me and Since
    I am a hot head I figure it best for me to steer clear of that rumble. I’m going
    to try to focus on determining the extent and magnitude of microsite bias. As best
    I can. Anthony and his team deserve somebody looking into that specific question.

    If it impacts the US land record fine. If not, fine.

    Kenneth has done some work down this vein using something akin to your suggestion.

    I will do my own thing and post accordingly.

  160. Posted Oct 4, 2007 at 1:30 PM | Permalink

    #155 steven mosher:

    But it can distort perceptions of the “match” between the anomalies

    I believe the whole definition of anomalies is based on changes from a reference period for the series being analyzed. A latitide/longitude/elevation-compensated comparison of absolute temperatures could also yield some interesting results, but I am focussing on the trends right now.

    Since GISS does not publish the 1951-1980 average, any absolute temperature comparisons would need to exclude GISTEMP. I don’t see that as a big loss since the GISTEMP results are not broken down by CRN anyways.

    The discussions over CRN12 versus GISS are getting to dang heated for me and Since I am a hot head I figure it best for me to steer clear of that rumble.

    I wish I would have followed you over here earlier.

  161. Posted Oct 4, 2007 at 1:31 PM | Permalink

    I’m feeling some urgency to get the uncertainties due to geographic coverage figured out. I would really appreciate it if somebody could put together a list of rural CRN3 stations. Thanks.

  162. Kenneth Fritsch
    Posted Oct 4, 2007 at 1:33 PM | Permalink

    I looked at (snooped) some time period trends using the USCHN Urban data set over the extended period from 1960-2005. I have a problem going back beyond 1960 because it is at that this point that significant numbers of data points are missing from this data set – and increasingly missing as one goes back in time.

    The CRN combinations that I looked at were the CRN45, CRN12 and CRN3 combinations and selected for the purpose of comparing larger samples and comparing them in order of audited quality. I looked at the entire 1960-2005 time period to present an overall look at trend differences and then went back in time to determine whether one could find a period of constant or nearly constant trends by CRN category and periods with changing trends by category. The trends were calculated and then adjusted geographically by my state by state procedure. The results are presented in the table below.

    The period 1975-2005 shows very little differences in the 3 CRN combination category trends while the period 1960-1975 shows a large difference between CR45 and CRN12 with CRN3 in the middle as would be expected from the quality ratings. The 1960-1985 shows similar orderings and differences as the period 1960-1975 with somewhat smaller differences even when considering the longer time span of this period.

    A simple explanation for the result shown here could be attributed to the quality differences detected by the Watts team snapshots occurring mostly 20 to 30 years ago and then remaining at a constant quality level over the intervening time. Before one could put any stock into this supposition one would need some independent evidence that this indeed was close to the general case. Further work is required to put any statistical significance to these results by going back and looking at differences between randomly selected groups from this population and for the time periods analyzed.

  163. Kenneth Fritsch
    Posted Oct 4, 2007 at 2:06 PM | Permalink

    Re: #158

    I wonder if we could better understand the effects of rating bias by taking a cluster of stations within a fairly close geographic dispersion and do pair comparisons.

    I was thinking along the same lines awhile back (still am, for that matter) and found 11 paired stations of CRN5 versus CRN12 that were, as I recall now, within a degree of latititude and longitude and mostly closer. I initially found a difference comparing them over the 2005 and 2006 time periods, but after Steven Mosher had to remind me that altitude would need to be considered, I went back suspecting that stations that close would be at nearly the same elevations. This flat lander was dead wrong and the adjusted results whil showing large differences in pairs show nbo overall difference on average. I used an elevation adjustment of 1.99 degrees C per 1000 feet of elevation difference. A number of these pairs required large temperature adjustments. I am aware that the proper elevation adjustment can varying depending humidity and other factors.

    I would like to go back and take another look at paired differences for if one can obtain the right pairs or a correct adjustment for pairs, the difference should be there if the CRN ratings are indicating a warm bias for poorer quality staions.

    Another consideration is that poorer quality stations might have biases in both directions which should be evident in differences of the variances between CRN categories.

  164. Joe Bowles
    Posted Oct 4, 2007 at 2:22 PM | Permalink

    Re 163

    I don’t think we can make the pair comparisons based on annual data. The annual data obscures too much information, especially since some of the microsite differences are probably seasonal. For direct comparison, we may need to move down to monthly or seasonal data.

    I see the logic of making an altitude adjustment, but it isn’t clear to me that there is one adjustment that works everywhere…especially in places with differences in humidity. My thought is that we should try to extract that directly along with the other components. We may want to get the observer reports and lay out differences between the sites, along with the CRN classification. We might get some interesting results trying multiple regression on the various site components.

  165. Sam Urbinto
    Posted Oct 4, 2007 at 2:36 PM | Permalink

    Temp, humidity, altitude, surroundings, wind speed. Need to know ‘em.

  166. steven mosher
    Posted Oct 4, 2007 at 2:52 PM | Permalink

    JohnV I will mail you the CRN3. I thought I gave you my whole database.

    TEST GRAPHIC

    Chart of OpenTemp ALL, OpenTemp CRN5, and CRN12R. Using GHCN Rural. 5year moving average, Trailing

  167. Posted Oct 4, 2007 at 2:56 PM | Permalink

    steven mosher:
    Thanks. I’m looking for the *rural* CRN3 stations. Do you have those split out?

  168. Clayton B.
    Posted Oct 4, 2007 at 3:14 PM | Permalink

    mosher,

    you doubled up on your “http:\\”. Right click the red x and select properties – you’ll see what I mean.

  169. Joe Bowles
    Posted Oct 4, 2007 at 3:31 PM | Permalink

    Re 165

    We might want to consider the high and low temperatures separately, too.

  170. Kristen Byrnes
    Posted Oct 4, 2007 at 3:32 PM | Permalink

    John V,

    I think including CRN 3, even if it’s rural, is a big mistake because you are trying to eliminate the micro site biases all together. We are already working on getting you enough data, please lend us your patience.

  171. Posted Oct 4, 2007 at 3:52 PM | Permalink

    #170 Kristen Byrnes:
    I agree that the CRN3 stations should not generally be grouped with the CRN12. I would like increase the sample size only for estimating the uncertainty. I can run many random sets of 17 stations through OpenTemp and use the differences between them to estimate the error bars on the rural CRN12 results.

    The inclusion of CRN3 stations in the random sets will increase the uncertainty estimates. It’s better to have over-report the uncertainty though, so I’m ok with that.

  172. steven mosher
    Posted Oct 4, 2007 at 3:57 PM | Permalink

    RE 167. Rural CRN3?

    No, I dont have those. I sent you all the cuts I did. My next cuts are going to be Instrument cuts
    and location cuts. ( christ its a multiple linear regression in pieces )

    I am very very leary ( but not adverse ) about the Urban/rural distinction. I leary about it because There are two indices
    that slice this data: 1980 population. 1995 nighlights.
    If you like I think I can cobble together datasets to let you do this. It’s handwork, and so will come
    with caveats.

    I have a couple cuts I didnt send you.. ALL_except_CRN12 and some stuff I was doing on on the Titusville
    site. that I havent tested

    It’s Relative to Anthony’s work. here is what I am doing. ( kinda like Kenneths work).

    PICK a class5. Use that site as a center. Pick sites within (Xkm) I picked approx 2deg in lat/lon

    That gives me a sample that is local to the class5. Now, compare the Class5 site to all its neighbors.

    There are some decisions here so its a work in progress and I dont mind being wrong.

    The best thing for me to do is an update of Anythonys excell. Might take a day or so.

    My sense. The class3, given the phenomenology of microsite bias should not be thrown out.
    We are talking a small bias that shows up in a bursty way ( most cases) and so it will just
    look like noise.

    Whatever bias CRN3 have IT is SUB NOISE. ( I say this stuff to see who comes back with a strong
    argument)

    So lemme know what you want

  173. steven mosher
    Posted Oct 4, 2007 at 4:00 PM | Permalink

    RE 168. Pass me the dunce hat. Test #2.

  174. steven mosher
    Posted Oct 4, 2007 at 4:10 PM | Permalink

    First, let me frame this. Based on decades of climate science, we have good reason to believe
    that the characteristices of class5 stations will bias the temperature measurements over time.

    1. Suggest your methodology to identify and quantify this bias.

    2. We have a program. It is Open. you can ask me or John or Clayton to run any comparison
    as long as you provide the list of stations. It takes me 5 MINUTES.

    3. I’m gunna do the comparisons I think are neat. You dont like it? Send me the station list you
    want to run. I will be your data monkey. Send the list. Station ID, lat,lon.
    txt file. easy as pie.

  175. steven mosher
    Posted Oct 4, 2007 at 4:11 PM | Permalink

    RE 173. Mosher your charts suck!

  176. steven mosher
    Posted Oct 4, 2007 at 4:28 PM | Permalink

    re170. Which will be warmer?

    1. CRN12 or CRN3
    1.CRN12R or CRN3R.

    If CRN12 is warmer what will you conclude?
    If CRN12R is warmer what will you conclude?

    What will the diffeence in trend be?

    Hypothesize.

  177. Posted Oct 4, 2007 at 4:40 PM | Permalink

    #176 steven mosher:
    “Warmer” really has no meaning (as I keep saying). To further complicate the idea of “warmer”, remember that each series is offset to the mean of all other series before it is added. (See my original post for details).

    In termes of trends, I would guess the following order of warming trend:
    CRN12
    CRN3
    CRN12R
    CRN3R

    That is, I guess that UHI would have more effect than microsite differences between CRN12 and CRN3. The same would not hold for microsite differences between CRN12 and CRN5.

    Like I said though, these are just guesses for fun.

  178. steven mosher
    Posted Oct 4, 2007 at 5:22 PM | Permalink

    RE 177. Crap. I forgot about that Offset. Warmer have a meaning, if we find something odd, but
    Warming ( trend ) Is more relavant to the AGW stuff.

    basically, your offset calc, means We are obligated to look at trend. Correct?

  179. steven mosher
    Posted Oct 4, 2007 at 6:32 PM | Permalink

    I’ll look at those when I get a chance.
    I’m tying to make that CRN3R file. It’s a lot of editing

  180. Clayton B.
    Posted Oct 4, 2007 at 7:08 PM | Permalink

    179,

    I’m tying to make that CRN3R file. It’s a lot of editing

    Would it be easier if the DB was setup for quick queries ;)? By the way, any luck with that file I sent?

  181. steven mosher
    Posted Oct 4, 2007 at 7:17 PM | Permalink

    RE 180… I havent looked at it. Between this and work its been like 20hours day on the computer

    The file with the rural data is a nasty file. Its on GISS.

  182. steven mosher
    Posted Oct 4, 2007 at 7:21 PM | Permalink

    A musical Interlude. Forgive me.

  183. Posted Oct 4, 2007 at 9:21 PM | Permalink

    steven mosher:

    Crap. I forgot about that Offset. Warmer have a meaning, if we find something odd, but
    Warming ( trend ) Is more relavant to the AGW stuff.
    basically, your offset calc, means We are obligated to look at trend. Correct?

    Of course not. You can run OpenTemp without the “/offset” option and it will skip that offset. It will probably compare better with GISTEMP and Steve McIntyre as well, since I don’t think either of those methods attempt to compensate for hot or cold stations.

  184. Posted Oct 5, 2007 at 8:11 AM | Permalink

    Key Trends Revisited

    Kenneth Fritsch said:

    I am having problems understanding the choice of temperature regime period selections. When I use the entire USCHN Urban data set from 1960-2005 I calculate: Trend = 0.039 and R^2 = 0.39. When I do the same calculation for the period 1975-2005 I calculate Trend = 0.057 and R^2 = 0.33.

    When I first checked this I got similar results. This morning I realized that I was using Excel’s RSQ() function with anomalies as the y’s and years as the x’s. That implies that I was calculating the correlation between the year and the measured temperature, not the correlation between a linear fit of the temperature and the measured temperature.

    This morning I redid the comparisons against a linear trend. I used the automatic trendline available in Excel charts. The results are a little different:

    CRN12 TOBS, Single Year
    1960 – 2005: R^2 = 0.274
    1975 – 2005: R^2 = 0.277

    The noise in the signal is drowning the trend. I did the same using a 5yr centred average:

    CRN12 TOBS, Five Year Centred Average
    1960 – 2005: R^2 = 0.635
    1975 – 2005: R^2 = 0.676

    This doesn’t mean a lot in the debate about key trends, but it does at least say that my quick visual inspection of trends was accurate in this case. That is, the correlation with a linear fit is stronger for 1975-2005 than for 1960-2005.

    Note also that Kenneth’s results were generated from all stations with urban adjustments, while mine come from only the rural CRN12 stations without urban adjustments.

  185. Joe Bowles
    Posted Oct 5, 2007 at 8:41 AM | Permalink

    I did a re-read of some of the prior posts and Anthony in 112 said something we may have missed”

    A big case was made by two presenters at Pielke’s recent conference to do away with Tmin and only look at Tmax due to Tmin having so many possible microsite biases compared to Tmax. Perhaps running opentemp on Tmax only with CRN12345 – CRN1234 would be a good test.

    If we are looking for microsite problems, we might want to run on Tmin since it is where most of the problems lie per the above quote. That might provide a clearner way of making comparisons to tease out the issues.

  186. Joe Bowles
    Posted Oct 5, 2007 at 8:44 AM | Permalink

    Re 184

    Thanks for pointing that out John V. I missed that entirely when I was playing with some of the data. It is very helpful.

  187. Posted Oct 5, 2007 at 8:53 AM | Permalink

    #185 Joe Bowles:
    I’d like to do some analyses with Tmax as well.
    If you’re using OpenTemp and USHCN data files, it’s a simple switch on the command line:

    /ushcn1={filename} for raw Tmax
    /ushcn2={filename} for raw Tmin
    /ushcn3={filename} for raw Tavg

    /ushcn1+={filename} for Tmax with TOBS adjustment
    /ushcn2+={filename} for Tmin with TOBS adjustment
    /ushcn3+={filename} for Tavg with TOBS adjustment

  188. Posted Oct 5, 2007 at 9:00 AM | Permalink

    OpenTemp v1 RC1
    The first release candidate for OpenTemp is now available:

    http://www.opentemp.org/_release/v1rc1.zip

    I have added a few new features since Tuesday:
    – ability to filter station data by start and end dates (in stations file)
    – comments in stations file are supported
    – optional prefix on output files (easier to identify result sets)
    – random selection of N stations from the complete list (for Monte Carlo analysis)

    There is a readme.txt file in the zip archive. I have included it below. I will try to work on better documentation.

    To confirm that the analysis code has not changed, I ran a few of my old analyses. I would appreciate it if other users (steven mosher) could check their old results as well.

    Until I get the http://www.opentemp.org website properly setup, please report any bugs here. (Steve McIntyre, let me know if you’d prefer that conversation to be moved elsewhere).

    =====

    OPENTEMP v1.0 RC1
    October 5, 2007

    Usage:
    OpenTemp
    [/ghcn=GHCN | /ushcn{uso}=USHCN]
    /rgn=[region]
    /stn=[stations]
    {/stnpick=NN}
    {/prefix=PRE}
    {/offset} {/os} {/oc} {/ocs} {/oo} {/om} {/oy}
    where
    GHCN is the name of a GHCN data file to read
    (GHCNv2 monthly data files are supported)
    USHCN is the name of a USHCN data file to read
    (USHCNv1 monthly data files are supported)
    {uso} is one or two characters defining USHCN options (default is avg raw)
    (‘1′,’2′,’3′ for max, min, avg temp;
    ‘+’,’A’,’C’ for TOBS, Filnet, Confidence)
    [rgn] NOT IMPLEMENTED: is the name of a file that defines the regions to be analyzed
    (currently the program is hard-coded to analyze the USA lower 48)
    [stations] is the name of a file with a list of stations to analyze
    – one station per line with: stationID, latitude, longitude, {start}, {end}
    – {start} and {end} are optional dates in the form YYYY.MM
    – for MM, use 01 for January, 02 for February, etc
    – comment lines can be included using the pound sign (#)
    {/stnpick} (optional) pick NN random stations from the complete list
    {/prefix} (optional) use PRE as a prefix for all output filenames
    {/offset} include to calculate and use series offsets to the region average
    (offsets will be written to offsets.csv)
    {/os} include this flag to write station details for all requested stations
    {/oc} include this flag to write cell data for the region (cell location and size)
    {/ocs} include this flag to write cell-stations (stations that can affect each cell)
    {/om} include this flag to write monthly averages for each region
    {/oy} include this flag to write yearly averages for each region
    {/oscr} include this flag to write scribal variation results

    Results will be written to data files with the following names (with optional prefixes):
    monthly.csv contains monthly overall averages for each region
    yearly.csv contains yearly overall averages for each region
    station.csv contains monthly readings for every station in a convenient format
    cell.csv contains cell information for the entire region
    cellstation.csv contains cell-station interaction results
    offsets.csv contains the static offsets applied to each series
    scribal.csv contains the fraction of matching values for all series at each station
    opentemp.log contains a complete processing log (same as console output)

    Notes:
    – Each output file is written only if the appropriate /o option is used
    – If the /offset option is used:
    – Monthly and yearly data are calculated using a static offset applied to each series
    (Offset is equal to the average difference between the series and the monthly averages)
    – station.csv has series data before applying the series offset
    – offsets.csv file is written
    =====

  189. Joe Bowles
    Posted Oct 5, 2007 at 9:11 AM | Permalink

    I was looking at Anthony’s map of the dispersion of surveyed sites again. Of course, the sample isn’t random, so we can’t infer whether the same proportions of CRN5 holds to the entire population.

    However, the particular dispersion of CRN5 sites in the population is of considerable importance since they affect the adjustment process leading to the GISS temperatures. The process (to the extent I can follow it) provides for a wonderful opportunity to spread the bias of the CRN5 throughout. But we also have the influence of the CRN4 sites, which so far we haven’t focused on.
    About 70% of the surveyed sites were CRN4 (55%) or CRN5 (15%). We still have 2/3 of the sites whose class is unknown.

    Making the assumption that the remaining sites have more or less proportional distributions, that means that CRN4 and CRN5 are having a significant effect on the adjustment process that permeates the GISS data, as well as the other data sets.

    Unless we can tease out the microsite effects, we are shooting in the dark as to how good or bad the ending data sets may be. In other words, comparison to the GISS temperatures may be adding additional bias to the analysis, since their data points are “contaminated” based on their methodology.

    Does this suggest that we should only be making comparisons to the data for the identified sites and, for the moment, ignore the unsurveyed sites?

  190. Joe Bowles
    Posted Oct 5, 2007 at 9:17 AM | Permalink

    Re 187

    Thanks for the information. I will give it a try when I finish some work that seems to be piling up. For some reason, my clients seem to think they should get priority.

  191. Posted Oct 5, 2007 at 9:18 AM | Permalink

    #189 Joe Bowles:
    My approach would be to leave out the un-surveyed sites for now.
    I would like to look at rural CRN3, CRN4, and CRN5 vs rural CRN12. We need to figure out how much warming is introduced by microsite issues distinct from UHI issues.

  192. SteveSadlov
    Posted Oct 5, 2007 at 12:51 PM | Permalink

    Ping to #91 – there is only one “vertical dip-slip fault scarp” in the entire record, and it it between 1915 and the early 1920s. All other sharp slopes are “natural topography.”

  193. SteveSadlov
    Posted Oct 5, 2007 at 12:55 PM | Permalink

    In Mosher’s chart in #173 – the completely unnatural “fault scarp” is between 1920 and 1924. If this was a financial metric, it would be a major red flag. There are no other features in the entire plot that look so “unnnatural.” All other sharp slopes have a much more “stochastic” look to them. Again, this is not science, it is gut feel from 25 years in the school of hard knocks. YMMV ….

  194. steven mosher
    Posted Oct 5, 2007 at 1:23 PM | Permalink

    RE 193.

    YES, and it happens across all sites.

    Big jump.

  195. steven mosher
    Posted Oct 5, 2007 at 1:25 PM | Permalink

    RE 193.

    I think I will look at year to changes and plot the distribution of changes.

    The change you spied was .6C in a year.

    THATS ALL THE GLOBALWARMING OF A CENTURY

  196. SteveSadlov
    Posted Oct 5, 2007 at 1:29 PM | Permalink

    RE: #195 – What you’ll see on Y-Y is that it is a two (or maybe three) step jump. Expressed w/ 5 year smoothing it ends up as a fault scarp. To be fair, there is a small downward fault scarp late 70s / 1980ish. But nothing like this. Yes, yes, I’ve heard all the stuff about “stochastic process” and “multistable / quantum” system. But even with those caveats, this early 20th century jump, among all others, is the most notable. My auditor’s luck says, drill down here.

  197. SteveSadlov
    Posted Oct 5, 2007 at 1:39 PM | Permalink

    1917 is the coldest year in this portion of GISTEMP and 1921 is the third warmest. Again, according the GISTEMP, FWIW ….

  198. SteveSadlov
    Posted Oct 5, 2007 at 1:48 PM | Permalink

    While the Hansen 9/10 adjustment does pull down 1917 slightly, it’s not a spike and the magnitude is only hundreths of the deg C. Interestingly, the Hansen 9/10 definitely pulls down the 1880s and 1890s noticeably. As a result, the entry point in the 20th century has been thereby slightly depressed.

  199. SteveSadlov
    Posted Oct 5, 2007 at 2:00 PM | Permalink

    Subtracting 0.1 deg C from ~ 1890 has the result of making the 1880 – 1920 trend slightly upward. Were it not for lowered ~1890, the trend 1880 – 1920 would have actually been either flat or slightly downward. More speculatively, if all values prior to the early 1920s have a systemic error or adjustment, such that the 1880 – 1920 mean is represented as a few tenths of a degree lower than it actually was, then the impact of that was to vastly increase the apparent overall 20th century rise. If both the 1880 – 1920 Hansen adjustment is wrong, and, the pre 1920 values are all recorded from the get go as too low, then in reality, there has been very little warming during the 20th century. Lots of caveats here, obviously, and again, YMMV.

  200. steven mosher
    Posted Oct 5, 2007 at 2:05 PM | Permalink

    SteveS. This stuff is OpenTemp. So No Hansenism. Its GHCN ( USHCN for the US)..

    I think their are a couple faults like you note.

    How many should we expect to find. IN A SERIES with autocorrelation? THAT’s the question.

    .6C IN ONE YEAR? That a huge forced change, especially following a downward trend. The OIL TANKER
    dont turn like a jetski!

  201. SteveSadlov
    Posted Oct 5, 2007 at 2:11 PM | Permalink

    RE: #200 – Indeed, the tanker anology. So, in OpenTempese, indeed, no Hansenism. There, my concern is solely the fault scarp. Whereas, in GISTEMPese, I care about both the scarp and the Hansen 9/10 (or whatever Hansen D’jeur may apply). In any case, these things all act to aid the claim of “unprecedented 20th century warming.” Who knows, maybe I am being an overzealous auditor, chasing after red flags which will turn out to be simple coincidences and stochastic oddities. But that is rarely the case in situations like this.

  202. steven mosher
    Posted Oct 5, 2007 at 2:25 PM | Permalink

    RE 201.. I’m wondering about a metric for yearly change with memory…

    HA wait.. acceleration. F=MA. What kind of force does it take to turn the good ship
    climate on a dime.

    Late night last night

  203. SteveSadlov
    Posted Oct 5, 2007 at 2:45 PM | Permalink

    Hmmm, my comment got eaten. Let me try that again. Hysteresis …… magnetic recording elements ….. read / write hysterisis in magneto-resistive stripes.

  204. Sam Urbinto
    Posted Oct 5, 2007 at 8:40 PM | Permalink

    Your comment, my homework.

  205. Kenneth Fritsch
    Posted Oct 6, 2007 at 1:46 PM | Permalink

    Re: #162

    I made an error in applying the Vlookup function in Excel and the various combination trends that I reported in Post #162 displayed above in this thread were amended to the values given in the table below. I found that the differences between CRN45 and CRN12 continue to appear significant but not as large for the 1960-1975 and 1960-1985 time periods as was initially reported. What I found with the corrected results was that CRN3 came more into play with trends similar to CRN12 and not as neutral as previously indicated.

    To the previous results I added time period trends for the CRN123 combination and all these combinations of CRN for the time period 1920-1960. Mining the data shows that the trend differences with CRN ratings is relatively flat in the 1920-1960 and 1975-2005 time periods and shows much larger differences in the time period from 1960-1985. Since we have no independent evidence for this effect operating here, one can only suppose that the quality differences exposed recently by Watts and Team had origins back some 20 to 40 years ago. Furthermore, I need to look at statistical significances of the 1960-1975 and 1960-1985 trends before I should really even suppose.

    Using individual CRN categories, with the small sample sizes they currently represent, gives a less clear picture than does combining groups and particularly using the largest groups in comparisons, i.e. CRN123 versus CRN45.

  206. Joe Bowles
    Posted Oct 6, 2007 at 3:07 PM | Permalink

    I wonder if we should constrain the analysis framework based on sample size. We are clearly losing a sizeable portion of the sample by rating as we go back.

  207. steven mosher
    Posted Oct 6, 2007 at 3:54 PM | Permalink

    SteveS.

    I did the Year-YEar chart. Fascinating. I took OpenTemp and ran all sites.

    The year over year changes ( all the two year trends.. hehe) were zero mean.
    And the range exceeded +1C to 1C. That is, the most extreme swings in yearly
    temps, exceeded the century trend. That’s some spikey data.

    The thing I found most interesting was the rapid reversals. Big falls to big rises.

    That’s either bad data or a very touchy process. And with the Ocean being this huge
    high freq filter I just can’t see those kind of swings. AM I being totaly stupid
    here?

  208. Posted Oct 6, 2007 at 4:26 PM | Permalink

    #207 steven mosher:
    The ocean has a lot of thermal inertia and so responds slowly. The atmosphere not so much. My whole issue with yearly comparisons (1934 vs 1998 for example) is that the differences are much smaller than the high frequency component. If we’re talking about *climate* (not weather), then multi-year trends are what matter.

    I am planning to look at the yearly differences between runs relative to the characteristic noise in the signal as defined by the year-year differences.

  209. steven mosher
    Posted Oct 6, 2007 at 4:58 PM | Permalink

    RE 208. Sorry, I know the ocean has huge inertia. I should have looked at golbal land/sea anomaly.
    Still, I was a bit stunned that the year to year mean was nearly zero, with excursions out to
    + – 1C. Sadlov had noted some fault scarps in the data and I was fiddling with a way of
    characterizing that. Plus, we have this recoginition that the series does have a memory
    And I’m wondering if we can characterize the probability of reversal.

    When you have a few years of increasing temps and increasing rate of temp change and you suddenly go
    negative on rate and absolute, you have some explaining to do.

    1. cant blame the network of sensors. Unless you want to claim you are blind.
    2. Its Pre industrial, so cant blame C02.
    3. Can’t blame the sun.

    Calling it the weather is shrugging.

  210. Willis Eschenbach
    Posted Oct 7, 2007 at 1:16 AM | Permalink

    John V, you say:

    Key Trends Revisited

    Kenneth Fritsch said:

    I am having problems understanding the choice of temperature regime period selections. When I use the entire USCHN Urban data set from 1960-2005 I calculate: Trend = 0.039 and R^2 = 0.39. When I do the same calculation for the period 1975-2005 I calculate Trend = 0.057 and R^2 = 0.33.

    When I first checked this I got similar results. This morning I realized that I was using Excel’s RSQ() function with anomalies as the y’s and years as the x’s. That implies that I was calculating the correlation between the year and the measured temperature, not the correlation between a linear fit of the temperature and the measured temperature.

    This morning I redid the comparisons against a linear trend. I used the automatic trendline available in Excel charts. The results are a little different:

    CRN12 TOBS, Single Year
    1960 – 2005: R^2 = 0.274
    1975 – 2005: R^2 = 0.277

    The noise in the signal is drowning the trend. I did the same using a 5yr centred average:

    CRN12 TOBS, Five Year Centred Average
    1960 – 2005: R^2 = 0.635
    1975 – 2005: R^2 = 0.676

    This doesn’t mean a lot in the debate about key trends, but it does at least say that my quick visual inspection of trends was accurate in this case. That is, the correlation with a linear fit is stronger for 1975-2005 than for 1960-2005.

    Note also that Kenneth’s results were generated from all stations with urban adjustments, while mine come from only the rural CRN12 stations without urban adjustments.

    ALERT! ALERT! AUTOCORRELATION DETECTED! AUTOCORRELATION PRESENT! PUT DOWN THE MOUSE, SIR, STEP BACK FROM THE KEYBOARD, AND NO ONE WILL BE HARMED!

    In a more serious vein, you desperately need to determine the significance of the R^2 value of the trends. My guess is that not a single one of them is significant . In addition, you need to calculate the standard error (or the 95% confidence interval) for the trends, to see if they are significantly different either from each other or from zero. In both of these calculations, it is mandatory to take autocorrelation into account. See Equation 22 here for Nychka’s method for accounting for autocorrelation.

    w.

  211. MrPete
    Posted Oct 7, 2007 at 6:03 AM | Permalink

    Re #209

    When you have a few years of increasing temps and increasing rate of temp change and you suddenly go
    negative on rate and absolute, you have some explaining to do.

    Just my sense from 30+ years of handling natural vs man-made data… people are biased in favor of finding a simple formula or trend to justify the idea that what goes up… will keep going up.

    Yet there are more confounding “natural” factors than we want to admit… because they don’t give us the control we like.

    I.e., my gut sense is climate is not as predictable is some imagine, and more self-correcting. And people/instruments are not as reliable/predicatable as we’d like.

    You may be simply seeing some of those self-correcting factors at work. We don’t understand them, so we call them outlier anomalies.

    You may be seeing methodology changes in the data. In 192x, did they change their rounding recommendation?

    You may be seeing “cultural weather bias.” Was there a heat wave, and people tended to read their thermometers higher?

    I agree: any sharp jump, down or up, is interesting.

  212. Kenneth Fritsch
    Posted Oct 7, 2007 at 7:08 AM | Permalink

    Re: #205

    I found a further error in converting -99s from the raw data to NA in that I had 4 zeros in one CRN4 station for the early 1960s. An error such as this can and did significantly change my results reported in the previous post (#205).

    The trend differences by CRN category are primarily confined, with the above revision, to the period 1960-1985 and to the added time period of 1945-1985. Using randomly generated groups from the population of station measurements that CRN45 and CRN123 were drawn from, indicated that the null hypothesis, that CRN45 and CRN123 have the same trends, could not be rejected at the customarily used 5% level (but could at a 10% level).

    I have not checked the period of 1945-1985 for statistical significance of the trend difference. I show a graph of the entire 1920-2005 time period and the anomaly differences for CRN45-CRN123. I used a power 3 polynomial primarily to point out the changing trend differences over that time period.

    Willis E, I know auto correlation will affect the significance of a correlation, but I am not sure what effects auto correlation will have on my looking at trend differences with randomly selected samples and the adjustments in statistical significance, if any, one should make.

    I am here to learn and if one learns by one’s mistakes I should have learned loads of late here.

  213. Posted Oct 7, 2007 at 7:21 AM | Permalink

    #210 Willis E:
    I’m not sure I understand your point. Are you suggesting that there is no statistically significant trend from 1975 to 2005? Or only that the *differences* between trends for different subsets is not statistically significant? I find the former highly unlikely but the latter seems reasonable.

    Anyways…

    I tried to follow your link for Nychka’s method, but it’s not publically viewable. It would be great if you could contribute your expertise to determining the significance of the trends. I have some ideas for estimating the variance due to geographical distribtion, but no ideas for microsite or UHI (other than avoiding UHI by using rural stations and homogenizing station groups by CRN rating).

  214. Posted Oct 7, 2007 at 7:25 AM | Permalink

    Kenneth Fritsch:
    I think your results using random sets could be very useful, but I am still uncertain why you are using USHCN urban-adjusted data. The urban adjustment combines the data from different stations (across CRN ratings), and therefore pollutes any trends that may be present.

  215. Kenneth Fritsch
    Posted Oct 7, 2007 at 7:42 AM | Permalink

    Re: #214

    The adjustmment for UHI has to be very small and I doubt that it will effect the comparisons I am making. There are adjustments other than UHI that take place in the progressive adjustments of the USCHN data between TOBS and Urban, but I have not been able to determine how they specifically would pollute the data as you suggest. What step(s) in the process do this polluting?

  216. Posted Oct 7, 2007 at 7:53 AM | Permalink

    Kenneth:
    I do not understand the USHCNv1 urban adjustment very well either. This is my understanding (correct me if I’m wrong):
    – they correct the urban stations using surrounding rural stations
    – the other corrections (TOBS, MMTS, Filnet, SHAP) are all local to the station being adjusted

    I am not confident that the “adjustmment for UHI has to be very small”.

  217. steven mosher
    Posted Oct 7, 2007 at 8:04 AM | Permalink

    Re 216.

    Karl’s approach is used by USHCN. Giss uses hansen et all

    Urbanization: Its Detection and Effect in the United States Climate Record” by Karl. T.R., et al., 1988, Journal of Climate 1:1099-1123.

    http://ams.allenpress.com/perlserv/?request=get-abstract&doi=10.1175%2F1520-0442(1988)001%3C1099%3AUIDAEI%3E2.0.CO%3B2

    Paper: http://ams.allenpress.com/archive/1520-0442/1/11/pdf/i1520-0442-1-11-1099.pdf

    Peterson also did a study, but it’s not relevant to the USHCN adjustment. The former is.

    You will references to Oke who you should read in the issue of microsite

  218. steven mosher
    Posted Oct 7, 2007 at 8:07 AM | Permalink

    Re 216. Have a look at Karls paper, citing above. We havent reviewed it here primarily because
    GISS were not using USHCN Urban Adjusts.

    Might be nice to get the stationlist from Karl

  219. Kenneth Fritsch
    Posted Oct 7, 2007 at 8:24 AM | Permalink

    John V, in a README Text file from the link referenced below I have excerpted the progressive steps that USHCN performs in adjusting the data. I have also reposted from your #95 post above the graph showing these progressive effects. Over relatively short periods of time I do not see a polluting step from filenet or Urban. They tend, in general, to cancel out each other’s small effects.

    If we cannot see the effects from the CRN categories on the final data set output then either their differences are not important or they are small compared to the noise level in the temperature signal – and might be adding to it. If the CRN categories affected the data at an intermediate level of adjustment and not the final adjusted data, what would that mean?

    ftp://ftp.ncdc.noaa.gov/pub/data/ushcn

    The Areal Edited data is the original (or raw) data that have been screened to flag monthly data that are suspect/outliers (over 3 standard deviations from period of record mean of the element).

    The Time of Observation data is the Areal Edited data that have been adjusted to remove the time of observation bias so that the data will be consistent with a midnight-to-midnight observation schedule.

    The Filnet data is the Time of Observation data that have been adjusted for the Maximum/Minimum Temperature System (MMTS) bias, station moves/changes bias, and contains estimated values for missing/outlier data.

    The Urban Adjusted data is the Filnet data that have been adjusted for bias due to urbanization effects.

  220. steven mosher
    Posted Oct 7, 2007 at 8:34 AM | Permalink

    Re219.

    One problem you’ll see is that Karl did a paired study based on population to determine
    Urban Bias. So, if you compare a RuralCRN5 with an urban CRN12 You have a mixed bag of sorts.
    To be fair Karl didnt have site surveys. He notes this weakness early on in the paper. The
    suggestion is to use large numbers to overcome the lack of understanding about the micro
    climate.

    So, It would be instructive to revisit KARL using Opentemp.

  221. Posted Oct 7, 2007 at 8:50 AM | Permalink

    Kenneth:
    The net adjustment (all stations) of USHCN final (urban?) is about 0.1F over a century. Admittedly, that’s a small effect. What I am not clear about is how the net effect is distributed across different stations.

    I guess what I’m saying is that urban adjustments add another unknown variability to the signal — and there’s plenty in the signal alrady. In the absence of a good argument in favour of including the urban adjustment, it seems better keep it simple by leaving it out.

  222. John Lang
    Posted Oct 7, 2007 at 9:50 AM | Permalink

    Remember the USHCN adjustments are on top of each other.

    TOBS adds 0.35F (from 1920) to the RAW

    MMTS adds 0.03F to TOBS

    SHAP adds 0.25F to MMTS

    FILNET adds 0.1F to SHAP

    and FINAL subtracts 0.1F from FILNET

    All together, the adjustments add 0.63F to the RAW trend (from 1920).

  223. Posted Oct 7, 2007 at 9:52 AM | Permalink

    John Lang:
    I chose to stop at TOBS because it is (relatively) uncontroversial. I wasn’t so sure about MMTS and SHAP. What is your opinion?

  224. Kenneth Fritsch
    Posted Oct 7, 2007 at 2:34 PM | Permalink

    Re: Post #212

    I am finally filtering out all my previous errors and I must now correct the trends for CRN45 and CRN123 for the time period 1945-1985 reported in the table (bottom) in Post #212 were actually for the period 1920-2005. The 1945-1985 trend results were calculated as listed below:

    CRN123 = -0.0152; CRN45 = -0.0003

    Using randomly selected groups in 60 simulations of the above comparison indicated that the null hypothesis (that CRN123 and CRN45 have the same trends) could be rejected well under the customarily used 5% level.

    I will double check this work for errors and report adjustments for geographical effects — if they are significant.

  225. SteveSadlov
    Posted Oct 8, 2007 at 3:36 PM | Permalink

    I’ve always been a “throw a rock at the hornets’ nest” kind of guy. So, I simply could not resist posting this on the thread over at RC regarding Svensmark et al, currently receiving a number of posts from Tamino. Here’s what’s pending on that thread. (In the background, sounds of angry hornets … Bzzzzzzzzzzzz!):

    —–
    # SteveSadlov Says: Your comment is awaiting moderation.
    8 October 2007 at 4:43 PM

    RE: #73 – Without certain assumptions regarding what TOBS adjustments for periods prior to 1950 ought to be, and without a strangely depressed general mean temperature value for the period after 1880 and prior to 1920, the entire 20th century showed very little warming. This latter was long attributed to Krakatoa, however, in reality, the most significant depression from that only lasted for about two years. Don’t believe me? Drill down on GISS. Something is rotten in the early 20th century, especially, in the totally artificial looking upward “fault scarp” in the 5 year smoothed GISS in between 1917 and 1921.

  226. Joe Bowles
    Posted Oct 8, 2007 at 4:26 PM | Permalink

    John V. says:

    October 7th, 2007 at 9:52 am
    John Lang:
    I chose to stop at TOBS because it is (relatively) uncontroversial. I wasn’t so sure about MMTS and SHAP. What is your opinion?

    At this point I am skeptical about all adjustments. The magnitude of each adjustment may fall in a reasonable range, but thee is no way to tell. I have no clue what the range of such adjustments are…and have the same problem with all of the adjustments being made.

    I think at some point we need to compile the range of adjustments, though how to do this escapes me. I still think that the adjustments will tend to be at the upper end of the reasonable range. If they would release this information, the actual adjustments would be a lot easier to accept.

  227. steven mosher
    Posted Oct 8, 2007 at 4:37 PM | Permalink

    RE 224.. Use IPPC categories of confidence… sorry.

  228. steven mosher
    Posted Oct 8, 2007 at 4:39 PM | Permalink

    Re 223.

    I think The MMTS adjustment is rather small and can also be double checked. SHAP
    is the one I want to see the details on.

  229. John Lang
    Posted Oct 8, 2007 at 5:16 PM | Permalink

    To John V. 223 – There have been a few post-audits done on the TOBS adjustment which has indicated the adjustment process is robust. So, I suppose we should just accept it. But I am always sceptical of these convenient adjustments which always add to the overall trend in temperatures and TOBs is the biggest overall adjustment in the trend.

    That is why I have always posted that you should present both the TOBS adjusted data trend and the RAW trend. The purpose of the CRN 1-5 labeling is really to see if there is a urban heat island effect (or some other bias/effect) in the poorly sited CRN 4, 5 sites. I think that using the TOBS series by itself should show this effect but the RAW data could also show whether there is a systematic bias in the TOBS adjustment as well.

    My only other comment to you is that your 5-year averaging has masked some important differences in the CRN 1-5 categorization and even the base period average chosen (1980-2000 versus 1961-1980) results in differing results. Show all your results and, in the process, more information is available on which to base conclusions, lack of conclusions.

  230. steven mosher
    Posted Oct 8, 2007 at 6:19 PM | Permalink

    I think the best approach is to Decouple the data issue. What do I mean?

    JohnV initially used GHCNv2. result? he was attacked in a bogus fashion.
    Ideally, Opentemp would take a variety of inputs and compute a result.
    Our focus ( indulge me) should be a creation of a package that allows
    anyone to load the data set they want, process it, and publish results.
    Then THEY are the lighting rod of data selection or ‘regime’ selection.

    I still use GHCNv2. I’ll make that clear in My presentations. In the end
    I do not think the CRN issue will TURN on data source selection. I reserve
    the right to be utterly wrong.

  231. SteveSadlov
    Posted Oct 8, 2007 at 7:15 PM | Permalink

    RE: #230 – It will turn on how accurate, in reality, are the data prior to 1950. Assuming of course there is any way to assess that.

  232. SteveSadlov
    Posted Oct 8, 2007 at 7:33 PM | Permalink

    http://wattsupwiththat.wordpress.com/2007/10/07/california-climate-pdo-lod-and-sunspot-departure/#more-308

    Look at accumulated departure from average, for temperature, for California. Look at 1900 – 1920.

  233. Posted Oct 8, 2007 at 9:25 PM | Permalink

    #229 John Lang:
    I think the 5-year average is important to show climate trends due to the very high yearly variation. That being said, I will also start including the yearly data. I hope Steve McIntyre will also start showing both. This is Climate Audit, so we should be looking at climate change not yearly variation in weather.

    I am planning to have a look at the high frequency component of the yearly temperatures changes. The yearly difference between result sets can not be properly interpreted without first characterizing the yearly temperature changes. There is roughly a century of reasonably good data and I plan to characterize it at three scales: century, decade, year. Differences between result sets can then be compared to the characteristic variability in the signal at the appropriate scale.

    I have to repeat that the choice of reference period only moves the trends up and down. It has no effect on the trends. While CRU and GISTEMP uses 1961-1990 and 1951-1980 respectively, I think the best reference period is the last 10 years of available data. They are presumably the most accurate years, and by shifting the results together in recent years the differences in early years are easier to see.

  234. Geoff Sherrington
    Posted Oct 8, 2007 at 9:38 PM | Permalink

    Re # 231 Steve

    Polite correction, if I may – it’s the PRECISION of data prior to 1950 that is more important than the ACCURACY if one is looking at trends. If a thermometer has a contant accuracy error of 1 degree, it will still show the trend provided it was read precisely (and we have to make that asumption in the absence of alternatives).

    I can get RAW T max and T min and some metadata for 1,700 Australian stations going back at times to the 1860s, with lat/long/alt. It is daily data. It is too big for my PC or my brain. Some of the stations have very little data, others have a lot.

    Australia is interesting because many sites started early (gold rushes of 1870s etc) have remained isolated and rural until today. Thus, no UHI adjustment is indicated. They are candidates for comparison with urban.

    My theoretical approach would be to divide all data into 5-year calendar periods. I would calculate the slope of best fit for each site for each 5 year period for both max and min. Then, for each 5-year period I would link the slope figure to create a continuous graph up to mid 2007. Within that graph there would be occasional changes to instruments, housing, TOB, etc but my gut feeling is that single events of these would even out in the weight of many observations.

    Then, having visited most of the 1700 sites in my career, I would rank them as rural or early urban or late urban and do a 3-way split and repeat the exercise. I base this proposal on a lot of reading of various approaches tried elsewhere and the cans of worms they have generated.

    I need help. Anyone interested?

  235. Posted Oct 8, 2007 at 10:12 PM | Permalink

    #234 Geoff Sherrington:
    I also like the idea of working with slopes but I am planning to approach it a different way. Maybe we should compare notes:

    – Take monthly differences starting with the most recent value and working backwards
    – Any months with station metadata indicating instrument, TOBS, or location changes are omitted (removes the need for TOBS-, MMTS-, and SHAP-type adjustments)
    – (optional) One or more buffer months are omitted to account for errors in metadata dates
    – For every month, calculate the average temperate change for available stations

    (I do not plan to switch OpenTemp to trends in the near future, but if I was that would be my plan).

    As for helping,I can write an OpenTemp input filter for your data if it’s in a consistent format.

  236. Geoff Sherrington
    Posted Oct 9, 2007 at 12:00 AM | Permalink

    Re # 235 John V

    Thank you for your reply. I have a completely open mind on the sharing of data and I can be contacted at sherro1@optusnet.com.au

    If you wish to swap notes on method, I no longer have a good computer and computing dept to help me. The data of which I speak come from the Bureau of Meteorology in Australia. I shall ask if they are still on open record and how much a DVD would cost.

    It has been painful to see the reverse engineering and skilful reconstruction of the USA data reported on these pages. I feel that there is a good chance that treatment of the Australian data will have learned from this USA pilot run. My preference for gradients arises from reading the USA efforts and from a career in geochemistry, where the data were more spatial than temporal, though replicate analysis for quality control was temporal.

    I do not wish to denigrate the work of Australia’s BoM with the following comment, but they seem to be heading towards an ever smaller public data set while claiming that is it higher quality. This seems to have stressed recent data and displaced old data. It also places stress on adjustments between comparison sites when these sites are quite distant. Geoff.

  237. SteveSadlov
    Posted Oct 9, 2007 at 4:07 PM | Permalink

    RE: #225 – RC censored it. Too difficult of a topic to be handled on a forum that consists of various pied pipers and their True Believer rodents.

  238. Kenneth Fritsch
    Posted Oct 9, 2007 at 5:16 PM | Permalink

    I have not found any errors with my previous 1945-1985 analysis indicating that the trend differences between CRN45 and CRN123 are statistically significant. I have included below two graphs showing the CRN45-CRN123 anomaly differences for 1920-2005 using a 5 year and 10 year moving average. What I have found is that the difference in trends between these CRN groups occurred most visually and in most sustainable form in this time period.

    I would like to make the same comparison using the John V TOBS data set. I have not looked at your site lately John V, but are all of the CRN categories, 1 through 5, available in Excel spreadsheets from your site through the TOBS Adjustment stage? I would not expect them to yield different results but think we need to do the comparison.

  239. Kenneth Fritsch
    Posted Oct 10, 2007 at 11:04 AM | Permalink

    Re: #238

    John V, please ignore my request for TOBS data for CRN1 through5. I have downloaded the USCHN MMS data set that will serve my purposes for comparisons. Actually getting the information into the correct form has become a much simpler process than I once anticipated.

  240. Posted Oct 10, 2007 at 11:10 AM | Permalink

    Kenneth Fritsch:
    Your results for urban adjusted CRN45 vs CRN123 are similar to my results for rural CRN12 vs all CRN5:
    http://www.climateaudit.org/?p=2069#comment-140922

    I have not yet done a difference plot (CRN5R minus CRN12R) but will do so soon. I’m waiting on Steve McIntyre to post his CRN12 gridded results as an independent check on my method.

  241. Joe Bowles
    Posted Oct 10, 2007 at 12:37 PM | Permalink

    I find the selection of five year smoothing interesting. As far as I can tell, there is no particular reason to use five years versus three or seven other than convenience. If anything, it might be interesting to look at the use of 11 year smoothing because that,at least, is justified by the sun cycle.

    My only objection to using mean smoothing is that it obscures the data. It might show more if we used median smoothing, since that preserves the data.

  242. Posted Oct 10, 2007 at 1:27 PM | Permalink

    Joe Bowles:
    I chose 5-year smoothing because that seemed typical from HadCRU and GISTEMP. That’s not a justification — I just chose to go with what was common.

    Out of curiousity I had a look at the other smoothing methods you suggested. In all I looked at 3, 5, 7, 9, and 11 year centred periods using mean and median. Since 1934 vs 1998 keeps coming up, I looked at the difference between the warmest year in the 1930s (c1934) and the warmest year between 1993 and 2003 (c1998):

    c1998 minus c1934 (GISTEMP, pre-SHAP, Sept 12/07):
    1yr: -0.02C
    3yr Mean: +0.24C
    5yr Mean: +0.16C
    7yr Mean: +0.16C
    9yr Mean: +0.16C
    11yr Mean: +0.20C
    3yr Median: +0.08C
    5yr Median: +0.08C
    7yr Median: -0.15C
    9yr Median: -0.07C
    11yr Median: 0.00C

    Interestingly, c1998 is warmer than c1934 for all of the mean periods (particularly for 3yr mean and 11yr mean). For the medians, c1998 is warmer for two smoothing periods and c1934 is warmer for two smoothing periods.

    For completeness I have plotted the 3yr, 7yr, and 11yr averages (mean and median):

    Note: The 5yr and 9yr averages were removed to make the plots easier to read.

  243. Posted Oct 10, 2007 at 2:04 PM | Permalink

    Of course, all of the results above are for the USA loer 48 only. The shape of the temperature trend for the world has a much less prominent peak in the 1930s/40s.

  244. Joe Bowles
    Posted Oct 10, 2007 at 3:24 PM | Permalink

    John V.

    I was surprised to find the same thing when I was playing with the data. I was playing with a couple different smooths. I started out by smoothing the months in the CRN 12 data using a three month median smooth of average temperature. Then, I found the min and max for each year based.

    I then tried using 11 yr.smooths separately on the average low by year and the average high.
    I ran regressions on them. The R^2 were low, but the results were significant at the .01 level.
    On the median low, 11 year smooth, the R^2 was about 16.8% p

  245. Posted Oct 10, 2007 at 4:03 PM | Permalink

    Joe Bowles:
    I didn’t understand a couple of items in your comment:
    – Which CRN12 results did you use?
    – What did you run regressions on?

  246. Joe Bowles
    Posted Oct 10, 2007 at 4:24 PM | Permalink

    John V

    Somehow my last one (244) got cut off.

    Lo Median, 11 year smoothing.
    SUMMARY OUTPUT

    Regression Statistics
    Multiple R 0.410326803
    R Square 0.168368085
    Adjusted R Square 0.161073069
    Standard Error 0.448379553
    Observations 116

    ANOVA
    df SS MS F Significance F
    Regression 1 4.640075805 4.640075805 23.07987632 4.77101E-06
    Residual 114 22.91904144 0.201044223
    Total 115 27.55911724

    Coefficients Standard Error t Stat P-value
    Intercept -11.25938569 2.421626879 -4.649513014 9.01295E-06
    X Variable 1 0.0060 0.00124327 4.804151988 4.77101E-06

    The one for high median average was:
    Regression Statistics
    Multiple R 0.378019304
    R Square 14.3%
    Adjusted R Square 0.135380161
    Standard Error 0.29196729
    Observations 116

    ANOVA
    df SS MS F Significance F
    Regression 1 1.620201384 1.620201384 19.00643218 2.86892E-05
    Residual 114 9.717918443 0.085244899
    Total 115 11.33811983

    Coefficients Standard Error t Stat P-value
    Intercept 15.12325011 1.576869048 9.590682323 2.50311E-16
    X Variable 1 0.0035 0.000809569 4.359636702 2.86892E-05

    I did one graph subtracting the median for the entire series from the scores and was surprised that the low median jumped significantly in 1998 and seemed to be flat at about 1.4 degrees C; while the median high jumped some, but just kept edging up after 1998, ending about .55 degrees C.

    I think the use of medians is probably giving an unbiased indication. It looks like the data supports the idea that we have seen something on the order of 1 degree C since 1890.

    I would post the graph, but don’t seem to be able to get it to copy.

  247. Joe Bowles
    Posted Oct 10, 2007 at 4:50 PM | Permalink

    John V.

    The file I used was CRN12_GHCNv2_Gistemp.

    I used a triplet median smoothing on the average monthly temperature, then, found the min and max of the smoothed medians for each year. I then applied median smoothing over 11 years for all years to the annual high and low developed as above, but excluded the stub in 2006. The Y variables were the 11 year smoothed medians for the high and low; the X variable was the year.

    I was just playing with the data to get a feel for it. I wasn’t surprised that the R^2 was low, but I was a bit surprised that it was statistically significant at the 0.001 level.

    I plan to play with the CNR5 data in the same way as time is available.

    My main goal was to see what the indications would be using medians rather than means in terms of seasonal effects. I thought that using the medians might get some of the noise out of the variables. I am still thinking about whether that is true or not.

    I started with the CRN12 data because it was the most reliable based on classification. At this point, my understanding of statistics is a lot greater than my technical capabilities with statistics…and my understanding is probably a good bit out of date.

    I love the stuff you are doing. I do appreciate your consideration of the 11 year cycle. If there is something in the data related to the solar cycle, it is logical that the correlation would be stronger with the 11 year smoothing.

    Interesting. That is true in terms of the low with 5 year smoothing, but the median hi is very close whether you use 5 or 11 years for the smoothing.

    Bear in mind, the stuff I was looking at was your table of average temperature data, not the high and low. I was interested in seeing whether the median low for a year had a different pattern than the average high. I figured the low average temps tend to occur in winter, and the high average temps in summer. I wondered whether the effects were seasonal.

    I am still thinking about it. I am not sure whether it means anything or not.

  248. Kenneth Fritsch
    Posted Oct 10, 2007 at 5:47 PM | Permalink

    Re: #240

    Kenneth Fritsch:
    Your results for urban adjusted CRN45 vs CRN123 are similar to my results for rural CRN12 vs all CRN5:
    http://www.climateaudit.org/?p=2069#comment-140922

    John V, I see a significant trend difference between CRN123 and CRN45 in the 1945-1985 time period, but I do not see this in your data analyses. What am I missing here?

  249. Posted Oct 10, 2007 at 6:02 PM | Permalink

    Kenneth Fritsch:
    Quoting myself from near the bottom of the comment linked above: “CRN5 shows substantially less cooling than CRN12R and GISTEMP from 1935 to 1975 (approximately 0.06degC per decade). This *could* be the time period when the majority of the CRN5 micro-site problems were introduced.”

    My bar graph of temperature trends for the key periods I used (the post after the one linked above) shows relative warming of CRN5 compared to CRN12R only from 1935 to 1975.

  250. Posted Oct 10, 2007 at 6:10 PM | Permalink

    I should’ve posted a link to my more recent analysis using TOBS instead of the older analysis using raw data:
    http://www.climateaudit.org/?p=2124#comment-144287

    The results are essentially the same: significant CRN5 warming relative to rural CRN12 from 1935 to 1975. The TOBS results show some CRN5 warming (relative to rural CRN12) from 1975 to 2005 and CRN5 cooling (relatie to rural CRN12) from 1915 to 1935.

    I hope to quantify the error bars due to the low number of stations in the next couple of days.

  251. John Lang
    Posted Oct 10, 2007 at 6:39 PM | Permalink

    John V., for further completeness can you plot the CRN 1,2 data: the CRN 4,5 data: versus the GISTEMP on an annual basis (as you have done with just the GISTEMP data in #242 above)?

  252. Posted Oct 10, 2007 at 7:07 PM | Permalink

    #251 John Lang:
    I can try to get to that. My plan is to be thorough and learn as much as possible from rural CRN12 (the best stations) before moving on to other station subsets.

  253. Clayton B.
    Posted Oct 10, 2007 at 7:34 PM | Permalink

    Hey JohnV,

    Have you looked at the effect of using smaller/larger gridding? Or are you going to wait until you implement the revised gridding technique?

  254. Clayton B.
    Posted Oct 10, 2007 at 7:58 PM | Permalink

    Here’s a chart of coastal stations (within 10km based on GHCN station data) vs. non-coastal stations run through my version of opentemp – absolute temperatures shown. I’m not quite certain which version of opentemp I’m using and I’m not quite sure which version of data I’m using.

    I also understand that opentemp’s annual average temperature calculation does not really make sense when looking at coastal versus non-coastal – but I still think there may be something to it. JohnV, can a new region be easily defined in the code to outline the inner contiguous states and leave out stations near the coast?

    I’m sure this has already been looked at…

  255. SteveSadlov
    Posted Oct 10, 2007 at 8:16 PM | Permalink

    Coastal generally means more people per square mile at present (whether urban or rural) than in the interior, and, a bigger absolute population delta between 1850 and present. Especially true for the Pacific Coast and the Southern Atlantic / Gulf Coast. In 1850 California had so few people that Stockton was the #3 city in the state. Today, it could be a nation state in its own right.

  256. Posted Oct 10, 2007 at 8:24 PM | Permalink

    Clayton B:
    OpenTemp is currently hard-coded for the USA lower 48. For a comparison like this where geographic bias is the whole intent you would be better off using the station.csv file and averaging the stations externally (vs using the geographic weighting).

    The OpenTemp analysis code has not changed from the first version I released. Newer versions have just added some file parsing options and fixed the problem of writing averages for incomplete years.

    As SteveSadlov pointed out, it can be difficult to draw any conclusions from these types of subsets. I continue to believe that we should be cross-cutting with more precision. For example, if the goal is to compare coastal to interior, then extract the rural CRN12 stations from the coast and interior. If there are not enough stations then the study can not be done. Without controlling for UHI and micro-site issues, thereare too many unknowns that could contaminate the results.

  257. Clayton B.
    Posted Oct 11, 2007 at 8:14 AM | Permalink

    #256,

    I guess my question was: Can I just modify the _MeshUSALower48 perimeter points to create an inner region? Will the code still use the coastal stations that are within 1000km in its cell averaging if they are outside of the new perimeter? Why would this not be the way to investigate this?

    I’ll probably wait until the new surfacestations list is released before digging into this anyways.

  258. Michael Jankowski
    Posted Oct 11, 2007 at 9:10 AM | Permalink

    Re#254 and #255,
    http://www.oceanservice.noaa.gov/programs/mb/supp_cstl_population.html has links to “Population Trends Along the Coastal United States: 1980-2008.”

    This spreadsheet goes back to 1960 http://www.census.gov/compendia/statab/tables/07s0029.xls – maintains the same percentage of coastal vs non-coastal. Of course, it does consider the Great Lakes as coastal, and I’m not sure if that is apples-to-oranges with your chart. And the Great Lakes population as a percentage of the “coastal” has dropped and shifted to the Pacific.

  259. Kenneth Fritsch
    Posted Oct 11, 2007 at 10:00 AM | Permalink

    Re: #249

    Thanks for pointing me to the bar graph of 1935-1975. I agree that that trend agrees with what I found 1945-1985.

    I now have a couple of questions for you or any other poster who might enlighten me:

    I downloaded the USCHN MMS data set and immediately realized that it has many missing data points and particularly with a direct comparison with the data set I have been using, i.e. USCHN Urban. I did some limited work with your derived “USCHN” Raw data set a while back, John V, and noticed that it too had considerable missing data points compared to the Urban set.

    My first question is: How did the missing data points get entered into the final version of the USCHN data set, Urban, from an earlier set, MMS, in the adjustment sequence used in the USCHN data set series?

    My second question is: How did you compensate in your calculations when using a data set with many missing points?

    The Urban set has very few missing data points and I did not find that a problem when calculating trends and trend differences. I am in the process of a station by station comparison using USCHN Urban, USCHN MMS and the one that you used originally which I believe has been referred to as USCHN Raw.

  260. Posted Oct 11, 2007 at 10:14 AM | Permalink

    Kenneth Fritsch:
    The missing data points are estimated by USHCN as part of the Filnet processing stage (the stages are areal, TOBS, SHAP, Filnet, and urban). From the readme.txt at the USHCN FTP site (emphasis added):

    “The Filnet data is the Time of Observation data that have been adjusted for the
    Maximum/Minimum Temperature System (MMTS) bias, station moves/changes bias, and
    contains estimated values for missing/outlier data.”

    OpenTemp calculates the average temperature every month using only the data available for that month. The geographic weighting is calculated independently every month.

    My original analyses were done using GHCNv2 raw, only because I was in a hurry and the file was easy to parse. Since the TOBS adjustment is well accepted I would suggest using TOBS data instead of raw data.

    There is also clean MMTS data (areal + TOBS + MMTS, no Filnet) available in the “OtherIntermediates” directory at USHCN (I don’t have the full URL with me right now).

  261. SteveSadlov
    Posted Oct 11, 2007 at 1:21 PM | Permalink

    RE: #258 – Go back further.

  262. Kenneth Fritsch
    Posted Oct 11, 2007 at 1:34 PM | Permalink

    Re: #260

    Some of the differences that I have observed and calculated by comparing the USCHN Urban and MMTS data sets using the CRN12 stations for the period of 1920-2000 are the following:

    The missing points for the Urban set are primarily in the earlier years, i.e. only 1% missing data in the period 1945-2005, while the time period 1920-1945 has approximately 10% missing data points. In the case of the MMTS data set the missing data points come more from the later years. It would appear to me that in performing the progressive adjustment from MMTS to Urban the process excluded some early data points without attempting to fill them in (with the averaging algorithm) and then with the algorithm filling nearly all the latter year missing points. This difference in handling by time period would indicate that there were some earlier missing data points that could not be “averaged in”.

    The absolute differences between the Urban and MMTS data sets averaged approximately 0.6 degrees F per station per year. These differences were sufficiently consistent over the years for most stations that the URBAN-MMTS trend was not all that large (approximately 0.0013 degrees F per year over the 1920-2000 time period). A few stations in the CRN12 group did show large trend differences between using the Urban and MMTS data sets.

    I have thought about the use of monthly data in attempts to overcome the missing data problem, but I have not been able to convince myself that it truly solves the entire problem. Could you convince me with some details on how you did it?

  263. JerryB
    Posted Oct 11, 2007 at 5:32 PM | Permalink

    An update of USHCN data up to May 2007 has been posted at
    ftp://ftp.ncdc.noaa.gov/pub/data/ushcn/
    and the pertinent additions may work their way through to GHCN,
    and thereby to GISS, in perhaps a week or so.

  264. Posted Oct 11, 2007 at 8:49 PM | Permalink

    Thanks for the heads-up Jerry B. I downloaded the new data tonight and ran CRN12R and CRN123R analyses. For those who like the top 10 list, 2006 for the USA lower 48 is not as warm as 1934 or 1998.

    I have a bunch of graphs for CRN12R and CRN123R vs GISTEMP, but I don’t have time to post them tonight. There is very good agreement between GISTEMP and both CRN12R and CRN123R. Standard deviation of the 1-year differences is 0.13C and 0.10C respectively with near zero trend (~0.03C/century).

    Will try to post the graphs tomorrow.

  265. Steve McIntyre
    Posted Oct 11, 2007 at 10:33 PM | Permalink

    They’ve deleted SHAP from the Other Intermediates. And where is USHCN v2? I suspect that they realize that it will be looked at this time.

  266. JerryB
    Posted Oct 12, 2007 at 6:46 AM | Permalink

    I would not render any particular interpretation of the reasons for USHCN V2 not
    being published yet. Five years ago, an NCDC person who was working on it,
    expressed the hope, if not the opinion, that it would be ready within a few
    months from then.

  267. Cliff Huston
    Posted Oct 12, 2007 at 7:11 AM | Permalink

    Re: 266

    Five years ago, an NCDC person who was working on it, expressed the hope, if not the opinion, that it would be ready within a few months from then.

    Ah, USHCN v2 is the Vista version then?

    Cliff

  268. steven mosher
    Posted Oct 12, 2007 at 7:27 AM | Permalink

    RE 264. I don’t think you can compare the GiSS ananomaly to your anaomaly for absolute difference.

    Only trend. The GISS anamaly is calculated from the mean of ITS data. Say 1950-1981 had a mean of
    11C. So, the anomaly = Actual temp that year – 11C. say 2006 has a 11.6C temp. Then it has
    a .6C anomaly.

    IF CRN12R has a 1950-1981 mean of 10.C and a 2006 temp of 11.6C you have 1.6C anomaly from 10C
    but a .6C anomaly from 11C

    Subtracting the mean shifts in Y, doesnt change the trend, BUT to compare anomalies in an
    absolute sense they must have the same figure for the base period

  269. Posted Oct 12, 2007 at 8:01 AM | Permalink

    steven mosher:
    You are right that I can *not* compare my absolute temperature results to GISTEMP absolute temperatures (and not only because GISTEMP absolute temps are unknown). I’m still confused about why I would want to make that comparison. Absolute temperatures are *very* sensitive to station locations and we are investigating global warming, which implies a trend.

  270. steven mosher
    Posted Oct 12, 2007 at 8:04 AM | Permalink

    JohnV sorry, I thought you were making that comparison. My fault.

  271. Posted Oct 12, 2007 at 8:15 AM | Permalink

    Steve McIntyre:
    I noticed the SHAP and MMTS intermediates were missing too. The dailies have also not been updated yet. I’m hoping they’re all in-progress.

  272. JerryB
    Posted Oct 12, 2007 at 9:11 AM | Permalink

    The SHAP (and MMTS) files have usually not been updated there, as indicated
    by the fact that the most recent set there was vintage 2000. That someone
    deleted the old files, but not the directory, may bode well.

    As for “recent” USHCN dailies, see ftp://cdiac.esd.ornl.gov/pub/ndp070/

    And for relatively recent daily max/min data, see ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/daily/
    but if you try to browse the \all\ directory, your browser may fall asleep;
    there are over 32000 links in that directory.

  273. Posted Oct 12, 2007 at 10:26 AM | Permalink

    CRN123R vs CRN12R
    With the release of the updated USHCN data I decided to have a look at the rural stations with a site quality ratings of 1, 2, or 3 (CRN123R).

    My previous work was done using only the 17 rural stations with site quality ratings of 1 or 2 that were approved by Kristen Byrnes (CRN12R). Due to the small number of stations, there were some legitimate questions about the uncertainties in the analysis.

    For the CRN123R stations, I looked up all of the stations from Anthony Watts’ spreadsheet that met the following criteria:
    – Rural
    – Non-Airport
    – CRN=1, 2, or 3

    It is likely that some of these stations can no longer be considered rural because the population has increased since the rural designation was given. However, until we have a list with current rural designations this was the best I could do.

    The lists of stations that I used are available here:
    http://www.opentemp.org/_results/20071011_CRN123R/stations_crn12r.txt
    http://www.opentemp.org/_results/20071011_CRN123R/stations_crn123r.txt

    The distribution of stations is shown in the maps below. The CRN123R stations are distributed more evenly and fully across the USA lower 48 than the CRN12R stations.

    And now for the results…

    The graph below compares the CRN123R and CRN12R yearly results. All graphs are normalized to the 1996-2005 reference period so that differences are easier to spot in the early years. The bars indicate the yearly differences between the results. A linear trend on the differences is also shown.

    The summary statistics for the differences (CRN123R minus CRN12R) are:
    – Mean = -0.01C
    – StDev = 0.14C
    – Max = +0.46C
    – Min = -0.43C
    – Kurtosis = 1.63
    – Skew = -0.06
    – R^2 = 0.923 (CRN123R anomalies vs CRN12R anomalies)

    It is very difficult to draw any conclusions from the yearly graphs due to the large temperature changes from year to year. The next graph shows the 5-year centred moving average for CRN123R and CRN12R. The summary statistics for the differences (CRN123R minus CRN12R) are:
    – Mean = -0.01C
    – StDev = 0.08C
    – Max = +0.19C
    – Min = -0.25C
    – Kurtosis = 1.10
    – Skew = -0.51
    – R^2 = 0.920 (CRN123R anomalies vs CRN12R anomalies)

    Conclusions:
    The CRN123R and CRN12R results are qualitatively and quantitatively very similar. Due to the larger number and improved distribution of stations, I plan to start using CRN123R as the basis for future comparisons.

  274. Posted Oct 12, 2007 at 10:28 AM | Permalink

    CRN123R vs GISTEMP
    Above, I compared the rural, non-airport CRN123 stations (CRN123R) to the rural, non-airport CRN12 stations (CRN12R). They were shown to be very similar. Due to the larger number and better distribution of stations, I decided to start using CRN123R as the basis for future comparison results.

    Here I will compare CRN123R to GISTEMP (Sept 12, 2007).

    The list of CRN123R stations is available here:
    http://www.opentemp.org/_results/20071011_CRN123R/stations_crn123r.txt

    The distribution of stations is shown in the map below.

    The first graph compares the CRN123R and GISTEMP yearly results. All graphs are normalized to the 1996-2005 reference period so that differences are easier to spot in the early years. The bars indicate the yearly differences between the results. A linear trend on the differences is also shown.

    The summary statistics for the differences (CRN123R minus GISTEMP) are:
    – Mean = -0.02C
    – StDev = 0.10C
    – Max = +0.23C
    – Min = -0.32C
    – Kurtosis = 0.11
    – Skew = -0.14
    – R^2 = 0.956 (CRN123R anomalies vs GISTEMP anomalies)

    The second graph is the same as the first, but using 5-year centred moving averages so that trends are easier to identify:
    – Mean = -0.01C
    – StDev = 0.05C
    – Max = +0.13C
    – Min = -0.12C
    – Kurtosis = 0.74
    – Skew = -0.56
    – R^2 = 0.972 (CRN123R anomalies vs GISTEMP anomalies)

    The third graph shows the trailing 22-year trend in deg C per decade. I previously used an 20-year trend but chose to increase it to 22-years to make it a multiple of the sun cycle. (The trend is now over 2 complete sun cycles, or 1 cycle if you consider the polarity). The summary statistics on the differences are:
    – Mean = -0.003 C/decade
    – StDev = 0.03 C/decade
    – Max = +0.05C
    – Min = -0.07C
    – Kurtosis = -0.06
    – Skew = -0.42
    – R^2 = 0.976 (CRN123R trends vs GISTEMP trends)

    Conclusions:
    For the USA lower 48, there is excellent agreement between GISTEMP and my results using only the best stations (rural stations with CRN=1, 2, or 3). The increased number and improved distribution of stations in CRN123R vs CRN12R increases my confidence in the agreement.

  275. Posted Oct 12, 2007 at 10:31 AM | Permalink

    The input data, scripts, and summary spreadsheet for my posts above are now available:
    http://www.opentemp.org/_results/_20071011_CRN123R.zip

  276. Joe Bowles
    Posted Oct 12, 2007 at 2:58 PM | Permalink

    Re 274

    It is interesting that the last PDO switch was in 1976 and that starts the biggest increase in temperatures. It is supposed to switch back to its cold phase in 2007-2008. If temperatures react as they seem to in the past, we may see a lot of cooling. From what I gather, it seems to have about a 30 year duration in each phase and that is about what the graph shows.

  277. Joe Bowles
    Posted Oct 12, 2007 at 3:10 PM | Permalink

    John V

    Re 242. I was taking another look at the effect of using the median vs the mean in the graphs you kindly ran. I think the median does get rid of some of the noise. I found it particularly interesting that the 11-year smoothing suggests something of a step function from about 1980-1982. I was amazed at how dramatic it was. Again, it seems to trigger off the 1976 PDO switch and seems to be relatively consistent with solar cycle 23.

  278. SteveSadlov
    Posted Oct 12, 2007 at 3:43 PM | Permalink

    On various of the past couple day’s plots, the early 20th century fault scarp. Looking as unnatural as ever.

  279. Posted Oct 12, 2007 at 10:41 PM | Permalink

    John V, in # 273 and 274, what level of adjustment of the CRN123R data are you using? Is it Areal, TOB, MMTS, SHAP, FILNET, or Final Urban? Somewhere along the line, apparently in FILNET, the station data are “homogenized”, ie cross-fertilized with data from adjacent stations. Could CRN123R then become contaminated with some CRN45 data?

    Delaware OH, for instance, has not had a daily reading since 1/30/01, but this does not stop CDIAC from publishing annual Urban-adjusted averages for it for 2001 – 2005. These phantom readings may have been “homogenized” into existence using Urbana OH data, for all we know. Even before 2001, the Final Urban series has a very different trend than the relatively unadusted (Areal?) series I got from UIUC MTAC, despite the lack of a Tobs change in the Delaware record. See graphs at http://gallery.surfacestations.org/main.php?g2_itemId=5278.

  280. Posted Oct 12, 2007 at 10:47 PM | Permalink

    #279 Hu McCulloch:
    I used the USHCNv1 TOBS-adjusted data. I have been avoiding Filnet and Urban precisely because of the cross-contamination issues. I have considered using MMTS or SHAP but they are not currently available.

    I wish there was a way to edit my posts.

  281. steven mosher
    Posted Oct 13, 2007 at 6:52 AM | Permalink

    RE 274. JohnV Are you comparing a CRN123R anomaly to GISSTEMP anomaly?

  282. Posted Oct 13, 2007 at 8:49 AM | Permalink

    #281 steven mosher:
    That’s right — I’m comparing the anomalies relative to a 1996-2005 reference period.

  283. steven mosher
    Posted Oct 13, 2007 at 9:48 AM | Permalink

    RE 282. What are the absolute temp of GISS during that period? Anomaly are used for Trend
    then cannot be used for “difference” charts like you did, as we agreed.

  284. steven mosher
    Posted Oct 13, 2007 at 9:48 AM | Permalink

    RE 282. What are the absolute temp of GISS during that period? Anomaly are used for Trend
    then cannot be used for “difference” charts like you did, as we agreed.

  285. steven mosher
    Posted Oct 13, 2007 at 10:10 AM | Permalink

    RE JohnV see 269.

    Differencing anonmalies is misleading when the anomalies have different means during the period.

  286. Posted Oct 13, 2007 at 10:44 AM | Permalink

    steven mosher:
    We agreed that we can not compare *absolute* temperatures. I am comparing temperature *anomalies*. It’s only misleading if the anomalies are represented as absolute temperatures.

    Think of the anomaly plots as the integral of single-year trends.

    The comparison I am makng is exactly the same as the comparison Steve McIntyre made when comparing his gridded CRN12 to GISTEMP. (There is one small difference in that I am now using a 1996-2005 reference period to make accumulated differences easier to see).

  287. steven mosher
    Posted Oct 13, 2007 at 11:34 AM | Permalink

    RE 286.

    yes, but you know it’s misleading. When the sample has a lower mean than the population
    during the period used to calc the anomomaly the graphs are collapsed in Y. When the sample
    is Warmer than the population they are widened in Y.

    I’m going to ask Nasa for the number, but until then I’ll adopt this approach.

    I ran open temp on all 1221 stations. the 1950-1981 mean came out at 11.20 and change.

    SO, I generated a GISS absolute based on the GISS anomaly + Opentemp_1951_80_mean.

    To calculate anomalies for CRN12.. or CRN5 or CRN whatever.. I’ll just subtract 11.2.

    But its about WARMING so I was kinda curious about you were differencing anomaly?

    I asked SteveMc the same question and he ignored it. You really shouldnt difference anomalies
    unless they are calculated according to the same base value. For trends, it doesnt matter.

  288. SteveSadlov
    Posted Oct 13, 2007 at 12:07 PM | Permalink

    RE: #287 – Some other highly interesting means would be 1925 – 2005, 1900 – 2005, 1925 – 1980, 1900 – 1980. Then it would be interesting to plug each of these in.

  289. steven mosher
    Posted Oct 13, 2007 at 12:44 PM | Permalink

    Here Ya go.

    This is GISSTemp compared to CRN12 ( all CRN12) reference period for anomaly is 1996-2005.

    http://img134.imageshack.us/img134/9173/giss19962005jy7.jpg

  290. steven mosher
    Posted Oct 13, 2007 at 12:46 PM | Permalink

    Gisstemp – CRN12 1996-2005 reference period

  291. steven mosher
    Posted Oct 13, 2007 at 12:48 PM | Permalink

    RE 289 done right

    GISS Anomaly versus CRN12 Anonmaly WRT 1996-2005.

  292. Clayton B.
    Posted Oct 13, 2007 at 1:05 PM | Permalink

    A look at OpenTemp averaging distances
    I ran OpenTemp at various cell point averaging distances to see the effect (if any). My initial thought is that 1000km is too large of a radius (see pic below):

    So I looked at 100km, 250km, 500km using all USHCN stations and hcn_doe_mean_data.

    Note that 42% of the cells in the 100km case had less than three contributing stations. 250km and 500km had no cells with less than three contributing stations.

  293. steven mosher
    Posted Oct 13, 2007 at 1:25 PM | Permalink

    RE 292.

    Dang. I think Hansen only tested 1200km versus 1000km. Now, the reason why he uses that approach
    is the ROW is sparse.

    Another thing, When I run all 1221 I still have stations outside the grid ( coastal) can we get that
    nit fixed?

  294. steven mosher
    Posted Oct 13, 2007 at 2:04 PM | Permalink

    Here ya Go. Three lines. Opentemp ALL CRN12345. OpenTempCRN123 OpentempCRN45.

    Goldilocks chart.

  295. Kenneth Fritsch
    Posted Oct 13, 2007 at 5:39 PM | Permalink

    The more I look at the differences between the Urban and MMTS USCHN data sets the more convinced I am that the differences are not primarily due the averaging algorithm used for filling in the missing data points into the Urban data set. Much of the differences for these two data sets are a nearly constant amount for a given station over a long period of time. The differences between data sets can be large for a given station, but the fact that differences are the same or nearly same for many of the years produces difference trends that are small, but I do not think insignificant. The differences (MMTS-Urban) in yearly average temperatures by station due to the differences in the MMTS and Urban data sets are summarized in the table below for CRN12 stations.

    What is perplexing is that in a few years worth of station data out of the total years in the time period (1920-2000) there can be very large deviations from the normally rather constant differences in temperatures due to data set differences. They are large and fit no pattern and thus make me judge that they are errors in the less adjusted MMTS data set that get adjusted in the most adjusted data set, i.e. the Urban data set. Based on my observations I do not see a good reason for using the lesser adjusted data sets in these analyses. At least I would think we should be considering in much greater detail what the differences in the progressively adjusted USCHN data sets comprises before we settle on one data set over another.

    Is the averaging method used to fill in missing data points in USCHN confined to an individual station or does it use a number of nearby stations? Even if it used nearby stations I am not convinced that such an averaging would significantly affect trends and might well be superior to using data that might have significant errors in it. The large differences one sees in absolute temperatures makes one very wary of ever comparing absolute temperatures – as I attempted to do in matching pairs of CRN12 and CRN5 stations.

  296. Clayton B.
    Posted Oct 13, 2007 at 5:41 PM | Permalink

    293,

    I understand why he uses the 1000km but there’s no reason to keep using that for just an analysis of the lower 48. I think using all USHCN stations the best is 250 km since it was the smallest distance that returned no cells with less than 3 stations.

    We still gotta get those 50 or so stations inside of the mesh.

  297. steven mosher
    Posted Oct 13, 2007 at 6:21 PM | Permalink

    RE 296. A bunch of them are in florida maybe JohnV or you can adjust the Mask.

  298. Clayton B.
    Posted Oct 13, 2007 at 9:21 PM | Permalink

    here’s what we have right now:

    I haven’t been able to find a better boundary list online; they’re too detailed.

  299. Clayton B.
    Posted Oct 13, 2007 at 9:26 PM | Permalink

    JohnV,

    If stations are outside of the boundary do they still contribute to cells that are within 1000km or are they ignored?

  300. steven mosher
    Posted Oct 14, 2007 at 6:57 AM | Permalink

    Clayton if you run all 1221 then you get a list of stations not in the mesh.
    I started to look each up, there were a bunch. keywest, fort lauderdale..

  301. Steve McIntyre
    Posted Oct 14, 2007 at 8:25 AM | Permalink

    I checked the plot comparing NASA and NOAA, which John V observed to be inconsistent with other info – a point with which I did not disagree. I’ve replaced the graphic, as I was not able to replicate the prior graphic, I’m not sure why. The calculation and plot script for the present graphic is here. The double-checked graphic shows a marked increase of NOAA relative to NASA since 1940 for the US Lower 48.

  302. steven mosher
    Posted Oct 14, 2007 at 10:53 AM | Permalink

    Well,

    I have been pondering this CRN thing. One thing that occurred to me is we repeatedly found good
    stations turned into bad stations by changes in equipment. Notably the MMTS which moved the
    sensor closer to buildings. So, I started to cut the data a bit differently.

    1. Here is a first Cut. CRN123 WITH the MinMax gauges. Old school. Here are the anomalies

    For GISS and For CRN123mm. 5 year smooth trailing.

  303. Anthony Watts
    Posted Oct 14, 2007 at 1:22 PM | Permalink

    Mosh let me make sure I understand the above graph.

    1. This is plots of traditional stations in Stevenson Screens using mercury max-min thermometers only.

    2. The GISS plot is the same stations, using GISS’s own data source.

    3. The CRN123mm plot is using USHCN data V1

    4. The output is from OpenTemp, most recent build.

    Did I miss anything in getting the parameters of this run defined?

  304. steven mosher
    Posted Oct 14, 2007 at 3:50 PM | Permalink

    OK. here are all the parameters.

    “Mosh let me make sure I understand the above graph.

    1. This is plots of traditional stations in Stevenson Screens using mercury max-min thermometers only.”

    A.I took your spreadsheet. I sorted on INSTRUMENTS. I Selected the MINMAX. I assume this LIG old style
    minmax in Stevenson shelters.

    B. I took that list of MinMax stations (200+). I sorted on CRN. I selected CRN1, CRN2, CRN3. THIS gave
    me 39 stations.

    C. I Fed this list of stations to Opentemp.

    2. The GISS plot is the same stations, using GISS’s own data source.

    A. No GISS is the GISS anomaly published on NASA. It represents the anomaly from 1950-1981. This
    IS ALL THE STATIONS USED by GISS. So, it includes the 39 stations. Basically, if I
    removed the 39 stations from GISS the picture would get worse.

    3. The CRN123mm plot is using USHCN data V1

    The CRN123mm plot uses the ghcnv2 data. I have not switched to johns new data set and it wont
    make a differene here. Later I will download the lastest data, but right now I am using the
    Orginal OpenTemp and orginal Ghcnv2 data. For the most part JohnV, Kenneth and I have
    had very similiar results without regard to data sources ( quibbling about Tobs, shap, filnet etc)
    Nothing dramatic. SteveMc and JohnV have the newer ushcn data. He can double check my work. The best thing
    you can do is a BLIND duplication of my station list. MINMAX ONLY, CRN123. give that to JohnV to run
    and steveMC. Clayton can run it too as can Kenneth.

    I Use 11.2C to create my anomaly. This is the 1950-1981 Mean for all 1221 sites.
    SteveMc and JohnV do something slightly different. But the difference in absolute
    temp was so huge, you cannot ignore it.

    4. The output is from OpenTemp, most recent build.

    I believe so. There havent been anychanges except some minor issues relating to 2006 and incomplete
    years. STILL, I would insist that JohnV, kenneth, Clayton, SteveMc, rerun.

    Did I miss anything in getting the parameters of this run defined?

    Not really. here are my concerns.

    1. My selection of stations. Clayton has a database built. he should double check. Same with StMac.
    2. Stations in the grid. I got 38 ( one dropped) thats still a low number.
    3. We are talking anomaly in my chart HERE. TREND is a differrent issue. JohnV and Kenneth will righfully
    pummel me If I dont point this out. Still, I believe the KEY to finding a trend difference is
    finding a difference in absolute C. ( HUNCH )
    4. ASOS is Warmer than Norm and Warming.
    5. GEOGRAPHY can screw this up. Especially altitude differences. But I can imgine why
    high alt sites would be minmax? Could be a correlation.

    Bottom line I have been slicing through the data in a bunch of different ways. I did the instrument slice
    today for the hell of it and bam.

    Need to get dates when each station switched over. Obviously ASOS and H0 83 dont go back to 1880

    Was that confusing enough?

  305. Anthony Watts
    Posted Oct 14, 2007 at 4:18 PM | Permalink

    Thanks Mosh, this bears further study. Steve Mc, perhaps its time to make new thread USHCN class look #3, this one is getting a bit dicey to load.

    I’ll look over the USHCN list and try to ID what max-min stations remain. Important to get them surveyed before they get converted to MMTS.

  306. Steve McIntyre
    Posted Oct 14, 2007 at 4:41 PM | Permalink

    Continued at 2201.

One Trackback

  1. [...] is excellent agreement between GISTEMP and my results using only the best stations,” Vliet explained. Vliet has published the code he used in an open source project called OpenTemp.org, and his [...]

Follow

Get every new post delivered to your Inbox.

Join 3,146 other followers

%d bloggers like this: