Climate Audit

steven mosher

RE 398. Kristin, Lets see if I can help

You wrote:

“TOB is not an issue for me, I think it’s probably pretty close.
My question about TOB was if it was a part of the data for certain years,
something JerryB answered in 391.”

Time of Observation adjustments are made when the guy reading the thermometer changes
his time of Observation. So, it depends upon when the changes were made, and what
change was made. I’ll link JerryBs Description. It is very lucid and added 6.23 points
to my IQ:
http://www.john-daly.com/tob/TOBSUM.HTM

“I’m not convinced on outliers, do you have a link for the tables?
I would like to get a better idea of it.
It will probably affect CRN5 depending on the amount of the mean.”

Ok, One class that I wish they offered in high school when I was young
was probability and statistics. Because those branches allow you to
DO STUFF. cool stuff. Win at poker and blackjack..

If you flip a fair coin 100 times, do you believe that you will
get 100 heads and no tails? What are the chances of this happening?
If you DO get 100 heads in a row, do you think the coin is fair?
or do you think the coin is biased?

Essentailly, they throw out data points ( like 1 month or 2 months) when
those data points are RARE events.. like fliping heads 10 times in a row.
It is not a class5 issue. Microsite bias is not a spike. If it were a spike
it would be detected. It is a sneaky bias. It creeps up.

OK. You wanted a link to tables. Start here at wiki.
http://en.wikipedia.org/wiki/Standard_score

A good table is here:
http://www.isixsigma.com/library/content/zdistribution.asp

So if you look at the last table for 3.49 Sigma, you will see this.

The chances that a measurement ( tempeature recording) is 3.49 Sigma
greater than the mean is 2 in 10000. The chances that the measurement
is less than 3.49 Sigma of the mean is also 2 in 10000.

If I deal you 5 cards, the chances that you get 4 of kind is about 2-3 in 10000.

So, when they look through the records they are throwing out extremely rare EVENTS.

make sense. They are not throwing out big chucks of data. They toss the few cases that
seem odd or rare. It doesnt effect the mean when you have 1500 monthly data points.

Here is an example. you have 1000 months of data. Temperature is 0C for 999 months.
One month in the middle of this has a recording of 100C. What is the average
(100)/1000 = .01C or is 100C an Outlier. A mistake.

“I doubt it would affect CRN1,2 sites ”

Outliers typically result from things unrelated to class12 or 5. The guy writes down 5C
rather than -5C. Or the thermemter breaks. CRN12 describes the physical being of the site.
An observer can make a reading error there just as easily as at a CRN5 site. See? For example.
Orland is a great site. To get the tempeature a guy has to walk out and write down the MIN
and MAX. Suppose he sees 62F and rights down 26F? THAT has nothing to do with asphalt. When
he writes down 26 and sends that data in…. The “outlier detector” says.. OPPS. I think
our observer made a mistake. Now usually ( hansen does this) they CHECK other stations in the area
HEY, sometimes you get 4 of a kind.

“and I thought John’s last post on CRN1,2 vs GISS was very compelling (with one question below)
but I’m not as convinced with his CRN 5 analysis.
I do not agree that the temperature spikes will be just 1 to 5 % of the time,
it’s probably closer to the percentage of sunny days for pavement and summer days for AC and etc,
of course that will also depend on geography because here in the Northeast the pavement is
usually covered by snow in the winter.”

We’ll John Needs to build an expectation ( a model of what he will see) in order to do a PROPER
test. I’ll do it using my expected value approach. Because it’s probably easier to understand but
less elegant than his approach.

A class 5 site can see a micro site bias of 5C.

1. This bias will apply to TMIN. The bias in TMEAN will be 2.5C
2. This bias will have the same seasonality as UHI. Summer is the worst. 25% of the year.
3. Now we are at .6C.. simply the microsite bias works at night in the summer.
4. Cloudy days cut the effect
5. Rainy days cut the effect
6. Windy days cut the effect.
7. Guess at cloud/wind/rain modulating half of the effect. .3C

now, this .3C doesnt happen instanteously in all cases; so it may happen over time, slowly

steven mosher

Anthony, I think I have a proposal for the pre 1900 records. Can you tell when
a station ad the CRS installed? Then I think we would have a historical fact
to base the exclusion on. This migt be different for different stations. The early
record is is exposure. CRS solves that, so perhaps excising pre CRS records is a
finer grained approach, you might have 1902 for some sites and 1904 for others.
That gives us a solid defensible case

John F. Pittman

Why introduce another bias, if it is only 1 day or so a month? Even if it was every time, then you would still have 15 good data points. The monthly average then to yearly average would make the bias small. Also since it can be either warm or cool biasing even when it is cool or warm, isn’t it a good assumption to assume iid?

MrPete

I honestly continue to fail to see why TOBS adjustments are so important. Perhaps I’ve missed it, but follow with me for a moment.

No matter what time of day the observations are made, if they are consistent for a given station, the observer is always recording the high and low temp for the previous 24 hours. It might be noon-to-noon, midnight-to-midnight, or 6pm to 6pm. But it is a 24 hour high/low. For that station then, the trend recorded is a valid daily trend. Period. Might be shifted 6 hours from another station’s measures, but it is still a valid daily record!

Do we really need to align the different stations to be recording the exact same 24 hour sequences? I see no value in this, and much opportunity to make a mess of the data by attempting to “adjust” for TOB.

Where am I confused?

Steve McIntyre

I mentioned this once before, but the logical way to check TOBS is through a new analysis using CRN data which is hourly. Empirically calculate the effect of 5pm versus 12 am measurements for stations in the same area.

JerryB

“I mentioned this once before, but the logical way to check TOBS is through a
new analysis using CRN data which is hourly.”

A problem with that currently is that CEN is only a few years old. The
“period of record” of the data may be too short to be sufficient for such a
study at this time.

Hu McCulloch

Jerry B (#463) says,

Re #458,

“Suppose that every day in July the low is 50 at 2AM, the 7AM temp is 60, the
high is 80 at 2PM, and the 5PM temp is 70.”

If max/min temps of successive days do not vary, TOB does not occur.

Actually, it occurs to me now that even with constant daily temperature patterns, TOB will occur, on the day of the conversion, and that this may be what Karl et al (see #458) are trying to quantify.

Suppose as above that the daily highs are always 80 at 2PM and the lows are always 50 at 2AM. If there is no change in procedure, PM observers will always report 80/50, as will AM observers. But if on July 15, say, a 5 PM observer switches to 7AM, since the thermometers (or MMTS)were reset the previous 5PM (when the temperature was 70), the reported high/low for July 15 will be 70/50 instead of 80/50, a big difference! The Karl et al diagrams start to make sense in terms of this sort of bias. Furthermore, for the month of the change , the immediate effect of the change (their TOB) is much bigger than the timing effect of attributing the last day of the month to the wrong month (the second term in their Drift Corrected TOB or DCTOB), as they note.

In this example, if August has the same pattern as July, the immediate effect would be that one of the Maxes is 10 deg F too low, making the average max for July about .3 deg too low, or the average min/max mean about .15 deg too low.

However, the effects of the conversion do not stop there. August will lose its last day to September, and so forth, so that September, Oct., Nov., and Dec. will all be a little too high. The net effect on the year will be that the Dec. 31 high (call it 10 deg. F) will be replaced by a July 5 PM temperature (70 in the example), so that one daily reading for the year will be 60 degrees too high. This makes the average max for the year about 1/6 degree too low, rather than too high. Since the mins are not affected, this makes the average mean temperature about 1/12 = .08 degree too low, using these hypothetical numbers.

In subsequent years, however, the change is a wash, since the year’s own Dec 31 high is just being replaced by the previous year’s Dec 31 high. The TOB effect therefore cannot permanently affect the mean temperature. The fact that TOB does have a permanent effect in the USHCN adjustments shown by John V at #376 strongly suggests that NOAA or whoever has mistakenly treated it like a perment bias, rather than correctly as a transitory bias. And the fact that they are adjusting the numbers up rather than down suggests that they forgot all about the end-of-the year consideration.

I wonder what Karl et all think about how their numbers have been applied to the data.

steven mosher

Gentlefolk. TOBS IS off the deep end. Let me be a bit clearer.

1. the ISSUE at hand is MICROSITE BIAS. NAMELY, Sites that show a SITING bias that has NOT been accounted for.

2. TOBS. No one has demonstrated any issue in actual data with TOBS adjustments that has anything
to do with the rating of the Site. None.

Now, JerryB has linked his stuff and I’ve linked his stuff and you should all read Karl’s paper
before asking another TOBS question or proffering another TOBS opinion.
Further you can go download some CRN data.. in 5 minute increments
and study the stuff.

Now, do I think that it would be a good idea to revisit TOBS? Yes. But this is not the place.
One thing I love about this place is we always know how to get back on topic. Over at RC
every discussion degenerates into ” The ice is melting” and “fuel from dog poo and wood chips”

aurbo

Here we go again. Re #1 and #9, and perhaps a few others:

TOB can have a significant effect on the Stations’ monthly (and annual) mean temps. This is pretty much along the lines of #14’s examples. The reason this happens is that temperatures at the time of observations are double counted. If the TOB happens to be near the normal time of minimum (or maximum) daily tempseratures, then these extremes will be counted twice while it’s opposite max (or min) is counted only once. Using the princpio ad absurdum, say the min temp occurs at ob time and is a spectacular 25°F below the daily normal. This anomaly will appear on the record for the 24-hour period ending at the TOB and the 24-hour period beginning at the TOB. The two different maxes for each day will be counted only once for each 24-hour period. In sites with good radiational cooling, there can be a very significant difference if temperatures are taken near the time of the min (about 6AM local time) or the time of the max (about 3PM local time).

Notwithstanding the above, as re #1, if the station continues to use the same time of observation throughout its record, the trends at the station will not be affected provided that the trends are the same for both the max and min temps. The real problem arises when TOB is changed. This creates a discontinuity in the station record which should be accoumnted for. Many long-term sites used the once-a-day max and min temperatures to construct their daily mean temperature record, and then, when replaced by automated recording temperature devices, switched to hourly observations with the means based on the 24-hour averages. In these cases TOB can cause a significant discontinuity in the mean temperature record.

All that being said, I have no problem with TOB adjustments provided the algorithms used to nomalize time of observation bias be carefully constructed to work with the particular time of day such obs are taken.

aurbo

TOB = Time of Observation Bias.

M. Simon

MrPete says:
September 23rd, 2007 at 9:09 pm,

Your confusion is simple. The reconstructors are trying to develop an “accurate” temperature record vs just finding the rate of change. If you just worked out the rate of change for consistent records (same inst, same site, same TOB etc.) then a lot of the errors in the temperature reconstruction method either fall out or cancel (not all of them).

In fact the best way to reconstruct a temperature series (IMO) would be to do the slopes and then pick a calibration period and integrate the slopes to find actual temps.

Alan Cheetham

Hansen 2001 refers to both Karl et al [1986] and Easterling et al [1996] regarding TOB adjustments. I haven’t been able to locate Easterling [1996] yet, but I have read Karl [1986]. What TOB model is used by NASA/Hansen or by GHCN? I am assuming that the Easterling [1996] provides an updated model, because the one in Karl [1986] has the following issues:

there can be considerable variability of the TOB from year to year. This year-to-year variability [in TOB] is … due to differences in the timing of frontal passages, cloudiness, precipitation, etc. …the TOB for any month and given year can be substantially different from the mean TOB. This suggests that estimates of TOB from a general model will be most appropriate when applied to a mean derived from a series of years… The technique which we develop in this article is most appropriate when applied to means comprised of a series of years to estimate nonclimatic trends

In other words, it is inappropriate to use the Karl [1986] TOB model to adjust individual months/years.

MrPete

Thanks for the interesting perspectives proffered. My take on what I’ve learned so far:

1) The published technique for TOB adjustments is not appropriate for climatic trend analysis

Karl et al themselves declare “The technique which we develop in this article is most appropriate when applied to means comprised of a series of years to estimate nonclimatic trends which are detrimental to spatial and temporal analyses of mean monthly maximum, minimum and mean temperature.”

I.e. (if I am correctly interpreting that statement) they have developed an estimation technique that is “most” appropriate for spatial and temporal analysis of mean temperatures… in other words, it is based on a generalized model of temperature change across latitudes (northing) and longitudes (time of day shifts due to sunrise/sunset/timezone shifts).

I understand the reasoning behind their adjustments. It’s not too different from the adjustment built into my chicken coop automated seasonal light timer: we can turn the light on for a certain period before/after sunrise and sunset. The timer needs me to provide a general sense of our latitude, and to preset the time of day. However, I must adjust the sunrise/sunset offsets, because solar time of day varies by longitude. (BTW, the movie Longitude is a fascinating look into the history of accurate time measurement!)

For reasons discussed below, I believe attempts to apply such calculations to specific stations and/or for climatic trend analysis, are misguided.

2) The day of a time of observation shift, data is invalidated.

If observer reads “late” by two hours, then a high/low occuring in the final two hours of the previous day, larger than today’s high/low, will overwhelm today’s data. Iif the observer reads “early” by two hours, then a high/low occuring in the final two hours of the previous day will overwhelm today’s data.

Has anyone seen analysts actually invalidating data measurements due to TOB shift? I mostly see attempts at “correction” for duplicate/missing data!

3) It’s assumed that measurement at certain times of day will record duplicate high or low values over time

Yes, if the daily high/low observation is recorded and reset at the exact moment of the daily high/low, that high/low will affect both the prior and next day (if it happens to be the actual high/low of both 24 hour periods.)

Daily highs typically fall in mid-afternoon; TO (Time of Observation) seems unlikely to affect the high, ever, for morning, evening or midnight observation times.

Daily lows… I’ve been examining Weather Underground Personal Weather Station data in my area, and I see few if any patterns. The low could be near midnight, near five AM, near sunset… it all depends. It seems reasonable that a random percentage of daily measurements will be influenced by TO.

But there’s the rub: whether consistent or random, the trend is not impacted by TO, only by anomalies when TO changes.

The Karl paper is a wonderful presentation of certain analyses. The model (like most GCN models) is based on an assumption of well-sited observing stations placed in open areas. No shadows, no sun-facing thermal masses, nothing impacting either the measurements nor the general regional temperature. The assumed picture seems to be one of gently sloping land covered by smooth vegetation or desert. Such landscapes do exist of course, but they’re hardly the norm!

What the Karl method fails to do (and does not attempt to do) is show statistical skill in detecting and correcting realistic TOB based on real-world data. I look at the five-minute observation curves for sites near me and see a lot of very unpredictable shifts. Clouds move in and prevent the night from being “frizzling” cold as my daughter used to say, or they prevent the day from becoming a scorcher. A hail storm turns the temperature curve upside down. Daily rain drains off afternoon heat potential and sets up the evening as the high of the day. Etc etc.

4) Specific site issues trump TOB

A sensor located close to the east side of a tree, hill or mountain will have early “sunset.” West of an obstacle? You get a late sunrise. Here along the Front Range of Colorado, and in many other places, such effects can’t be avoided. But is that a bias or part of the reality of climate? Geography impacts climate! Should we “adjust” for this? I don’t think so, not if the “siting” issue is natural and impacts the region of interest.

As Stephen Mosher has indicated, other issues can easily overwhelm any potential for TOB. If the sensor is placed on a roof (or, to be extreme, near a wall of water barrels, in the exhaust plume of an A/C unit, etc et), the nearby thermal mass, thermal source or cooling source will have a massive effect on the daily temperature curve… potentially far more impact than the time of observation.

My bottom line: in theory, TOB seems a reasonable calculation. In practice, what I’ve learned is that more data should be invalidated (due to observation-time change)… and there ought to be a whole lot FEWER attempts at micro-adjusting for TOB because it really is not all that predictable.

A topic that seems pertinent to me: the recent discussion elsewhere on CA about proper handling of missing data. I’m not sure that’s been fully resolved.

Philip_B

Some points to bear in mind about TOB.

1. The bias in TOB is only relative to the convention of determining average temperature as the mean of min and max temperatures over the 24 hours prior to midnight. That is, it’s not a real bias, in the sense of an error.
2. In temperate latitudes there is a symmetrical seasonal effect in TOB. In practical terms this means while monthly TOBs can be significant (as much as 1C) around the equinoxes, over the year monthly TOBs tend to cancel out and are much smaller over a year.
3. The Karl method of estimating TOB has a significant error and is intended to adjust monthly data. Karl said the error was around 25% of the estimated adjustment. As far as I can determine, annual TOB is just average monthly TOB. Given point 2, annual TOB is probably mostly estimating error.
4. The TOB adjustment in the US data (sorry, I don’t have access to the computer where the links are saved) since 1950 has increased from a small negative adjustment to around 0.2C positive adjustment. This represents at least a third of total twentieth century warming. Note that a positive adjustment could only occur in the real world as a result of a wholesale shift in time of observation from evening to morning. In reality, over this period, the shift would have been to automated measurement, which presumably occurs at mid-night and hence cannot have a TOB. What we should be seeing is a steady reduction in TOB to zero, rather than the steady increase we do see. I. e. the TOB adjustment is definitely wrong.
5. The problem in sorting through this (mess) is that current automated measurements cannot have TOB and you need to reconstruct TOB in old manually collected data in order to determine what TOB adjustment is needed to handle the change from manual to automated measurement. My suggestion for removing the TOB problem is to just use data collected automatically to determine the temperature anomaly and see what trend there is and then statistically graft that on to the manually collected record (as per the Hockeystick). BTW, I’m not a statistician.

Brendan

Aurbo (#22) – Thanks! My background is heat transfer, and I taught graduate lab courses, including issues dealing with psychromatic ratios (ie, mass transfer). But I used all modern equipment, and so understanding that historical reference was illuminating. I assumed that such a device was possible (human ingenuity, and an understanding that observors in the 19th and 20th century would want to do a good job of recording max and min). I get to pass this on to a friend of mine to show that he was incorrect…

JerryB

While I have mentioned that TOB does not depend on passing warm or cold
fronts, I had never quantified how much it does not depend on them.

Let me use the word spike(s) for brief changes of temperature up, then
down, or down, then up, by at least some particular number of degrees.

When I did my TOB study, I deleted 5 such spikes of at least 20 degrees
F, not because of their possible effects on TOB, but because of a concern
that they might be erroneous data entries. I have no doubt that the one
of +95 F/-100 F was an erroneous data entry.

Also, I counted spikes of at least 3 F (128,981), and 5 F (17,240),
simply from curiosity, but did not attempt to quantify their effects on
TOB. BTW ,the frequency of such spikes varies greatly from one region to
another.

Comment #27 in this thread seems to include an assumption that TOB does
depend on passing warm or cold fronts.

“Yes, if the daily high/low observation is recorded and reset at the
exact moment of the daily high/low, that high/low will affect both the
prior and next day (if it happens to be the actual high/low of both 24
hour periods.)”

So, from the stations used in the study I selected one from Colorado:
Denver/Stapleton airport, which ranked #10 in the 5 F or more spike
count, but only #16 in the 3 F or more, not because it didn’t have its
fair share of those spikes, but because such small spikes are common in
more places.

Then, I went and flattened the spikes, first those over 9 F, then those
over 5 F, then 3 F, and then 2 F, and after each successive flattening,
reran the TOB calculations. Following are the summary results, starting
with the original that has already been published. The effects of the
spikes do seem very minor, and in some respects, contrary to what some
might have expected.

9 Yrs 50.71 -0.47 -0.19 0.12 0.39 1.62 1.26 0.89 0.64 0.48 0.34 One spike greater than 20 F removed
9 Yrs 50.72 -0.47 -0.19 0.12 0.39 1.62 1.26 0.89 0.64 0.48 0.34 Spikes greater than 9 F flattened
9 Yrs 50.71 -0.47 -0.19 0.13 0.39 1.63 1.26 0.89 0.64 0.48 0.35 Spikes greater than 5 F flattened
9 Yrs 50.72 -0.48 -0.19 0.13 0.40 1.65 1.27 0.90 0.65 0.49 0.35 Spikes greater than 3 F flattened
9 Yrs 50.71 -0.49 -0.20 0.13 0.41 1.68 1.30 0.92 0.66 0.49 0.35 Spikes greater than 2 F flattened

JerryB

steven,

Some graphs might be useful.

I am wondering what you could hope to see with the midway change, and am not
sure that I understand just what it is that you are suggesting be changed.

Are you suggesting, for example, 4.5 years of midnight followed by 4.5 years of 7 am?

Mike B

#34 I believe I have what you’ve suggested, but I don’t have a place to upload the image.

JerryB

Mike B,

You might try http://www.imageshack.us .

I just tried it for the first time, and we shall see if it works.

IanH

Someone needs to reset my perspective. We’re talking about a simple instrument that captures a high & low over a measurement period, that is then recorded at some time, be that 7am, 5pm or midnight. The TOB is supposed to cater for a change from one measurement time to another on a particular day. So worst case you need to discard one or two days samples when you make such a shift. The instrument itself continues to record a peak high & low regardless. Please try and explain without using graphs that go beyond +/-1day how this can affect the min/max temperature other than on the day of measurement change.

Paul29

I pulled down the last year of hourly temperatures (01:00 1 September 2006 through 20:00 31 August 2007) for Pittsburgh PA from the National Weather Service web site (http://www.erh.noaa.gov/pbz/hourlyclimate.htm). Of the 8,756 hourly readings, twelve were missing and replaced using a linear approximation based on the surrounding temperatures. The last four readings on 31 August 2007 were missing.

I computed the minimum and maximum for each 24-hour block starting at each hour of the day. So for the full year, I ended up 364 max/min pairs for each of the 24 possible data collection times. I found the midpoints for each pair. I then found the average of these 364 midpoint temperatures for each of the 24 collection times. The first column is the time of the daily data collection (recording the min/max pair from the previous 24 hours) and the second column is the average temperature for that collection time based on 364 readings.

0:00 51.84
1:00 51.74
2:00 51.67
3:00 51.57
4:00 51.45
5:00 51.27
6:00 51.11
7:00 50.97
8:00 50.90
9:00 51.06
10:00 51.26
11:00 51.51
12:00 51.74
13:00 51.89
14:00 52.10
15:00 52.37
16:00 52.61
17:00 52.78
18:00 52.81
19:00 52.69
20:00 52.44
21:00 52.22
22:00 52.06
23:00 51.94

There is a 1.92 degree F difference based on when the readings were taken. Readings that are taken in the late afternoon have the highest yearly average while those taken in the early mornings have the lowest.

pk

[IMG]http://i225.photobucket.com/albums/dd204/beach_040/TOB.jpg[/IMG]

Mike B

Rats! Sorry, I’ll have to come back to this later.

steven mosher

RE 36. JerryB IT WORKED.

So I downloaded your HOURLY data from TUSCON.

Then I created three vectors.

A vector of Tmean ( Tmax+tmin)/2 @ 7AM
A vector of Tmean @ 2400.

I was able to load 1984 – 1991. ( excell limits)

So things to Observe.

If you observed temp at midnight, you would see a TMEAN of 69.2 F
If you observed temp at 7AM you would see a TMEAN of 68.8 F

THAT IS NOT THE POINT OF TOBS ADJUSTMENTS FOLKS. Let me explain why.

If you check the TREND of a midnight OBS for this site it was .0004F day.
If you check the TREND of a 7AM OBS for this site it was .0004F day.

NOW, what happens if halfway through the history of this site we SWITCH from Observing
at Midnight to Observing at 7AM?

The MEAN goes to 69.0 The trend goes to .0001F day.

So, Go get a JerryB file. Download it. Compare the differences in means and trends when you
change the Time of Observation.

JerryB

Ian,

The bias occurs without changing the time of observation. Changing the time
of observation will change the bias.

The bias results from an accumulation of differences in temperatures of
consectutive days. Just two days of data may, or may not, indicate the
kinds of differences that cause the bias, but even if they do, just two
days of data will not indicate the magnitude of the bias.

At either page linked by this page you can see
two days of data that do indicate the kinds of differences that cause
the bias. Unfortunately, I may have crammed too much stuff into that two
days of data, but be that as it may, the necessary stuff is there.

JerryB

pk, Paul29, Mike B, steven,

We seem to have some actually independent replication going on here. This may
be unacceptable among some practioners of “climate change science”. 🙂

captdallas2

49 Steven Mosher, said: THAT IS NOT THE POINT OF TOBS ADJUSTMENTS FOLKS. Let me explain why.

The MEAN goes to 69.0 The trend goes to .0001F day.

So long term trends of raw data should match, long term trends TOB adjusted data if the TOB adjustment is properly calculated? Hmmmm, what a novel concept! If each time the time of observation, or the site is relocated or the instrumentation is changed, the site was treated as a new site instead of trying to micro manage the data to one uniform continuous site?

captdallas2

I referenced a paper by Thomas Blackburn a while back on TOB. The most interesting part of an interesting paper was how he worded something about the midnight ideal reading of daily observations. Just for grins assume that early AM is the optimum time for TO, what happens?

MrPete

Ummm… I finally woke up and found JerryB’s data links ;)… THANK YOU Jerry for going to all that trouble!

Now, I have a very basic question about the calculations made. I think I’m seeing a basic methodological error.

As stated, each high/low calculation incorporates 25 observations, not 24:

“The past 24 hours” will include the observations at both the beginning and the end of those 24 hours, and so will include 25 observations unless some data are missing. The “average (smoothed) hourly temperature of the past 24 hours” uses half of the first, and half of the last, of those observations (plus all of the other 23 observations). The number of consecutive observations will usually be 25, and if it is not, some data are missing, and 24 hour periods that are missing data will not be used in the summaries. If the “hour of most recent occurrence” is 25, it indicates that that occurrence was the observation at the beginning of that 24 hour period (i.e. it was 24 hours old).

I believe this is a basic error in analysis.

For each high/low record, only the most recent 24, not 25, observations should be used. Otherwise, high/low records 24 hours apart will incorporate, in essence, a one-hour overlap of measurements (since the measurements are hourly). This error will introduce a bias in the result.**

To consider this in a bit more detail, consider the case where the precision of our timing is to the second, and our measurements are made hourly. Then a 24 hour high/low calculation should be made incorporating data from T-0 to T-23h59m59s, not T-0 to T-24h — the measurement exactly 24 hours ago should not be part of the high/low calculation.

This becomes more obvious in a variety of computer science applications, where number space overlaps or gaps can cause serious errors in implementation of algorithms from random number generation to sort/search/hash tables etc. But all in all it’s the same problem. Getting the edge effects right is crucial.

Sadly, I’m not up to speed enough on R to code this (looks like a VERY interesting language, by the way!) But I do recommend looking into the result of correcting this error.

JerryB

Re #59,

If the hourly measurements were hourly high/low measurements, then 24
pairs would be sufficient for a 24 hour period. However, the hourly
measurements in these instances are point measurements, i.e. what is the
temperature at the time of observation. In order to use such
measurements to approach what 24 hour max/min measurements would record,
one must include the 24 hour old measurements.

Jon

How do you correct the TOB adjustments to account for variations between solar time-of-day and wall time-of-day?

Peter Hartley

The time of observation issue arises only because we are trying to measure maximum and minimum temperatures. However, it seems to me that Jonathan Lowe has shown that we might learn much more about climate theories by looking at trends of temperatures taken at the same time every day. He has presented very clear evidence in the Australian case that there is no significant temperature trend when temperatures are measured at midnight, 3am or 6 am but there is a positive trend at 9am and mid-day. From his previous work on individual stations, I think he is also going to show us that temperatures recorded at 3pm and 6pm every day also show a significant positive trend since WWII. This information would apper to be much more informative about which types of models better explain recent temperature changes than does any trend in maximum or minimum temperatures. Does anyone know whether Jonathan’s findings extend to other locations eg the US?

JerryB

Re #66,

The fact that hourly, much less six hourly, point temperature
measurements do not tell us what temperatures occurred between those
measurements is not in dispute. It is also completely extraneous to the
question of whether, or not, to include 24 hour old point measurements of
either such intervals when attempting to approach, as closely as either
such data permits, 24 hour max/min observations.

Barring operational errors, with 24 hour max/min observations, the
temperature ranges of consecutive days always at least meet, if not
overlap. When using point measurement data, if one does not include 24
hour old point measurements, one can get gaps between consecutive day
temperature ranges, which imply either operational error, or in the case
of what you propose, methodological error.

steven mosher

RE 65. Yes. That is not in dispute.

If a site is measured at the same time throughout its history.. you will get a trend figure.
( see my 49)

So you put two CRS shelters in you back yard. read the min/max at Midnight in one ( and reset)
Read the min/max in another at 7AM ( and reset)

Depending on your lat/lon and season, generally speaking the AVERAGE of the 7AM will be cooler
than the Midnight.

The TREND will be the same.

SO if the TOBS is the different for two collocated stations, you will see a BIAS in absolute temp
but No bias in trend.

Climate science is about the trend.

The TOBS adjustment comes into play when 1 station changes it TOBS throughout its history.

IF you start collecting temps at Midnight and then SHIFT to 7AM you will induce a shift in both
absolute and trend measures. USHCN TOBS correction attempts to Debias those changes.

I’d challeng TOBS on other grounds

captdallas2

69 Steve Mosher, if the raw data is indeed raw, trending the data per site should be the most accurate method of building the overall trend, via anomalities (sp) as John V. recommended. Instead of challenging, why not just build a better mouse trap? Kind of like Kung Fu, bend with the data, don’t force corrections.

steven mosher

RE 71.

Oh. Sorry. here is my position.

1. IF you are going to follow the ‘peterson, easterling, hansen” method of
creating a REFERENCE SITE that has the longest consecutive record as possible…
THEN.

a. You must adjust for TOBS changes at that site. They do. ( I want to audit that)
b. You must adjust for Instrument changes at that site. They do ( I want to audit that)
c. You must adjust for changes in site location( elevation). (They do… etc etc etc)

2. If you are going to follow a JohnV approach, you might not have to make these adjustments.

Agreed. That’s an attraction of JohnVs method. STILL. Unlike Mann and hansen and others I would
like to have Willis and JeanS and UC and SteveMc and Roman and professional statitiscians
weigh in on the decisions. JohnV and I are engineers. Well he is, I was. For us stats is a tool.
Sometimes we hammer with a screwdriver.

My goal here is focusing the questions on those areas that have not been questioned and verified.
JerryB has 9 years of hour data from 190 sites. The orginal TOBS study was 7 years of data from
107 sites. There are some issue here, but it is not a rich vein. Does it fascinate me? sure.

captdallas2

73 Steven Mosher, I appreciate that approach it is very normal. Outside of the box take a fictitious site called FT. Hansen, TX. 1890 to 1920 the soldiers on guard duty checked the temperatures every 6 hours starting at 0600. In 1920 they installed a brand spanking new max/min LIG thermometer and read it at 0700 hours. In 1992 due to Clinton base closures the LIG was replaced with an MMTS station.

1890 to 1920 station trend 0, 1920 to 1985 station trend 1, finally 1992 to present station trend 3. With the exception of the MMTS that requires correction because it sucks, you just have trends with variations above and below each trend’s average. Not the arbitrary 1951 to 1980 or whatever, the anomality of each trend. If the time of observation changes, thermometer breaks etc. new trend.

This reduces the influence of instrumentation error except for MMTS. You should find that pre-1950 divergence decreases. It is the KISS part of engineering school that I didn’t sleep through that makes me think this way.

Then pick any thirty year period to determine baseline for the combined trends. Auditing when Hansen and Gavin both say that have nothing to do with the less than optimum data is a waste of time. Do it the right way.

Philip_B

Jonathan Lowe’s analysis shows that minimum temperatures are increasing without corresponding increases in night-time temperatures. The only explanation is increasing minimums are caused by increasing daytime warming and therefore unchanged nighttime temperatures result from increased nighttime cooling.

This is directly contrary to the OCO driven warming hypothesis and in a rational world would be held up as a classic case of evidence disproving theory.

It also raises some interesting questions about exactly how temperature data can be used to determine if warming is occuring or not. Jonathan’s analysis is persuasive that over the last 50 years daytime warming has increased, but so has nighttime cooling by an approximately equal amount.

This may seem off topic for a TOB discussion, but increased daytime warming and increased nighttime cooling would tend to increase TOB due to increased diurnal range. Note that min and max temperatures give a misleading picture of diurnal range at least in the Australian data.

MrPete

(I’ve not yet had a chance to dig out my discrete/continuous number transform resources…will have minimal authoritative material to add until then.)

JerryB writes:

Barring operational errors, with 24 hour max/min observations, the
temperature ranges of consecutive days always at least meet, if not
overlap.

This is more or less correct. It depends on the device used, since some recorders automate the process completely. And a mechanically reset hi/lo thermometer will have (minor) reading gaps. But that’s actually immaterial to the question.

Proper management of high/low readings must be designed to avoid incorrect outcomes. And, by definition, one incorrect outcome would be to allow a moment to exist in two different time periods. One must choose: does midnight apply to day one or day two. No matter how finely you chop the times, each moment must apply to only one time period.

When using point measurement data, if one does not include 24
hour old point measurements, one can get gaps between consecutive day
temperature ranges, which imply either operational error, or in the case
of what you propose, methodological error.

To include the 24 hour old measurement in the current “day”, then the current measurement should be reserved as the beginning of the next day.

Otherwise, as you have so ably demonstrated, the “inter-day” reading is counted twice, biasing the results. No reading should be counted twice. It simply introduces the potential for error in analysis.

This is all a matter of properly envisioning the two dimensional space, in this case of temperature vs time, with respect to the discrete point measurements.

The common mistake is to assume that discrete point measurements define the outline of a curve that can be used to calculate an area for averaging, etc. That’s a pretty good approximation in many cases, and it makes for pretty line graphs, but it is not actually accurate.

Discrete measurements are points on a grid. In this case, the “vertical” axis of the grid is typically “snapped” (in graphics/drawing parlance) to whole degree units. And the “horizontal” axis is typically “snapped” to hours. One cannot simply remove the grid and presume to obtain correct results in infinite precision real number space.

[Yeesh. You’ve woken up some very old analysis I used to know by heart ;)… I do need to find that paper! I think I should be thanking you for that, Jerry… this was one of the more enjoyable portions of a past life :-D]

We need to adjust our thinking to recognize the underlying grid. Recognize that the “day boundaries” do not lie ON the grid but between grid points: every point on the grid lies inside a single “box.”

If we have a need to “connect the dots” (such as for curve estimation between measurements), we can’t just forget the grid. Yes, we can draw an imaginary continuous line between our measurements, but when we want to fix a given temp/time estimate along that curve, we recognize it will resolve to a spot on the grid, never in between grid points. And wherever that grid point is, lies inside only one “box.”

The fact of the underlying discrete measurement grid has significant implications on correct management of calculations. Many people never have worked through this question. As I said earlier, it’s typically a computer science problem only brought up when creating things like microcode for division algorithms, correctly implemented 2-D path intersections, and so forth.

Once this has been correctly implemented, any given time of observation will show no trend compared to any other time of observation…assuming changes in time of observation are treated as beginning a new, unrelated, series.

And now I see clearly: that’s the correct answer for handling changes in time of observation with respect to trend analysis. A new time of observation is simply the start of a new trend line. In the horizontal (time) axis, the two lines are discontinuous: either there’s a time gap or a time overlap. But in either case, one cannot simply connect the end of the previous trend line to the beginning of the new one. One must treat them as separate trend lines.

MrPete

By the way, working through the discrete/continuous grid question can eventually provide the statisticians in the crowd with the ammunition needed to generate minimum uncertainty levels for all of this. And the answers are a bit different than most would assume.

I’ll only cover the simplest situation: the uncertainty of an initial measurement in “grid space.”

Suppose we see 10 degrees at 6am, presumably +/- 0.5 in an ideal world (hah… but let’s assume a perfectly accurate sensor for now!)

Because of “the grid”, and the various ways that grid calculations can be handled in real world computer code, proper handling of the underlying uncertainty of that value should actually be as high as +/- 0.99999…, not just 0.5.

Without more information we do not know how the initial “10 degrees” grid point is arrived at. Is “10” used for an underlying (continuous) range of 10.0 to 10.999…, or 9.000…001 to 10.0, or 9.5 to 10.4999…? We don’t know.

Making it practical, it’s perfectly reasonable to see this entire range of uncertainties in the temperature record:

– Some sensors are read using a rounding algorithm. 0.5 below to 0.5 above.
– Some sensors are read using truncation (observers always report whole numbers!), which means below-zero temperatures get truncated UP by up to 0.999 degrees, and above-zero temps get truncated DOWN by up to 0.999 degrees.

And that’s all in an idealized, perfect sensor world!

MrPete

Re #82, there’s no such thing as “the temperature at the moment between two consecutive time periods”.

The temperature at that moment is either in one period or in the other. It cannot be in both. No individual moment is in two adjacent days, hours, minutes or seconds. It has a discrete identity.

Let’s simplify even further. One reading per year. I hope it’s obvious that we would never look at the following…

2004: 110
2005: 100
2006: 80

…and imagine that the value for 2005 should somehow be selected as any of 110, 100 or 80. The 2005 value is 100, period. In this case, 100 is both the high and the low for 2005. The value for 2004 and the value for 2006 are immaterial.

Extending from one reading per year to more has no impact on this truth. Each reading exists in only one time period, never two.

Philip_B

Conclusion
[9] We certainly realize that the conterminous United States
represents only 1.54 percent of the Earth’s surface area, and analyses of that areal unit may have limited interpretations for any global temperature record. Nonetheless, we show clearly that adjustments made to the USHCN produce highly significant warming trends at various temporal scales. We find that the trends in the unadjusted temperature records are not different from the trends of the independent satellite-based lower-tropospheric temperature record or from the trend of the balloon-based near-surface measurements.
Given that no substantial time of observation bias would be contained in either the satellite-based or balloon-based measurements, and given that the time of observation bias is the dominant adjustment in the USHCN database, our results strongly suggest that the present set of adjustments spuriously increase the long-term trend.

Click to access USHCN_Balling_2002GL014825.pdf

JerryB

Re #83,

“Re #82, there’s no such thing as “the temperature at the moment between
two consecutive time periods.”

Since every moment in time is between two consecutive times periods,
it would appear that you have attempted to banish temperature. 🙂

“The temperature at that moment is either in one period or in the other.
It cannot be in both.”

Perhaps your moments are much, much, larger than mine, but temperatures
commonly persist through many, many, of my moments.

“No individual moment is in two adjacent days, hours, minutes or seconds.”

True, but extraneous. I have not suggested, nor implied, that moments
straddle time periods.

BTW, it would seem appropriate that you stop posting false assertions about
what my method, or algorithm, or calculation, would do with your simple examples.

MrPete

Re #87, “Since every moment in time is between two consecutive times periods,
it would appear that you have attempted to banish temperature. :-)”

😀

I’m not suggesting the temperature disappears. Nor am I suggesting that time stops. I’m saying that measurements (of temperature or really almost anything) are at discrete, measurable moments, not continuous. And that time is recorded at discrete moments

“The temperature at that moment is either in one period or in the other.
It cannot be in both.”

Perhaps your moments are much, much, larger than mine, but temperatures
commonly persist through many, many, of my moments.

I realize this is not easy to grasp. Again, I’m not suggesting the temperature fails to persist. I’m saying the moment recorded belongs to only one period.

“No individual moment is in two adjacent days, hours, minutes or seconds.”

True, but extraneous. I have not suggested, nor implied, that moments
straddle time periods.

If so, then please explain your statement, and the algorithm that implements it:

The tempeature at the moment between two consecutive time periods is
the temperature at the end of the first time period, and it is also the
temperature at the beginning of the second time period.

As I read this, you are saying the moment at the end of the first time period is identical to the moment at the beginning of the second time period.

Suppose that the moment under discussion is 12:00:00am, January 1, 2007.

To me, that moment is ONLY part of 2007. It is not part of 2006. And any measurement at that moment is part of 2007, not part of 2006.

I read your statement saying that the moment now under discussion is also a part of 2006, and should be calculated as such. This is incorrect.

BTW, it would seem appropriate that you stop posting false assertions about
what my method, or algorithm, or calculation, would do with your simple examples.

If they are false, then prove it. The documentation and data sets say that your method calculates hi and low from N+1 samples for periods with N samples taken. How did I misuse that?

JerryB

Re #89,

“Again, I’m not suggesting the temperature fails to persist. I’m saying
the moment recorded belongs to only one period.”

And I am saying that the temperature, not the moment, belongs to two.

The tempeature at the moment between two consecutive time periods is the
temperature at the end of the first time period, and it is also the
temperature at the beginning of the second time period.

“As I read this, you are saying the moment at the end of the first time
period is identical to the moment at the beginning of the second time
period.”

I don’t know how you could read it that way; the temperature, not the
moment, at the end of the first time period is identical to the
temperature, not the moment, at the beginning of the second time period.

BTW, it would seem appropriate that you stop posting false assertions
about what my method, or algorithm, or calculation, would do with your
simple examples.

“If they are false, then prove it. The documentation and data sets say
that your method calculates hi and low from N+1 samples for periods with
N samples taken. How did I misuse that?”

My method estimates monthly, and yearly, TOB based on hourly temperature
observations. The first step of my method is to find sets of several
years of hourly temperature data. Your examples would not get past the
first step of my method.

MrPete

For completeness, I’ll respond to this right now:

Jerry wrote:

My method estimates monthly, and yearly, TOB based on hourly temperature
observations. The first step of my method is to find sets of several
years of hourly temperature data. Your examples would not get past the
first step of my method.

The method of estimation should work correctly anywhere, yes?

I will specify that my example is for a planet with one hour per day, one day per month, and one month per year. Thus, there is one sample per year. Perhaps the code does not handle such a case flexibly, but that’s not my problem. The underlying methodology should work no matter how few hours there are in a day, etc. QED.

(No, I’m not being silly. We’re dealing with a subtle boundary condition that’s best understood under simplified conditions. Throwing a lot of data at the problem is not helpful for understanding.)

Sam Urbinto

If I measure 10 C at 10:00:00 and then at 22:00:00 I have two 10 C readings 12 hours apart. If I get 8 the first time and 12 the second time, I have an 8 C and a 12 C measurement 12 hours apart. (And of course, the same average temp “for the day”)

If I measure the temp on 31 Dec 2006 23:59:59 and then 1 Jan 2007 00:00:01 I have two readings, probably the same number, 2 seconds apart. If I do that again in a year, unless at that time there’s some weather event different than the year before, I’ll probably get pretty much the same numbers.

If I measure temp every minute for a day, add all the values, and divide the total by 1440 I have that day’s average temp. Which information tells me basically nothing because there’s a whole bunch of combinations of the 1440 values that would give me the same average. But if I did that in December in San Francisco, Greenland and Panama, I could probably tell which was which.

If I measure my interior oven temp every second after set it to 400 F and turn it on, I’ll probably get a flat diagonal rising line over time as it heats. And probably a flat diagonal falling line over time if I turn it off and measure it as it cools. And if it’s been off for a long time, something around room temperature minus however the insulation affects it. If I add the high and low and divide, I get a meaningless number. Even if I knew how long the oven had been on and knew the slopes of the heating and cooling.

#92 Not a problem. I really have absolutely no idea what either of you are even talking about. Is it a discussion of sampling interval? Statistics? What temperature beer freezes? Who would win the Kentucky Derby if horses had rockets strapped to their heads?

MrPete

Sam is more or less understanding this intuitively. Take two readings in a day and they determine what you know about temperature for that day. (It’s easy to understand when the readings are not at the edge of the day. The challenge is how to properly handle the edges of the day.)

I really have absolutely no idea what either of you are even talking about. Is it a discussion of sampling interval? Statistics? What temperature beer freezes? Who would win the Kentucky Derby if horses had rockets strapped to their heads?

😀 The challenge we’re working through is how to properly perform the analysis that underlies the topic of this thread: Time of Observation Bias… and in particular how to handle the specific situation where you have a series of hourly temperature values. This is what Jerry’s data collection contains.

It’s already quite complete enough. The temperature persists through
the boundary between consecutive time periods, whether you prefer that
boundary to be a microsecond, or a nanosecond, or something smaller.

Yes, the temperature persists. The problem is with the boundary definition and how it is handled.

The boundary is not a time period. It is a dividing line that separates time periods; it defines which measurements go in one period or the other.

And the hard part to understand, is that we cannot simply switch from a discrete data set containing measurements at specific times, and convert to an assumed infinite-precision timeline linking those data values. To do so invites analysis errors. My simple example demonstrates the kind of error that arises.

I’ll stick with 24 hour days.

If you insist! It isn’t difficult to expand the simple example.

Heres an example using 24 hour days, and simple repeated data on each day:

Dec 31 2006: 24 samples, 0 degrees.
Jan 1 2007: 24 samples, 10 degrees.
Jan 2: 24 samples, 20 degrees.
Jan 3: 24 samples, 30 degrees.
…
Jan 31: 24 samples, 310 degrees. (Talk about AGW! :-D)
Feb 1: 24 samples, 320 degrees.
…

My method:
Jan 1 hi/lo/avg = 10/10/10
Jan 2 hi/lo/avg = 20/20/20
etc
Jan 31 hi/lo/avg = 310/310/310
January overall average = 160.0 degrees

JerryB’s method (for ANY observation hour, using my simplified data):
Jan 1 hi/lo/avg = 10/0/5
Jan 2 hi/lo/avg = 20/10/15
etc
Jan 31 hi/lo/avg = 310/300/305
January overall average = 155.0

Reverse the temps to be 320 on Dec 31 2006, 310 on Jan 1, 2007, 300 on Jan 2…10 on Jan 31…

My method: average remains the same at 160.0 degrees.

Jerry B’s method:
Jan 1 hi/lo/avg = 320/310/315
Jan 2 hi/lo/avg = 310/300/305
…
Jan 31 hi/lo/avg = 20/10/15
January overall average = 165.0 degrees.

My method gives an average temperature of 160.0 degrees for the month, no matter which day has which temperatures.

Jerry’s method gives an average January temperature of anywhere from 155 to 165 degrees, depending on the order and sequence of the daily temperatures.

We can randomize the days, do whatever you like. My method will always record the correct average value for each day and for the period. Jerry’s method will record a wrong answer any time the analysis (high or low for the day) comes from the final value in the previous period.

MrPete

Thanks for that reference, Steven! Here’s the link.

The first thing that stares me in the face about the manual is their rounding instruction:

Record these readings, as well as the current temperature, to the nearest whole degree. If the reading to the right of the decimal is 5 or greater, round off to the higher figure; i.e., 39.5 should be recorded as 40.

They’re using the old round-half-up method. It’s biased. Rounds up 5/9 of the time (5,6,7,8,9) and down 4/9 of the time (1,2,3,4). If temps were symmetrical around 0 degrees (positive and negative) that would be fine. But they’re not. I wonder if GISS attempts to “correct” for this? 😀

Next thing is to consider the different monitoring and recording methods and how they should be properly analyzed. Hmmm…. time to get some real work done and come back later…

Chas

Hmmm: R users ( and people who know how to use Excel) should avert their eyes now, I think.
Here is recipe for creating the TOB for a simple sine wave day, in Excel:
Note: maybe start with 100 rows and then drag it all down to a 1000 or more (even 10k) to get a cleaner result

-fill down a column with an incremental series 0,15,30,…
In the next column convert this to radians. in the next column (NC from now on!) sine this. NC, multiply this by -5 to give a reasonable diurnal variation [ the factor should be negative to flip the sine wave so it is cool in the morning] and add say 20 to give a reasonable average temp. NC, fill with normally distributed random numbers mean zero and SD ,say, 2. NC, add this noise and the sine wave temperature column.
NC, divide the 0,15,30 column by 7.5 to get hours. NC, ‘MOD’ this hours column by 24 to get time of day. NC, this should be the MAX of the previous 24 hours ‘noised’ temperature. NC, the MIN of the previous 24 hours temps. NC the average of these MAX and MIN columns. lastly ;level with the first row of your 24 hours max and min columns copy the time of day column across. These last 4 columns want a name putting above the info and then you can drag over them all (included the name) and create a pivot table&chart.

Very long winded, apologies. It is in fact a lot easier and less messy to install one of the freebie monte carlo add-ins which refreshes the random numbers and collects the information from chosen cells as many times as you want.

Out of interest, the result that I get from this sine-wave day is very similar to PK’s plot in #48.

steven mosher

RE 99.

I’ve beat the rounding drum before but no Indians came running to help

To make matters worse they record and round TMIN and TMAX. Then the add TMIN
to TMAX and divide by 2 and round up again.

One could suppose that this method introduces no bias into the trend.

Sam Urbinto

I think that’s why I called it an RC type discussion, they seem to go around in circles a lot there with nobody understanding they’re not really having a discussion on the same topic! Actually, come to think of it, it’s not just RC. I suppose it’s everywhere, just seems more so on ‘the alarmist’ sites than on ‘the denier’ sites.

Mike B

#99

You bring up another important area: determining what time the thermometers were actually read and reset. Even if you develop a fantabulous bias correction, it is useless if read times are innaccurate or unreliable.

JerryB posted the FTP site for the USHCN meta-data. The USHCN rates the reliability of time of observation with a variable that has three levels:

G for good.
F for flaky.
missing.

Is a station with a missing observation time reliability indicator flakier than a flaky station?

It’s stuff like this that drives me crazy.

MrPete

Perhaps an initial outcome of this first round can be evaluation of the underlying uncertainty introduced into the raw data record, from errors inherent to min/max reading methodologies, to improper rounding rules and more.

All of that takes place even with perfect sites, perfect observers, perfect observation timing.

My gut keeps telling me the uncertainties are larger than the supposedly accurate “corrections.” But I don’t suppose the media care.

Jon

If you measure the max temperature near the afternoon average temperature peak, on a day that is warmer than both its neighbors, there is a good chance that the temperature near the observation time will also be recorded as the peak for one of the adjacent days. Thus, high maxes tend to get double counted. The same is true for low minima if the recording time is near the early morning average temperature min.

This description is wrong–but the effect is real. Given a certain phase relationship between the min-max measurement period and the signal, there is some probability that min-max will capture the peak and trough. As the period moves such that one of the end points approaches one of the peaks, you no longer capture the peak of every period. i.e., two troughs (or peaks) end up in the same interval. Thus rather than double counting you actually undercount extrema and pick arbitrary interior points.

This effects occurs because the peaks are not uniformly spaced at 24hr intervals. Rather they jitter with a long-run period of 24hr. Physically this is something like amplitude modulation. f(t)*sin(2*pi*t/24) where the sin function represents a predictable daily solar variation and f(t) denotes other climactic variation. f(t) is responsible for the TOB.

Hu McCulloch

Correction — in #110, I wrote

Since about 28% of stations switched from pm to am between 1941 and 1985, this is about -0.25 deg. C, which would justify an upward adjustment of +0.25 deg. C. This is at least close to the approximately +.35 deg. C. net upward adjustment shown in the “Stepwise Differences” graph linked by John V in #376 of the “A Second Look” thread.

In fact, the “Stepwise differences” graph linked by John V shows approximately +.35 deg. F, or about +.22 deg. C, net upward US average TOB adjustment circa 1941-85. This is even closer to my back-of-the envelope calculation than I had thought!

Mike B

Sorry, this is a test.

Philip_B

You bring up another important area: determining what time the thermometers were actually read and reset. Even if you develop a

fantabulous bias correction, it is useless if read times are innaccurate or unreliable.

It would matter were the recorded Tobs used to determine TOB. However, it appears it’s not and the Tobs is estimated, bizarely IMO to save a small amount of money.

The fraction of observers recording at various hours of the day was calculated and interpolated for intervening years (extrapolated for subsequent years). For these seven states, the ending time of observation was grouped into three categories: AM, PM, and MD. The AM category included observers who ended their climatological day between 3 AM and 11 AM; the PM category between noon and 9 PM; and the MD category between 10 PM and 2 AM; all local standard time. The fraction of observers in these categories was calculated, and it was assumed the 7 AM observation time best represented the AM category; the 5 PM observation time, the PM category; and midnight for the MD category. The reason for the simplification was to test if a faster method, requiring significantly less bookkeeping and keypunching, could not provide nearly as good results as calculating the fraction of observers at each of the 24 hours of the day.

http://www.cdc.noaa.gov/USclimate/Data/

The TOB process needs to be fully documented. I suspect the reason it isn’t is both there are significant errors in the process and it looks embarassingly amateurish.

Mike B

#117

Wow. Thanks Phillip.

Perhaps even more revealing:

The time of observation was determined at each station within a climate division during January of the years 1931, 1941, 1951, 1965, 1975, and 1984 for the states of California, Colorado, Illinois, Indiana, New York, North Carolina, and Washington. The fraction of observers recording at various hours of the day was calculated and interpolated for intervening years (extrapolated for subsequent years).

and

As a result, by assuming 7 AM observation for all AM stations and 5 PM for all PM stations, a good estimate of the median bias is obtained for all AM or PM observations. Furthermore, nearly all the MD stations observed at midnight.

There’s lots of assumin’ goin’ on out there.

Look at the states listed there, and imagine trying to estimate family incomes or gasoline prices from that sample. I interact regularly with government agencies (Census Bureau, Department of Energy, Bureau of Economic Analysis) that wouldn’t dream of something so sloppy.

But I guess the right people liked the result, so it lives on.

Phil

#120 Sorry, reposting to get rid of the pesky less than symbol:

#119 However, they also say:

Differences of the biases were small ([less than] 0.3 Deg. F.) for those calculated by categorizing the ending time of observation into three categories compared to those obtained from calculating the fraction of stations with observation times at each of the 24 hours of the day. This is attributed to the preponderance of AM observation times falling between 6 AM and 9 AM, and PM observation times falling between 4 PM and 7 PM. As a result, by assuming 7 AM observation for all AM stations and 5 PM for all PM stations, a good estimate of the median bias is obtained for all AM or PM observations. Furthermore, nearly all the MD stations observed at midnight.

Thus, an additional incremental uncertainty of about 0.3 Deg. F. (~0.5 Deg. C) should be added to the estimation uncertainties (“errors of prediction”) in Karl’s paper (between 0.1 and 0.3 Deg. C). for the TOB process alone.

Phil

#122 Sorry, bad proofreading.

JerryB

One notion that I have is that it can be very difficult accurately to
describe in a general way the temperature sequences that lead to TOB. If
one considers the afternoon of day 1 of the Peoria sample,
the 1600, 1700, 1800, and 1900 observations are less than that day’s
high, and the first three of them are lower yet than the previous
afternoon’s corresponging observations. Also, the second two of them are
lower than the first two of them. So they may not seem too interesting.
And yet, they become the max observations of four of the next afternoon’s
hypothetical 24 hour max/min observations.

While considering some of Hu’s and Jon’s recent comments, it seemed to me
that some things that I had not counted in the hourly data might be of
interest. Here are some preliminary numbers from Cedar City, Utah, data
which may relate to discussions of “double counting” and “under
counting”.

Numbers of hourly observations of current 24 hour max temperature in 24
hour periods ending at 36167 hypothetical 24 hour periods of observation:

3 instances of 14 occurrences
2 instances of 13 occurrences
7 instances of 12 occurrences
2 instances of 11 occurrences
18 instances of 10 occurrences
20 instances of 9 occurrences
39 instances of 8 occurrences
78 instances of 7 occurrences
125 instances of 6 occurrences
249 instances of 5 occurrences
756 instances of 4 occurrences
2185 instances of 3 occurrences
7593 instances of 2 occurrences
25090 instances of 1 occurrence

Number of instances in which the current 24 hour max is at exactly the same
temperature as the 24 hour old 24 hour max: 3314, just under once per
11 hypothetical periods of observation.

Numbers of hourly observations of current 24 hour min temperature in 24
hour periods ending at 36167 hypothetical 24 hour periods of observation:

1 instance of 15 occurrences
5 instances of 12 occurrences
5 instances of 11 occurrences
7 instances of 10 occurrences
10 instances of 9 occurrences
9 instances of 8 occurrences
13 instances of 7 occurrences
58 instances of 6 occurrences
154 instances of 5 occurrences
988 instances of 4 occurrences
3884 instances of 3 occurrences
9911 instances of 2 occurrences
21122 instances of 1 occurrence

Number of instances in which the current 24 hour min is at exactly the same
temperature as the 24 hour old 24 hour min: 3554, just over once per
10 hypothetical periods of observation.

In case the wording of “Numbers of hourly observations of current 24 hour
min(or max) temperature in 24 hour periods ending at 36167 hypothetical
24 hour periods of observation:” is obscure, the following may help.
After the program identified the 24 hour min (or max) for a given 24 hour
period, it went back and counted how often that temperature occurred in
the hourly observations for that period, and recorded that count.

JerryB

Re #117,

The discussion at the link in that comment is not about how TOB is adjusted
in USHCN, but in some other NOAA activity, and should not be confused with
USHCN TOB adjustments.

steven mosher

re 128.

I found a TOBS event on the first page I looked at

pk

Also look at Marysville, October 1980. It looks to me like when temp at time of observation is lower than the min temp, they select the temp that looks more in line with surrounding days. Sometimes they let the min temp stand and cross out the temp at observation; and sometimes they change the min temp to what it was at time of observation.

steven mosher

RE 131.

The thing nobody knows yet is how these written records get input.

You have 1221 stations. 1514 months since 1880.

That’s a stack o paper

MrPete

OK, next step in my noodling on this:

A lot of effort has gone into analysis (and display) of station data based on the assumption that the data can be treated as a valid time series, with data points that (for the most part) describe a temperature “line” through time.

The more I think about it, I’m beginning to visualize the data (so far, mentally) as a scattergram of independent measurements, including at individual stations:

* Many data points are missing

* Many data points should be treated as missing or at least “fuzzy”, as the time of observation is inappropriate for a proper 24 hour hi/lo reading

* Many data points (big picture) have variant time of observation

This raises a few questions for me. Can anyone provide insight?

* Does R have good tools for dealing with sparse datasets?

* What are some good statistical methods for addressing sparse data?

* Any bids on good/proper/best ways to “bin” sparse data collections, so as to properly recognize trends?

This last one seems a tough problem.

One thought experiment: imagine a sine wave, sampled at regular intervals. Remove data points. Obviously a bad idea to simply connect across the “gap”. With a sine wave, we know the shape of the missing data and can easily fill in.

With temp observations, we don’t know the missing data. My goal is not to fill in the gaps, but properly analyze the trend without allowing the gaps to bias the analysis.

Now I’m itching to play with a scatterplot of measurements in 3-D space. Temp vs Day-Of-Year vs Years. Spinning that picture just might bring insight and inspiration on how to handle this. Then, take a set of such plots, one per station, and overlay them. Feels like a typical 3-D surface “fitting” problem…

(I realize that many have “moved on” from such elementary thinking. Sometimes, going back to the beginning can be helpful. This whole TOBS thing continues to feel like an exercise in learning to properly handle missing and invalid measurements.)

Steve McIntyre

#128. Pete, in some of the forms, there’s an additional column which records the time(typo) temp at measurement time. This is a substantial increase in relevant information for those stations. You can readily spot the affected days, as the temp at measurement time for the prior day=the next day’s max. (Update Note: Same for min in applicable situations, as Jerry B observes below. Only one or the other will be relevant in a typical case.) ) For those stations, one could do a spot check on the NOAA algorithm by calculating the “normal” difference between the max and the (say) 5pm value, perhaps allowing for some nonlinearity in temperature. Use this to estimate the missing max value when it’s been overridden by the previous day 5 pm value.

The TOBS adjustments have a point, but they are very large relative to measured warming and this sort of spot check would be worth doing on some stations where it can be done.

As to missing data, that’s a large topic and I don’t think that there are any magic bullets. For example, the issue of the Monday reading after a lost weekend – I presume that max-min on Mon afternoon would be for the
max-min for the 3 days since Friday. You’d have to do something like a TOBS adjustment for this, but intuitively it seems to me that it would only increase the range a little.

steven mosher

RE 137. Yes, I fact the first page I looked at for Orland had such a incident. I documented it
for JerryB above.

My sense is we could be counting the hanging temperature chads for a long time

JerryB

Re #139,

steven,

I took a look at that, and at the “USHCN Daily” data, to see how it got
interpreted. Also, while the operator made the change on the 18th, the
17th did not receive similar treatment by the operator.

In the USHCN Daily file, the min for both days was picked up from the AT OBSN
column. Also, presumably because the time of observation was 8 AM, the daily
max is recorded as that of the day before the day of observation.

MrPete

Geoff Sherrington’s contributions here are quite interesting to our discussion. That discussion majors on interpolation due to geolocation, and minors on time-based interpolation. But the issues are the same.

steven mosher

Hey JerryB !!!

Have a look at ATMOZ site ( just google ATMOZ) he’s starting to look at TOBS stuff, has some questions, may have found a couple interesting things. I suggested he talk to you, but didnt have
contact info for you.

steven mosher

thx jerry

steven mosher

Karls Tobs paper needs a fresh look

steven mosher

Kristin, you want the tobs paper

JerryB

Re #152,
Geoff,

Passing cold snaps, or warm ones, are minor factors of
TOB. TOB is mainly a result of ordinary, day to day,
variations of temperatures. Varying reset times will
merely corrupt the readings; it will not make TOB go away.

ChrisM

I have always doubted TOBS corrections. However, being a power station engineer, I have access to a lot of high quality monitoring data done to a lot higher accuracy than usual temperature measurements Processing that data, shows TOBS is real – as the summary below shows:

Average ((min+max)/2) Temperature Data
Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Nov year Oct year
TOBS
midnight 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 13.59 13.54
6am -0.21 -0.96 -0.16 -0.22 -0.31 -0.33 -0.35 -0.06 -0.30 -0.10 -0.33 -0.29 -0.45 -0.39 13.26 13.21
7am -0.10 -0.37 -0.09 -0.26 -0.31 -0.42 -0.48 -0.14 -0.42 -0.16 -0.13 -0.34 -0.21 -0.16 13.32 13.26
8am 0.36 -0.31 0.03 0.01 -0.13 -0.22 -0.15 0.03 -0.30 -0.06 -0.18 -0.10 -0.18 0.04 13.48 13.41
9am 0.22 -0.10 0.14 0.12 0.03 0.01 0.10 0.22 -0.14 0.11 -0.01 0.10 -0.01 0.11 13.64 13.58
10am 0.04 -0.07 0.22 0.18 0.03 0.18 0.24 0.33 0.05 0.26 0.05 0.15 0.15 0.12 13.74 13.68
11am 0.02 -0.07 0.29 0.21 0.03 0.22 0.37 0.43 0.17 0.34 0.10 0.16 0.25 0.13 13.79 13.75
noon 0.04 -0.07 0.36 0.24 0.05 0.25 0.57 0.55 0.25 0.50 0.21 0.16 0.30 0.15 13.86 13.82
1pm 0.06 -0.03 0.48 0.27 0.13 0.24 0.69 0.59 0.32 0.60 0.24 0.27 0.37 0.20 13.93 13.89
2pm 0.11 0.04 0.56 0.34 0.22 0.26 0.75 0.69 0.37 0.60 0.37 0.36 0.46 0.35 14.00 13.95
3pm 0.21 -0.23 0.68 0.55 0.44 0.46 0.81 0.70 0.41 0.62 0.19 0.58 0.38 0.40 14.05 14.00
4pm -0.10 0.19 0.70 0.51 0.43 0.46 0.61 0.52 0.19 0.46 0.39 0.39 0.56 0.46 14.04 13.99
5pm 0.00 0.18 0.59 0.45 0.44 0.36 0.46 0.42 0.11 0.37 0.29 0.30 0.48 0.43 13.97 13.91
6pm -0.11 0.14 0.43 0.34 0.32 0.28 0.33 0.35 0.11 0.30 0.25 0.20 0.42 0.34 13.88 13.83
7pm -0.15 0.19 0.38 0.35 0.28 0.23 0.26 0.26 0.10 0.20 0.20 0.16 0.26 0.31 13.83 13.78
8pm -0.13 0.13 0.18 0.20 0.07 0.17 0.22 0.18 0.08 0.19 0.14 0.13 0.20 0.15 13.75 13.69

Sorry about the formatting but I don’t know how to fit spreadsheets in.

The 4pm data shows TOBS is about 0.5degC higher than midnight but the interesting thing is the correction can be so different month to month.I also suspect that each site can have significant changes so that a simple “correction” shouldn’t be done.