Sorry to bother, but my reading of the subroutine is that if no record is ranked higher than ‘UNKNOWN’ then the first record with the longest length is used. If there is at least one record with a rank greater than ‘UNKNOWN’ then the first record with the highest rank is used regardless of length.

Anyway this confirms John G’s point and I need to modify my algorithm a little. As to why the ordering should seemingly be non-commutative in its impact on trends (if indeed that is the case), that should prove another worthy topic for investigation.

def get_best(records):
ranks = {‘MCDW’: 4, ‘USHCN’: 3, ‘SUMOFDAY’: 2, ‘UNKNOWN’: 1}
best = 1
for rec_id, record in records.items():
source = record['dict']['source']
rank = ranks
if rank > best:
best = rank
best_rec = record
best_id = rec_id
if best > 1:
return best_rec, best_id
longest = 0
for rec_id, record in records.items():
length = record['length']
if length > longest:
longest = length
longest_rec = record
longest_id = rec_id
return longest_rec, longest_id

I took a quick look at one of the stations with two records and a large warm bias applied to one of the records: CEKUNDA (GISS ID 22231532000). If you compare the two graphs here and here, you can see that the record is obviously biased downward (this is why Al Gore invented tabbed browsing). So something about our assumptions regarding the first record is incorrect.

Two other stations quickly jumped out and I think help point to an answer: BARGUZIN (GISS ID 22230636000) and BORZJA (GISS ID 22230965000). I see that the first record is at least tied for the latest record, if it is not clearly the latest record. If it is tied for the latest record, I noticed it is also at least tied for the longest record, as measured by the number of valid annual averages. Interestingly, Borzja’s second record is longer than the first, as measured in months, but equal as measured in years. The first record seems to have been selected as the starting point in the Step 1 process. Interesting…because it has more estimated values than the second record (a whole ‘nother topic).

At any rate, the ordering seems to be:
1) sort by record with the latest annual data
2) if tie, sort by record with the longest annual record
3) if tie, use the record with the smallest record index.

Now, in Russia (and the rest of the world outside the US), the only way the first record index has a snowball’s chance in an AGW world is if no MCDW (post 1990) record exists for the station. Well, that seems to happen a lot in the rest of the world, and it happened with Borzja, Cekunda, and Barguzin. And a bunch of others.

What I find really interesting, and will investigate further, is that the first record for those three stations (which “ceased operation” c1990) is cooler than the other records. That means the other records for that station are biased downward to the first record. However, for many, many stations in Russia that continued reporting past 1990, the first record is warmer than most, if not all, later records that overlap it. Thus, the first record is biased downward toward the other records. I find it interesting that, no matter how we slice it, the early Russian record gets biased downward. I didn’t think they were that oppressed.

Thank you for your patience re my #14. Occasionally, very occasionally, there is a Eureka moment but this was not one of them. You guys are far better at that and far closer to analysis of the developments.

I continue to have serious misgivings about (a) adjusting a data string based on an overlap (why up, why down, which one?) instead of simply averaging the overlap pairs (b) infilling missing values by interpolations that make dubious assumptions about constancy of weather over time (c) inappropriate assigning of weights to weighted averages (d) using a search range of 1000 km. and a linear drop-off as well (e) rejecting outliers instead of using them for valuable metadata interpretations.(f) as John G says, using a method that favours one dataset through order of selection over another when no apparent basis for preference exists.

Can I please ask another basic question? Why does one not take a simple arithmetic average of all available values at a point at a time, using the only value if missing data reduce you to just that one? (and using no value if one does not exist).

Exactly!

I think a “data flow” diagram will help people understand the “maze o data”

A picture is worth 988.6784679 words

There are some data sets that are much continuous and reliable, without the flaws that the step 1 algorithms are supposed to fix. They could be artifically modified to exhibit the flaws, and after running through the step 1 algorithms are the flaws fixed, by comparision to the original flawless data set?

This would illuminate whether the step 1 algorithms are creating any of the postulated problems / biases that have been mentioned by various comments(e.g., #4). It would be a valuable addition to distributing the step 1 algorithms, to distribute these test data sets.