Expectation-maximization is explained at http://en.wikipedia.org/wiki/Expectation-maximization_algorithm

Cheers — Pete Tillman
ClimateAudit 101 team
http://climateaudit101.wikispot.org

I’ve tried a couple Octave on Imac seems to load them OK. Octave does use it’s own binary format for saving data though it can save it in matlab format. It doesn’t tell what format it is when loading it just does it.

Mark

Mark

Why he would even put a mean-of-means in for a test is a bit of a puzzler since it is guaranteed to cause nonsense.

Chefen, it has long been my position that The Team does nothing accidentally, though they often do not understand the implications of their flawed methods. I did analyze the differences last summer in a very cursory manner and concluded that the intermediate results were similar, but not identical, though I did not dig deep enough to determine where the differences may lie, nor did I analyze the final answers. Since I no longer have the original code (obviously others do, so there’s no issue) due to a computer malfunction (dead drive), I have never finished the work (I’m also overloaded for other reasons).

Btw, I recently finished a Kalman project myself, chefen.

I’ve seen my share of bad code, I’ve seen terrible code, but trying to understand SDR’s code was — well — an experience

It was unbelievably bad.

Mark

If at first you don’t succeed, hack about with the data analysis until your results are significant.

Or as one associate of mine put it, “They tortured the data until it confessed.”

How much irony would you like?

When Ross McKitrick mistakenly assumed that a program required data in degrees instead of radians, this was picked up by an Australian academic, and copied to multiple blogs and right around the internet as “Ross McKitrick not knowing the difference between radians and degrees”.

What Ross in response did was proper and exemplary – he issued a corrigendum, an impact assessment showing the scale of the error to the final result (as it turned out, it had very little effect), and a new corrected paper the very next issue.

Compare that with Scott Rutherford. He has a clear mistake pointed out to him in data collation which any undergrad would get marked down hard on. He admits that this might have an effect on the results but declines to show what the effect of the error is. The Australian says NOTHING about it. Rutherford just deletes the datafiles and the source code to cover his tracks. He doesn’t issue a Corrigendum. He doesn’t issue an impact analysis. His faulty paper remains in the record with no word from the journal or editor that there is anything wrong with it.

And Rutherford carries on regardless.

I do not understand why some academics should get massive deference for mistakes when if a data analyst made such a mistake in the Toronto stock exchange he’d have to consider changing careers.

I like my irony hard and bitter – what about you?

Steve: Stock analysts make mistakes without having to change careers. I agree that the conduct of Rutherford and Mann compares very unfavorably with Ross’.

UPDATE (June 20, 2007):
Since the original publication of this work, we have revised the method upon discovering a sensitivity to the calibration period.

[snip]

When, as in most studies, data are standardized over the calibration period, however, the fidelity of the reconstructions is diminished when employing ridge regression in the RegEM procedure as in M05 (in particular, amplitudes are potentially underestimated; see Auxiliary Material, section 2).

[snip]

Steve: C’mon, John A, a little more irony.