Comments on: Replicating Juckes' CVM

By: Kevin

Kevin — Sun, 03 Dec 2006 00:15:20 +0000

#31 SCA also is widely used for TSA. Have you had a look at that?

By: EP

EP — Fri, 01 Dec 2006 17:37:10 +0000

Speaking of stats, does anyone have any pointers to material on the net that covers the types of validation stats discussed here? I’m afraid my Stats 101 course notes are somewhat incomplete…

By: bender

bender — Thu, 30 Nov 2006 21:39:11 +0000

if one was an expert in it and no other languages, I can see why one might

Hmmm, expert at Python vs. not expert at statistical innovation.

I know which approach I would choose, if I were a climatologist in that boat and didn’t want my hide tanned during review.

By: John S

John S — Thu, 30 Nov 2006 21:33:20 +0000

I used to use GAUSS a lot for some econometric work – it is just a relatively high-level programming language and is not specifically a statistics package. But it was more flexible than the, then, alternative RATS. And is still useful in many situations that the current package, EViews, can’t handle. But GAUSS was targeted at the statistical market in that it is optimised for matrix algebra and contained a lot of prewritten optimisation packages – so it was great for maximum likelihood. You use what you know, but some tools are better than others. I, for example, would never contemplate running a panel regression in anything other than a dedicated statistics package – that’s just asking for trouble – but basic regressions can even be run in Excel these days (although this is surely a travesty of a similar order to death by Powerpoint).

By: Nicholas

Nicholas — Thu, 30 Nov 2006 21:03:33 +0000

Steve: It appears to smooth only if “self.time_res” is set, and if so prints out a message. Do you know that time_res is set when this function is called? Does the message get printed? I assume time_res stands for “time residuals”?

Python is, as I understand it, an object oriented/string processing language. You can use it for anything, but I don’t think I would select it for mathematical applications. However, if one was an expert in it and no other languages, I can see why one might.

By: KevinUK

KevinUK — Thu, 30 Nov 2006 18:45:37 +0000

#25 bender

“Is it in the do_hockeystick.py function?”

Are you sure you didn’t mean

“Is it in the do_hockeystick_CENSORED.py function?

KevinUK

By: bender

bender — Thu, 30 Nov 2006 18:17:35 +0000

Juckes appears to have written functions from scratch to do elementary operations

Could have been someone else wrote it for him. Either way, this does shows the, errr, ‘idiocy’ of using Python over R. Why reinvent the wheel?

By: Steve McIntyre

Steve McIntyre — Thu, 30 Nov 2006 17:20:21 +0000

Something else I just noticed in the do_regress code. Juckes smooths the proxy before doing the regression. In R, you can do regressions using the function lm and return all sorts of diagnostics. For anyone planning to do statistical work, it seems like a much more sensible language than Python, where Juckes appears to have written functions from scratch to do elementary operations.

#################################################
## currently only programmed for opt = ‘mbh’
#################################################
## calculates x Y^t (Y Y^t)^{-1}
## where:
## x = self.predictor in the calibration period: i.e. the temperature pcs.
## Y = self.predictand: i.e. the proxies.
###################################################
def regress_coeff( self ):

ac = self.predictor[:,-self.lcal:]
if self.time_res:
print ‘smoothing predictand in inverse regression’
sp = basic_utils.smooth( self.predictand, self.time_res, pad=1 )
self.gg = dot( ac, sp )/dot( sp, sp )
else:
sp = self.predictand
self.gg = dot( ac, transpose( sp ) )/dot( sp, sp )

By: Steve McIntyre

Steve McIntyre — Thu, 30 Nov 2006 17:14:14 +0000

I found something in the do_regress.py that does some flipping:

self.load_predictand( )
if proxy_id == ‘mann_etal1999′:
for k in range(3):
ss = sum( self.predictor[k,-self.lcal:]*self.predictand )
if ss < 0:
self.predictor[k,:] = – self.predictor[k,:]

The term sum( self.predictor[k,-self.lcal:]*self.predictand ) will have the same sign as the correlation of the proxy to the target. But on its face, this only flips for MBH and doesn’t appear to apply to the Union reconstruction. I can’t tell why it would fail to flip the “adjusted” pc in the suffix “pc” reconstruction. Maybe it’s to do with how the range is defined. Martin, are you there? The flipping in the Union CVM is still unexplained – other than it was an “error”. OK, but how did the error get there?
Now that I think of it, it would be interesting to check whether the Chesapeake series was flipped in the Moberg CVM.

By: bender

bender — Thu, 30 Nov 2006 15:13:56 +0000

Is it in the do_hockeystick.py function? Kidding.