A further advantage of using monthly differences is that the incidence of missing values is proportionately lower, in that one or two missing months do not mean sacrificing the whole year’s data. For many Arctic and Antarctic data sets this is of considerable practical importance, as many will have noticed.

It will be pretty obvious that the main, large scale, conclusions from Annual and Monthly Difference data will be closely similar, but the latter turn out to be (in general) far more interesting, to me anyway.

I’ve posted in other threads, just occasionally, regarding the widespread – indeed almost universal – practice of fitting simple linear models to climate data. The advantage of adopting this type of underlying model (assumption?) is that it is very simple. My strong opinion is that it is over-simple, since virtually every plot of climate time series data makes it abundantly clear that climate does not behave linearly on almost any time scale. Why not use the observations themselves to help in the choice of a model that is much more feasible and more plausible? Well, perhaps no-one ever got fired for fitting a linear model! This does not mean that it is in any sense an optimum model.

The technique I use, and which I strongly recommend some people here to try, is first to form the monthly differences, by whatever method comes most naturally. These, by definition, have a mean zero. Now, form their cumulative sum by simply summing the MDs successively. Any missing value yields a missing value for the cusum, and is simply ignored in subsequent processing.

The resulting data set will be grossly autocorrelated, but this, for present purposes, is immaterial. We are not interested in the intimate statistics of the cusum but merely its general form.

Now plot the cumulative sum on the time axis. In very many cases the plot will be striking. What you may find is that the cusum plots as what appears to be a collection of roughly straight segments of varying duration which terminate in an abrupt change of slope which heralds another roughly straight segment. The pattern may be repeated several times on different scales. However, in many data sets there is a “grand scale” linear section that may endure for even a hundred years, indicating a very stable long term temperature regime, with several brief excursions in both directions.

To see these things for yourself I would advise initially looking at data for the North West Atlantic (Greenland/Iceland) for which there are several individual sites available. There is also a “consolidated” set published by Vinther et al in 2007, which goes back to the 18th century. If you use this one you will see a most spectacular cusum plot with a very pronounced change point at the end of 1922 (September) of about 2 C. Yes 2 degrees C, occurring in the space of a month or so. Prior to that date a remarkably stable regime existed, with a very slight but significant rise. After the “event” the climate was again stable at a higher temperature for around 60 years. The conventional view of the data is that there was a marked change in temperatures over the first part of the 20th century, although no-one, as far as I am aware, has noticed that the temperature increase was actually a step change.

The cusum has /no/ predictive properties. Indeed, the typical stepwise nature of climate change seems to me to indicate that reliable area or regional forecasting is impossible. Vinther’s data up to Sept 1922 gives absolutely no hint of a change. Just fit trend lines to the data partitioned at that date to assure yourself that the contemporary observer/analyst would have had zero expectation of a change until it occurred.

Many other data sets exhibit similar phenomena, usually masked by the normal climate noise until uncovered by old-fashioned industrial quality control techniques. I could write endlessly on this topic!

If you could email me I’d be able to provide uch more on this.

Robin

P.S. My first “Reply” seems to have been lost somewhere. I’ve had to re-write this stuff – rather tedious.

Meanwhile, all these details are disgressing pretty far from Roman’s topic of the validity of Shuman’s figures as cited by Steig. Let’s move further discussion over to the Serial Correlation thread.

Using the dwtest(lmtest) in R on the AWS annual reconstruction 1957-2006, I obtain a DW stat = 1.4182 and a p= 0.01165 with every trial. I used the default with the “pan” algorithm (sample size less than 100). Iterations from 10 to 100 did not change the DW stat or the p value.

When I used an alternative normal approximation with dwtest(lmtest) I got a p-value = 0.01216 and the DW stat remained at 1.4182. I will try the annual TIR when I recover from having grandkids over for the week. Thanks for the warning on metlab since a man of my age cannot afford to lose any more teeth – implants are darned expensive. It shall be MATLAB for ever and a day.

Thank you, Hu. Please don’t forget a reminder when it’s posted. I’ve dug up some interesting early numbers on satellites as well. Grist for the mill.

Contacting people in some USA agencies is now getting difficult, even old buddies, because they fear for their futures if they open up.

Using the dwtest(lmtest) in R on the AWS annual reconstruction 1957-2006, I obtain a DW stat = 1.4182 and a p= 0.01165 with every trial. I used the default with the “pan” algorithm (sample size less than 100). The iterations were 15.

Something must be wrong with one of the routines, since MATLAB gave me p = .01510 for the same problem. What are you getting for TIR?

MATLAB’s routine doesn’t mention iterations. Perhaps this is somehow related to the discrepancy, but I have no idea how this PAN algorithm works.

Last year, I tried printing out the orginal Durbin and Watson 1950 and 1951 Biometrika articles to learn where their critical values come from, but got blown away. I see now that both the numerator and denominator have exact chi-square distributions (provided D&W’s otherwise peculiar formula is used), but evidently these are not independent, else the ratio would just have an F distribution. The distribution of the ratio may be related to the arcane Wishart distribution which generalized chi-square to non-independent contributions. It arises also in Brown’s paper on Calibration which UC has called to our attention.

BTW, Jim Durbin visited OSU ten years ago or so, and gave a very stimulating series of lectures on Kalman filtering, which served as the basis for his subsequent book on that topic, and got me excited about using it for some problems I was working on. Quite a long and productive career!

The R routine dwtest (in lmtest) sounds like it uses the same non-random PAN algorithm as MATLAB’s dwtest, but it can’t be exactly the same, since the MATLAB version defaults to the N(0, 1/T) approximation to the distribution of r1 above T = 400, while R’s version defaults to this above T = 100. (Either will let you override the default, however.)

The numbers we are getting for the DW for annual averages of Steig’s AWS series are off in the 5th significant digit (1.4182 vs 1.4189). This is a little odd, but of no practical consequence. I just took the file as was, and didn’t round any intermediate results. Computing DW itself (if not its p-value) is straightforward, and should be replicable to machine precision.

I believe we think/know that that test is the same as the one in Metlab.

Isn’t Met’lab where they produce that stuff that makes your teeth fall out??

Using the dwtest(lmtest) in R on the AWS annual reconstruction 1957-2006, I obtain a DW stat = 1.4182 and a p= 0.01165 with every trial. I used the default with the “pan” algorithm (sample size less than 100). The iterations were 15.

Can you match these with R’s dwtest? If you rerun R’s durbin.watson, do you get the same results each time, or just close?

For several trials with annual AWS reconstruction in R using the durbin.watson (car) test I obtain 1.4182 for the DW stat every time, but the p changes with each trial: 0.020(T1),0.022(T2),0.014(T3) and 0.024(T4).

I need to try the other DWtest to which you linked me. I believe we think/know that that test is the same as the one in Metlab.

Lol, I thought I was seeing things.