A post just to let you know that laypeople appreciated the paper also. I would like to see a comment or two from Hu M on the author’s calculations and methods.

Beran [11]

is a good reference, linear trend is fitted to Jones monthly NH 1854-1989 temperatures, 95 % confidence interval for time trend [-0.000158 ... 0.000802] if long memory is taken into account.

But AR4 Table 3.2. says that

The Durbin Watson D-statistic (not shown) for the residuals, after allowing for first-order serial correlation, never indicates significant positive serial correlation.

so Beran must be wrong.

Somewhat relevant text I cited here :

The ACLRM is presumed to be ‘the’ appropriate model, even though the residual autocorrelation could have arisen in numerous alternative ways, one of which is that the error follows an AR(1) process.

ACLRM = Autocorrelation- Corrected Linear Regression Model

Thanks for posting this. It is interesting, though not surprising, especially to those who’ve been paying attention to the LTP literature (Koutsoyiannis in particular).

With respect to the time trend coefficients, the values are positive though they are statistically insignificant, which is in contrast with the results based on I(0) disturbances. The fact that the time trend coefficients are found to be insignificant in our work does not necessarily rule out the hypothesis of warming in the temperatures since this can be a consequence of the sample size used or even misspecification of the model.

If someone has access to the IMPROVE-series, I think it might be worth repeating the experiment with that dataset.

Persistence and Time Trends in the Temperatures in Spain
Luis A. Gil-Alana

Abstract:

This paper deals with the analysis of the temperatures in several locations in Spain during the last 50 years. We focus on the degree of persistence of the series, measured through a fractional differencing parameter. This is crucial to properly estimate the parameters of the time trend coefficients in order to determine the degree of warming in the area. The results indicate that all series are fractionally integrated with orders of integration ranging between 0 and 0.5. Moreover, the time trend coefficients are all positive though they are statistically insignificant, which is in contrast with the results based on nonfractional integration.

So persistence matters after all when deciding whether a trend is significant. The IPCC’s treatment of this issue was a real scam. Remember that after the 2nd review round they had inserted the following paragraph based on reviewer comments (who were drawing from peer-reviewed literature) to which the lead authors had no response except hand-waving and denial:

Determining the statistical significance of a trend line in geophysical data is difficult, and many oversimplified techniques will tend to overstate the significance. Zheng and Basher (1999), Cohn and Lins (2005) and others have used time series methods to show that failure to properly treat the pervasive forms of long-term persistence and autocorrelation in trend residuals can make erroneous detection of trends a typical outcome in climatic data analysis.

Then with no authorization this paragraph was subsequently deleted from the published IPCC report. They can remove the text about persistence from the IPCC report, but they can’t remove the persistence from temperature data.

What was stumping me was how to write out a recursive one-pass formula for the likelihood. Evidently there just isn’t one!

Hosking [1984] presents a remarkably accurate approximation to the FARIMA (arfima) likelihood function that avoids using the full correlation matrix. Although this amounts to just a numerical shortcut — the Cholesky-decomposition approach will work — it is of practical importance: Long-term persistence may require keeping track of thousands to millions of lagged correlations, and the corresponding correlation matrix thus may have millions to trillions of elements.

Hosking, J. (1984), Modeling persistence in hydrological time series using
fractional differencing, Water Resour. Res., 20(12), 1898– 1908.

Wooster OH, which is also in CRU and is relatively unurbanized, shows a distinct downtrend since 1920 or 1930 in the 3220 data. I haven’t tried to test it for any kind of significance yet, though, as I’m still trying to figure out how to get data through 2006 with and without all the adjustments. (It switched from midnight to 0700 and then 0800, which is a relatively small TOB, but I might as well use the official adjustments for this.)

I’m guessing that the downtrend for Wooster will also be insignificant, but that the difference between the two stations since 1948 will have a significant trend, indicating progressive urbanization of the Columbus airport site, and therefore the unsuitability of it and stations like it for CRU.

My point was just that IPCC’s using it to test for AR(2) serial correlation, is a fairly obvious, if naive and invalid, thing to do.
I don’t think their wording on Table 3.2 claims that their DW is testing for LTP. They merely dismiss LPT as raised by Cohn and Lins out of hand, and then use their DW (incorrectly) to test for AR(2). Their wording isn’t terribly clear, however, whence the confusion. They can be legitimately faulted for using DW instead of Durbin’s h (or some other valid test), but they would not be the first to make this mistake.

Durbin (1970) showed that DW gives inconsistent results when applied to the residuals of an AR(1) model, and proposed instead his h statistic. This converts DW into an approximate serial correlation coefficient using rho ~~ (1-DW/2), and then jacks it up by a factor which is the square root of a certain quotient. This has an asymptotic N(0,1) distribution.

Many people are frightened away from Durbin’s h by the fact that the denominator of the quotient can be negative, resulting in an imaginary test statistic that is hard to place on a table of normal critical values. However, as the denominator approaches 0, the statistic approaches infinity, and hence clearly indicates a reject. My view is that a negative quotient should thus be interpreted as giving a statistic that is “beyond infinity”, and therefore an even easier reject.

Inference and testing with “Long Memory” or fractional differencing (where the correlations die out slowly as a power function rather than geometrically) has long baffled me. However, on thinking about it over the past couple of days, I think the following brute force method would work for an ARFIMA equation of the form P(L)(I – L)^d y = Q(L)e: First, figure out the autocorrelation matrix in terms of d and the variance of e. (There are messy equations somewhere for the FI part of this.) Then compute the likelihood as a function of d, var(e), and the P and Q coefficients, and maximize this numerically. The null hypothesis d = 0 is on the boundary of the parameter space, and so the LR for it doesn’t have the usual chi-square distribution, and in any event, if the y’s are the residuals of a first-stage regression (eg of temp on a time trend, CO2, or tree rings), d and/or the AR coefficients will probably be biased downwards. However, all the parameters can be median unbiased along the lines of the Monte Carlo method of Andrews (Econometrica 1993), something I have gotten to work recently in the AR(p) case. This procedure should also give an exact finite sample p-value for a null such as d = 0 (conditional on the median-unbiased estimates of the other coefficients.)

What was stumping me was how to write out a recursive one-pass formula for the likelihood. Evidently there just isn’t one! In fact, the only way to simulate such a process (as required by Andrews’ method) is to use the Cholesky decomposition of the n X n autocorrelation matrix.