The following shows the Brickwall weights for y(150) with a cutoff of 15, to operate on data vector x(0:300). This is essentially the same as the graph in #156 above, except now t = 150 is exactly in the center of the sample, since there are now 301 points running from t = 0 to t = 300. The weights are exactly symmetrical about t = 150, and are a discrete modification of the sinc weights that apply in continuous, unbounded time.

The next graph shows the same thing, but now for y(250). The weights have just rotated to the right 100 time units, with the weights that have dropped of the right end being added back on the left end. This rotation occurs because the DFT actually thinks the data is periodic, ie x(301), if observed, would be exactly the same as x(0). In other words, the DFT brickwall filter is equivalent to using the symmetrical filter for the center point, and then endpadding the the missing last 100 or whatever points with the first 100 or whatever points.

With stationary noisy data this wouldn’t have an obvious effect. However, with uptrending data it creates a cyclic effect as the left and right ends of the data alternately get positive or negative weights.

When you rotate the weights clear to the end , the endpadding becomes even more noticeable with trending data, because now x(0) and x(1), etc are getting almost as much weight as x(300) and x(299), etc!

This is why the first and last smoothed values are almost equal in the first graph in #136 above.

The Brickwall DFT filter’s peculiar other-end-endpadding makes its end properties even stranger than Mann’s double-flip or Rahmstorf’s extrapolative endpadding. At least they are endpadding with data from the same end of the data series!

There is a categorical difference between constructed indices and measurable reality. In the case of surface-station data, physical processes–known and unknown–determine the time history (signal) of temperature at the measurement site. “Global” indices constructed from highly incomplete geographic coverage of sites are nothing more than statistical samples of the concurrent average temperature troughout the globe. Such averages, even if known quite completely, do not necessarily conform to the laws of physics, because the dry bulb temperature alone is an inadequate measure of prevailing energy levels. And in your example of stock market indices, until the advent of index funds, futures and options, the common saying about index values was “you can’t trade on them.”

Thank you for revealing that HP uses variable weights for a fixed value of the smoothing parameter. This makes it a linear curve-fitting operation (in contradistiction to proper linear filtering) and renders it unsuitable for scientific work, where the precise effect of data smoothing upon the spectral (or autocorrelation) structure needs to be known for a host of purposes.

Graphing them all at once as in Fig. 1 in the post creates a visually interesting pattern, but it is hard to see the individual curves. Unlike the fixed-knot spline, the HP weights have modes that do not oscillate with t.

The four graphs you provide have mechanisms attributed to them. href=”#comment-354904″>Basil (#134)

What assurance can you give that these mechanisms or interpretations are both necessary and sufficient to explain the response for graphs 2-4?

Take the hypothetical case where climatologists were still unaware of ENSO, but were aware of some other unrelated physical processes. Would the graphs be labelled with these other processes? Would they be any more right or any more wrong than your attributions?

What if there exists an undiscovered mechanism that accounts for parts of graphs 2-4?

Well, we know now that Newtonian mechanics does not account for everything we can observe in the physical world, but the fact that there existed undiscovered mechanisms that account for part of what we observe in the physical world doesn’t invalidate the usefulness of classical mechanics. I didn’t suggest that the mechanisms I was attributing to the various graphs were capable of explaining them comprehensively. I was really making a much less grandiose claim than you would have make. It is a truism that climate varies on all time scales. I was just illustrating that with the graphs. That’s all. I wasn’t proposing a general theory of climate.

John S, #176

BTW, your chosen example also shows how incoherent the low-frequency temperature variations are in different regions, with differences that are as large or larger than the variations. Similar disparities are often seen between intra-regional stations. A “global temperature signal” is merely a statistical construct, rather than a ubiquitously measurable reality.

Yes, my chosen example illustrated how different the low frequency variation can be regionally, but within the 9 divisional data sets available to the US, they do break out into three roughly coherent regions, along geographically rational subdivisions (the two coasts, and the continental interior). As for a “global temperature signal,” that it is a statistical construct doesn’t mean that there might not be discernible influences on it. Just like stock market indexes remove the “systematic risk” of individual investments, a global average removes a lot of the variance in regional temperatures, leaving behind something notionally indicative of what is happening globally to terrestrial temperatures.

function [y,W] = hodrickp(x, lambda)
% y = hodrickp(x)
% y is smoothed version of x
% Formula derived in Ekkehart Schlicht, “Estimating the Smoothing Parameter
% in the So-Called Hodrick-Prescott filter”,
% http://epub.ub.uni-muenchen.de/304/1/schlicht-HP-3-DP.pdf
n = length(x);
P = toeplitz([-2 1 zeros(1,n-2)]);
P = P(2:n-1,:); % P generates second differences
W = inv(eye(n) + lambda*P’*P);
y = W*x;
% This routine is inefficient for large n. An interative routine
% based on the Extended Kalman Filter exists that takes
% less memory and is faster.

The weights in W are nearly symmetric near the center of the sample, and become highly skewed towards the ends. They do not respond to the data (x), but do respond to nearness to the endpoints. Furthermore, they are bilateral, in that they reach both forward and backwards (except at the last point, of course). Near the center, the weights have a bell-shaped center, with 2 negative wings, but no oscillations beyond the first overshoot, unlike the “Clark” filter in #156, or even the spline in the post.

HP has been used very little ouside of econometrics and many of us here are trying to fathom its operational principles. If, for any set value of the smoothing parameter, HP indeed uses a fixed linear combination of data uniformly throughout the record, then the frequency response can be calculated from the coefficients (or weights). I’ve not seen this done anywhere. Although one can supposition some correlation structures that would render the Kalman filter no longer data-adaptive, it remains inherently a unilateral (i.e., causal) operation. This raises anew the question of end-treatment. Given the apparent zero-phase response of HP, one expects it to be–by contrast–bilaterally symmetric, with smoothed values provided everywhere by circular treatment of data, similar to IDFT reconstruction. Perhaps you can point us to the econometric literature where these specifics are discussed.