Bernie, your comment (from September 6th), i.e., “. . . all these FIR filters have zeros with real part 1/2. A lovely result – that means almost nothing as far as I know!” prompted my memory of the Riemann Hypothesis, a proof of which is the current Holy Grail in prime-number theory (and much besides): “The nontrivial zeroes of the Riemann zeta function all have real part equal to 1/2.” As I recall, the conjecture applies to “local” Zeta functions. Perhaps your “lovely result” means something after all!

Just sayin’.

FYI: I am a (retired) physicist who dabbles in complex analysis and knows next to nothing about digital-signal filtering.

]]>* even though the general point you are making **is correct** about the length of the impulse response function …

I understand what you are saying, but consider the case where you’re using a shorter-length IIR filter with padding and you cut it off at the original end-point before padding.

I’m pretty sure it is the case that you’ll end up a smaller end effects with the shorter filter, even though the general point you are making about the length of the impulse response function of the filter, compared to what you would get with a traditional FIR filter.

Obviously this is not a general result of IIRs (you can’t use forward backward filtering for example), but because we’re discussing *truncating* the IPR, you can in principle (and I think) in practice end up with a shorter region near the end point influenced by the filter.

Here my initial responses are in { }. Detailed comments below the quotes.

Carrick commented above:

“Why FIRs? They don’t seem appropriate for this application due to the amount of padding you need to do… end-point effects matter here. Recursive (IIR) filters greatly reduce how many taps you have, and reduce the end-point effects.” {False}.

To which UC responded:

“I don’t think so. IIR is just to hide the end-point effects.” {Also False}.

Carrick then replied to UC:

“We are interested in the series near the end-points, and FIR requires an excessive amount of padding for this application. You can reduce the number of taps (the filter length) using IIR filters.

Hence this reduces the number of points near the end-points affected by the smoother. This isn’t a matter of just “hiding the end-point effects”, using IIRs actually reduces it.” {Nope}.

Misconceptions all around here! Two comments in clarification.

(1) An IIR filter needs fewer coefficients (path multipliers) relative to an FIR filter for a comparable sharpness of performance in the frequency domain (narrow low-pass transition band, band-pass width, etc.). But, you will need approximately the SAME significant length of the actual impulse response. (See my graphs at Sept. 7, 11:28 AM). Do not confuse number of coefficients with the length of the impulse response. They ARE the same for FIR but not for IIR of course. This is just Fourier transform stuff (z-transform or Discrete-time Fourier transform actually), and the corresponding “Uncertainty relationship”. If delta-f gets small, delta-t must get larger. No equivocation. The FIR version has a “coefficient” (path multiplier) for each value of the impulse response. The IIR design has fewer coefficients (smaller by a factor of 5 or 10) for a similar performance. See again link below, pages 19-20):

But if you look at the most significant portion of the IIR’s impulse response, it will be about as long as an equally performing FIR. IIR has NO ENDPOINT ADVANTAGE. So – Why are some misled in this regard?

(2) The Malab function “filtfilt” is not restricted to IIR. It is in fact often used for IIR (like Butterworth) because we may want to arrive at linear-phase. So this brings up a second point. A filtering ACHIEVED from IIR using filtfilt may APPEAR to have fewer endpoint problems than FIR. But this is due to the refinements at the conclusion of filtfilt for treating ends. It’s artificial and misleading. Use the full convolution, not “filter” or “filtfilt”.

]]>UC:

I don’t think so. IIR is just to hide the end-point effects

We are interested in the series near the end-points, and FIR requires an excessive amount of padding for this application. You can reduce the number of taps (the filter length) using IIR filters.

Hence this reduces the number of points near the end-points affected by the smoother. This isn’t a matter of just “hiding the end-point effects”, using IIRs actually reduces it.

]]>I have been aware of the influence the detrending curve has on the model derived from the residuals and the following example illustrates it well. As noted previously using the residuals from a linear regression of the GISS global mean from 1880-2013 results in a best fitting model of ARMA(4,0) with no statistically significant cycles in the residuals as estimated from a smoothed power spectrum. Now using residuals from the same GISS series and a 5th order spline smooth to define the trend produces a best fit model ARMA(1,0) and with a smoothed power spectrum showing significant cycles at 3.5, 5, 8 and 60 years. The smoother used in this case to define the deterministic trend was one that fit very well, qualitatively, if not quantitatively, the published estimated effects of GHGs and aerosols on global temperatures.

As an aside a 1000 member simulation of the ARMA(1,0) (ar=0.41) series with a standard deviation of 0.0856 with no trend added yielded 19.4% of 975 possible 25 year periods with a statistically significant trend. The mean absolute value of the trends was 0.61 degrees C per century. The residuals of the ARMA series (white noise with no added trend) had 3.5% of the possible 25 year periods with statistically significant trends with an absolute values of trends of 0.51 degrees C per century. It shows that it is not difficult to select a proxy response that can have 25 year periods of significant and modern era instrumental-like temperature trends from a series containing no deterministic trend but rather only red and white noise.

]]>“Why FIRs? They don’t seem appropriate for this application due to the amount of padding you need to do… end-point effects matter here. Recursive (IIR) filters greatly reduce how many taps you have, and reduce the end-point effects.”

I don’t think so. IIR is just to hide the end-point effects

]]>Thanks Bernie, interesting comment!

]]>