I’m still working on this looking at the behavior of some simulated data and trying to get a better understanding of the impact of measurement errors of a couple of different kinds on what this all looks like. Hope to post all the graphs soon, should be interesting.

]]>* When you are comparing tropical trends, is it correct that the datasets you are using in the first plot (“x” for surface and “y” for troposphere) are the column labeled Trpcs (UAH data from KNMI), the column labeled -20.0/20.0 (RSS data), and the “tropical” HadCRUT3 data available at http://climexp.knmi.nl/data/ihadcrut3_tr.dat, respectively for each data source? If not these, what specifically are you using (full url’s if possible!)?

I got as much data as I could from KNMI. In particular, the surface temperature for all of the datasets is KNMI. It is not the HadCRUT3 data you reference, however. It’s from e.g. http://climexp.knmi.nl/data/icrutem3_hadsst2_0-360E_-20-20N_n.dat . I picked that because I could specify the latitude band, as some of the source data is 20N/S, some is 30N/S.

For the others, I got what I could from KNMI. From memory, that was UAH and not RSS. For the KNMI data, I didn’t use preselected datasets. I called out the exact latitude belt I wanted. The UAH data from UAH just say “Trpcs”, and despite looking, I’ve never seen a latitudinal definition. I used true tropics on that one, 23.5 N/S, as I do when someone doesn’t say, but it could be 30N/S.

* Have you tried running your code with “x” === “y” to confirm you get a straight line at 1.0 for the perfect correlation case?

Yes, but only by accident …

* Your plots here look quite different from the ones you posted over at Lucia’s blog not long before: http://rankexploits.com/musings/2008/who-expects-a-tropical-tropospheric-hot-spot-from-any-and-all-sources-of-warming/#comment-7692 – what did you have to correct to get from those to the above plots?

At lucias, I had attempted to use all of the data to determine the overall amplification (regression line). I regressed all of the datasets of a particular length (e.g. 8 months) at once.

I found that it led to instability at longer time frames. In investigating that, I found a better way to do it. The code I posted takes the regression line of each individual e.g. 8 month time span, and then averages (actually, takes the median of) the results for a final answer.

What I’m seeing that’s different from your graphs so far is:

* the trend ratio for RSS Trop TLT vs. HadCRUT3 (tropics) is greater than 1.50 from 42 months through 253 months, with a peak of 1.564 at 102 months, using what you have claimed as your method. The final decline was only to 1.42 at 359 months.

* Using a simple averaged ratio of the two trends with time (what Jeff Id suggested in #38 here), I see high variability up until about 130 months, then roughly stable ratio of about 1.3 to about 300 months, then declining to about 1.1 at 359 months.

* for UAH Trop T2LT vs. HadCRUT3 (tropics), I’m seeing a ratio greater than 1.50 from 44 months through 144 months, with a broad peak around 1.541 from 78 to 95 months. This declines to 1.16 at 360 months.

* Again using the simple averaged ratio of trends with time, UAH vs HadCRUT3 (tropics) shows very different behavior, with even negative averaged ratios at 129 and 131 months, and for longer periods never exceeding about 0.7, dropping to 0.43 at 360 months.

It’s kinda difficult to say much without seeing the graphs. However, it sounds like they start low, peak, and then drop over time. For me, the datasets are all too crude to posit accurate numbers. I look at the shape, to try to understand the dynamics. You’ll get slightly different results with different assumptions about the latitude bands, and with different data sources. For the model results, I took the true tropics for atmosphere and surface. I used the model surface and atmospheric data from the cited location.

* I then looked at a third “metric” technique, taking every possible pair of measurement dates, and getting the average and standard deviation of satellite temperature differences for each given range of surface temperature differences. I have some more refinements I want to do on this, but in principle under a regime of steadily rising surface temperatures (not far from reality), this magnitude-based measure should give a similar metric as your time-based one, but with some assurance that there is a substantial surface temperature difference involved to drive an associated troposphere temperature change. For RSS tropical TLT the ratio climbs steadily with increasing surface temperature differential, from about 1.4 (1.1 for UAH) when delta T_surface = 0.2 K, up to about 1.7 (1.7 also for UAH) when delta T_surface = 0.7 K.

I’d like to see your results. It’s an interesting idea. It would be easy to modify the R function to calculate that. Just change the matrices. You say you want “some assurance that there is a substantial surface temperature difference involved to drive an associated troposphere temperature change.” That’s what the regression line shows, I thought. Your procedure will show the change in amplification with temperature, which is a thought that hadn’t occurred to me. It’s kinda like the Kendall Tau procedure, which utilizes the same metric (difference between points). Is the increase roughly linear?

Interestingly, just as you find above, this is a decidedly tropical-only phenomenon. Plugging in the global satellite and surface temperatures instead, the ratios stick very close to 1.0 (slightly below 1.0 for UAH).

Absolutely. In fact, the effect is stronger at 20N/S. Take a look at the HadAT2 data for that band. I should actually do a look by narrow (say 5°) latitudinal bands on the satellite data, to see where the action is. Another thing for the unending list.

I’m working on some more refinements but hope to post graphs and everything else shortly – anyway, I’d appreciate clarification from you if I’ve missed something obvious here, because the numbers are turning out not quite to agree with what you have posted here. Thanks.

Please post what you find. As long as the shapes agree, the difference in numbers is likely not too meaningful.

The key to me is that despite using different datasets and different bands, there is a common shape. There is not much amplification at short timescales. It increases for five to ten years, then begins a slow decay. That’s what’s striking about the pattern. And I haven’t been able to reproduce that using red noise … and I’ve tried a pile of them. ARMA(.9,-.3) produces a pattern like that, but only one time in ten.

w.

]]>* When you are comparing tropical trends, is it correct that the datasets you are using in the first plot (“x” for surface and “y” for troposphere) are the column labeled Trpcs (UAH data from KNMI), the column labeled -20.0/20.0 (RSS data), and the “tropical” HadCRUT3 data available at http://climexp.knmi.nl/data/ihadcrut3_tr.dat, respectively for each data source? If not these, what specifically are you using (full url’s if possible!)?

* Have you tried running your code with “x” === “y” to confirm you get a straight line at 1.0 for the perfect correlation case?

* Your plots here look quite different from the ones you posted over at Lucia’s blog not long before: http://rankexploits.com/musings/2008/who-expects-a-tropical-tropospheric-hot-spot-from-any-and-all-sources-of-warming/#comment-7692 – what did you have to correct to get from those to the above plots?

What I’m seeing that’s different from your graphs so far is:

* the trend ratio for RSS Trop TLT vs. HadCRUT3 (tropics) is greater than 1.50 from 42 months through 253 months, with a peak of 1.564 at 102 months, using what you have claimed as your method. The final decline was only to 1.42 at 359 months.

* Using a simple averaged ratio of the two trends with time (what Jeff Id suggested in #38 here), I see high variability up until about 130 months, then roughly stable ratio of about 1.3 to about 300 months, then declining to about 1.1 at 359 months.

* for UAH Trop T2LT vs. HadCRUT3 (tropics), I’m seeing a ratio greater than 1.50 from 44 months through 144 months, with a broad peak around 1.541 from 78 to 95 months. This declines to 1.16 at 360 months.

* Again using the simple averaged ratio of trends with time, UAH vs HadCRUT3 (tropics) shows very different behavior, with even negative averaged ratios at 129 and 131 months, and for longer periods never exceeding about 0.7, dropping to 0.43 at 360 months.

* I then looked at a third “metric” technique, taking every possible pair of measurement dates, and getting the average and standard deviation of satellite temperature differences for each given range of surface temperature differences. I have some more refinements I want to do on this, but in principle under a regime of steadily rising surface temperatures (not far from reality), this magnitude-based measure should give a similar metric as your time-based one, but with some assurance that there is a substantial surface temperature difference involved to drive an associated troposphere temperature change. For RSS tropical TLT the ratio climbs steadily with increasing surface temperature differential, from about 1.4 (1.1 for UAH) when delta T_surface = 0.2 K, up to about 1.7 (1.7 also for UAH) when delta T_surface = 0.7 K.

Interestingly, just as you find above, this is a decidedly tropical-only phenomenon. Plugging in the global satellite and surface temperatures instead, the ratios stick very close to 1.0 (slightly below 1.0 for UAH).

I’m working on some more refinements but hope to post graphs and everything else shortly – anyway, I’d appreciate clarification from you if I’ve missed something obvious here, because the numbers are turning out not quite to agree with what you have posted here. Thanks.

]]>Points taken. I am trying to work on different simplified aspects of why that graph shape crops up, but not on model data or Monte Carlo to such a significant degree, so I seek to try different math approaches and then cogitate. I’m so rusty I have to cogitate a lot.

The informal hypothesis is that there is persistence at various time intervals that might not be all annual; and that knowing this might help put brackets around what can be predicted from what, in terms of time lags or event lags.

]]>I did not detrend my data, nor did I remove outliers. What I removed was the monthly average of the data, which leaves both the outliers and the underlying trend.

In my layperson’s view Geoff’s and Willis E’s metrics are two different animals. Detrending Willis E’s would not make sense, since he is comparing trends. The issue with his metric is the overlap.

Geoff’s metric is comparing standard deviations, not trends, and in my mind the best approach there is to detrend and use a pooled estimator of the standard deviation. Overlap is not issue with Geoff’s metric.

]]>I feel, inturively and from past dabbling, that the best answer lies that way. I do not intend to detrend or to remove outliers. That is not done in geological work for which this was developed.

I did not detrend my data, nor did I remove outliers. What I removed was the monthly average of the data, which leaves both the outliers and the underlying trend.

w.

]]>Kenneth and others earlier, thank you.

I was trying to see whether the graph shape, being different to both Monte Carlo and model, was produced by pure mathematics or by a consequence of use of weather data and its patterns. Therefore, I did the simplest exercise I could devise and the graph shape was present. However I still have not answered the question of graph shape, only possibly eliminated some answers.

Willis, when you start to talk about complexities of overlapping systems, you are edging ever closer to geostatistics. You might even end up there. In the next few days, time permitting, I will run some geostats and report the results. I feel, inturively and from past dabbling, that the best answer lies that way. I do not intend to detrend or to remove outliers. That is not done in geological work for which this was developed.

]]>http://en.wikipedia.org/wiki/Pooled_standard_deviation

The calculated results listed in the table below show, counter to my thoughts in a previous post, that the differences in the standard deviations as one progresses to longer time slices is not driven entirely by whether one uses an average of the standard deviations or the pooled estimate. A large difference, as suggested by Willis E, occurs in the absence of detrending. Even with detrending and using a pooled estimate it can be seen that shorter time pieces give smaller standard deviations. That observation is in line with Willis E’s comments on the Hurst effect.

To better understand the comments made in published papers about the differences in short and long term temperature trends between the tropical surface and troposphere, I think I will use the above approach to look at those two temperature series.

]]>