New Hurricane Data Archive

On Aug 26, 2006, Judith Curry made the following comment about cyclone data, mentioning that re-processing had been done back to 1983:

The tropical cyclone data really is rather a mess, the NATL is definitely the most reliable, so I am focusing on that data set (with all its warts) until the global satellite data is reprocessed and reanalyzed. This has already been done back to 1983 (paper in review), they should be able to go back to 1977 with alot of hard work, then before 1977 it will be a bit dodgy since it is not clear what kind of shape some of that data is actually in. The problem is that until very recently, people have not been using the hurricane data as a climate data record (mainly used for regional damage estimates etc), so there hasn’t been much incentive until recently to try to get this data in shape.

An enormous ftp archive on all hurricanes from 1983 to date is now online (the directory ftp://eclipse.ncdc.noaa.gov/pub/isccp/b1/hurr which hosts the actual data is dated Sept 18, 2006 so we’re keeping you up-to-date).

The host page is here ; with the ftp location here. The new data set is descibed here . To give you an idea of the size of the archive, the zip file for an individual basin for an individual year range up to 678 MB in size. So there has been no stinting on providing raw data.

Right now, it’s an imposing data set, but certainly more data than I’m interested in or can handle. I don’t know whether there are plans to provide summary data (the Best Tracks data in total is much less than 1 MB in size.)

18 Comments

  1. Gary
    Posted Oct 23, 2006 at 7:23 AM | Permalink

    Let the games begin, although an audit of the data surely is the first step. Something that huge is bound to have a typo or two.

  2. Paul Penrose
    Posted Oct 23, 2006 at 7:51 AM | Permalink

    The vast bulk of it is probably satelite imagery.

  3. Wolfgang Flamme
    Posted Oct 23, 2006 at 9:44 AM | Permalink

    Looks like pure netCDF collections to me…

    That’ll show these arrogant amateur wannabe scientsts ….!

  4. Ken Fritsch
    Posted Oct 23, 2006 at 9:50 AM | Permalink

    Re: #1

    Let the games begin, although an audit of the data surely is the first step. Something that huge is bound to have a typo or two.

    From the explanation of the reanalysis of the data we have:

    Since the B1 data include 1.4 TB in more than 250,000 files, a subset is needed to 102 facilitate TC research.

    ..This new data record has been recently applied to the reanalysis of global trends 137 in tropical cyclone activity. The accuracy of recently documented trends [Emanuel, 2005; 138 Webster, et al., 2005] was in question based on the heterogeneity of the existing tropical 139 cyclone records [Landsea, 2005]. To address this, [Kossin, et al., 2006] applied the 140 homogeneous satellite data to the estimation of TC intensity and formed a new intensity 141 record that was used to reanalyze the trends.

    Abstract from Kossin, et al. 2006 states in part:

    These sharp changes cannot be attributed to errors in the hurricane data base. The end result has been a substantial, 100-year trend leading to related increases of over 0.7oC in SST and over 100% in tropical cyclone and hurricane numbers. Superimposed on the evolving tropical cyclone and hurricane climatology is a completely independent variation in the proportions of major and minor hurricanes (compared to all named storms). This has no distinguishable net trend, is associated with concomitant variations in the proportion of tropical and subtropical hurricane developments, and may arise largely from internal oscillations of the climate system.

    In summary I would conclude that any auditing of this reanalysis without the aid of a cadre of graduate students just became neigh on to impossible. If I have the correct Kossin et al., 2006 then I would also guess that the data (reanalyzed) has been already used to confirm that an SST increase of 0.7 degrees centigrade over the past 100 years has caused a doubling of hurricanes and tropical cyclones. On the way, apparently, is a paper to connect the increased numbers of hurricanes and TCs directly to AGW.

    Is the full Kossin paper available at this time for the general public to read?

  5. Steve McIntyre
    Posted Oct 23, 2006 at 10:08 AM | Permalink

    Kossin recently stated here

    To address this issue, we used historical satellite data to construct a more consistent global data record for the 23-year period 1983–2005, and we compared the new data to the existing records that are being used to measure these behaviors and trends. We found that the upward trend in the Atlantic and downward trend in the East Pacific are well-supported by the new data, but the trends everywhere else (Western and Southern Pacific, Northern and Southern Indian Oceans) are significantly inflated by poor data quality. Since SST is increasing in all basins, our results challenge the paradigm that increasing SST alone will cause a concomitant increase in hurricane activity, and underscore the need for research toward better physical understanding of the relationship between climate and hurricane activity.AMS Environmental Science Seminar Series

    Holland of Webster Curry et al at the same location had the following abstract:

    Hurricane variations in the North Atlantic region contain a mix of short and long-term variability superimposed on a distinct trend. The short term variability arises from interannual changes associated with large-scale circulation changes, such as the El Nino. Long-period variations in named storms and hurricanes over the past century are distinguished by relatively stable regimes separated by sharp transitions in the 1930s and 1990s. Each regime has seen 50% more cyclones and hurricanes than the previous one and each is closely associated with a distinct increase in eastern Atlantic sea surface temperatures. These sharp changes cannot be attributed to errors in the hurricane data base. The end result has been a substantial, 100-year trend leading to related increases of over 0.7oC in SST and over 100% in tropical cyclone and hurricane numbers. Superimposed on the evolving tropical cyclone and hurricane climatology is a completely independent variation in the proportions of major and minor hurricanes (compared to all named storms). This has no distinguishable net trend, is associated with concomitant variations in the proportion of tropical and subtropical hurricane developments, and may arise largely from internal oscillations of the climate system. The period of enhanced major hurricane activity during 1945-1964 arose entirely from this oscillation. However, while there is no trend in the proportion of major hurricanes, the increasing cyclone numbers has lead to a distinct trend in the number of major hurricanes. Other presentations will show that the change in ocean temperatures is largely due to greenhouse warming, which leads us the compelling conclusion that the overall trend in named storms, hurricanes and major hurricanes is also substantially influenced by greenhouse warming.

    Have you got the two abstracts conflated?

  6. Ken Fritsch
    Posted Oct 23, 2006 at 10:41 AM | Permalink

    Have you got the two abstracts conflated?

    I absolutely blew it in attributing what was abstracted from Holland for the 10/20/2006 AMS seminar to Kossin and your presentations are the correct versions for each. These abstracts would appear to put Holland and Kossin in nearly direct opposition in interpreting past data.

    The follow-up should be how can the contrasting views of Holland and Kossin be better understood and definitely removing my previous post (it is misleading and will become a constant reminder of my embarassing error).

  7. Jonathan Schafer
    Posted Oct 23, 2006 at 2:38 PM | Permalink

    #6,

    No need to be reminded…You’ve “moved on” from that previous post, lol…

  8. Steve McIntyre
    Posted Oct 23, 2006 at 9:07 PM | Permalink

    I wrote to Kossin as follows:

    Dear Dr Kossin,
    I congratulate you on the archiving of the enormous new hurricane data base. Are there any plans to produce summary information in a MB-range more similar to the existing Best Tracks information?

    I received the following answer which someone might be able to explain for me:

    Dear Steve,

    The data we constructed was designed for long-term trend analyses and as
    such we maximized temporal consistency in lieu of absolute accuracy of
    the 4x-daily fixes. So our new data is not ideal as a surrogate or
    replacement of existing best track, but serves as a complement. We could
    call it a “consistent track”.

    That being said, we do plan on making the data available after our paper
    has completed the review process.

    Thanks for your interest.

    With kind regards,
    Jim Kossin

  9. Posted Oct 24, 2006 at 1:31 AM | Permalink

    My guess is that he wasn’t sure what you meant by “MB-range.” You have to be careful which abbreviations you use with meteorologists; at first I thought you meant millibars (mb). ;)

  10. David Smith
    Posted Oct 24, 2006 at 5:06 AM | Permalink

    Re #8 I think he’s saying that he smoothed the storm tracks. He saw the phrase “existing best tracks” and assumed that your query is mainly about the paths the storms took.

    Hurricanes tend to bounce and jump around and the centers are not always easy to find. In reanalysis, the analyzers use the best information available and develop a best-track for the records.

    For some reason, he wants smooth tracks. I cannot guess why that would help whatever he’s doing with the data.

    His answer took 53 words and was written in Academic Lite, but I guess it sounds better than, “I smoothed the data”.

  11. Gary
    Posted Oct 24, 2006 at 6:59 AM | Permalink

    Re #8: In my work I constantly get requests for data and usually end up playing 20 Questions to find out what the requesters really are after. It’s much better when they tell me what they want to use the data for and I can trim away the non-relevant stuff. Steve, explain what you want to do and ask how likely the data in present form will meet your criteria for quality control. For people willing to share data, this courtesy will encourage them; for those with something to hide, it will smoke them out. Either way you gain some knowledge and don’t inadvertantly put someone off who otherwise could be helpful.

  12. Ken Fritsch
    Posted Oct 24, 2006 at 9:17 AM | Permalink

    Re: #7

    No need to be reminded…You’ve “moved on” from that previous post, lol…

    Jonathan, thanks for reminding me that I have moved on. I declare that I have officially moved on, but not before I serve penance by detailing the differences in the views of Greg Holland and Jim Kossin.

  13. Roger Pielke Jr.
    Posted Oct 24, 2006 at 2:00 PM | Permalink

    FYI related to this dicussion:

    http://www.ametsoc.org/atmospolicy/documents/Kossin102006.pdf

    All presentations:

    http://www.ametsoc.org/atmospolicy/environmentalsssarchives.html

  14. bender
    Posted Oct 24, 2006 at 3:09 PM | Permalink

    Thanks for the link RPJ. Spurious trends. That’s why we concern ourselves with ergodicity. Spurious correlations. That’s why we concern ourselves with mechanistic modeling. This hurricane trend stuff is crap.

  15. Ken Fritsch
    Posted Oct 25, 2006 at 12:16 PM | Permalink

    Re: #13

    Thanks to RPJ’s links to Kossin’s and Holland’s views on TS intensities and numbers, the underlying data validity and what it means vis a vis SST and global warming, my serving penance has become easier — providing my following summaries are reasonably accurate.

    J. Kossin shows how trends in TS intensities can be inflated by using what he describes as data with inconsistencies. He shows after reanalyzing TS intensity data from around the globe and making it “homogenous” that, while the Atlantic basin continues to show an upward trend, the globe as a whole does not. He asks the question why the Atlantic is acting so differently when the SSTs have increased in all areas analyzed and suggests that a more complete view can be obtained through “better physical understanding”.

    G. Holland confines his arguments and observations to the Atlantic basin and shows dramatically increasing numbers of tropical cyclones (TC) in that area over the past 100 years (he shows data for the numbers going back over 150 years). He further claims that in this basin and over this time period that SST leads TC increases and explains over 60% of the variance in TC numbers. Holland sees no trend over this time period of the proportion of major hurricanes to TC, but points out that the number of major hurricanes shows the same dramatic increasing trend as the number of TC. Finally Holland finishes with the following statement: “The strong relationship between increases in storm and hurricane numbers and increases in SSTs leads to the inescapable conclusion that the majority of current hurricane activity is a direct result of greenhouse warming.”

    I know which view/approach I prefer here and I doubt much that it will be the one that the popular media will be quoting.

  16. bender
    Posted Oct 25, 2006 at 2:41 PM | Permalink

    Ergodicity in a chaotic terawatt heat engine. Maybe if the climate game were to be replayed it would be the Atlantic basin that would be inhaling and one of the others that would be exhaling.

    Trends or transients?

  17. Ken Fritsch
    Posted Oct 25, 2006 at 4:24 PM | Permalink

    Maybe if the climate game were to be replayed it would be the Atlantic basin that would be inhaling and one of the others that would be exhaling.

    Possibly in the Kossin world, but unlikely in the Holland version. Noticed that Holland is not saying that the “current hurricane activity in the Atlantic basin is a direct result of greehouse warming” but rather “current hurricane activity is a direct result of greenhouse warming” and all based on a presentation confined to Atlantic data. Holland’s concluding statement can be quoted directly by the popular media with no alterations required.

  18. bender
    Posted Oct 25, 2006 at 5:16 PM | Permalink

    transients, not trends

Follow

Get every new post delivered to your Inbox.

Join 3,251 other followers

%d bloggers like this: