Hurricane Data Compilation Thread

UKweatherworld has a thread where people don’t comment but simply contribute references on the Holocene Optimum. I’m going to start doing things like this as a way of keeping track of things. People have contributed data but it’s impossible to keep track of. This thread is going to consist only of scripts and data relevant to replicating the hurricane papers. Any general commentary will be deleted. I’ll put some of the data sets into *.txt files so that they can be called. Please provide references to some of the data sets.

Prior discussion http://www.climateaudit.org/p?790, http://www.climateaudit.org/p?803, http://www.climateaudit.org/p?822.

Emanuel, Nature 2005 url Website
Landsea, Nature 2005 url

Webster publications
Webster, P.J., G. J. Holland, J. A. Curry, and H.-R. Chang, 2005: Changes in Tropical Cyclone Number, Duration, and Intensity in a Warming Environment, Science, 309 (5742), 1844-1846. url
Webster P.J., J.A. Curry , J. Liu, and G. J. Holland, 2006: Response to comment on "Changes in tropical cyclone number, duration, and intensity in a warming environment", Science, 311 (5768), 1713c url
Mann and Emanuel, EOS, 2006. url
Curry, J.A., P.J. Webster and G.J. Holland, 2006: Mixing Politics and Science in Testing the Hypothesis That Greenhouse Warming Is Causing a Global Increase in Hurricane Intensity. Bull. Amer. Met. Soc., 87 (8), 1025-1037 url


32 Comments

  1. Steve McIntyre
    Posted Oct 5, 2006 at 3:13 PM | Permalink

    Here’s a script which gives the location of Kaplan’s SST data. It’s in Netcdf format. Fortunately there’s an R package which reads this format. I can’t say that I know my way around the format but I was able to almost instantly recover the relevant data and make average temperatures for the 6 Webster et al basins.

    Script http://data.climateaudit.org/scripts/hurricane/read.kaplan.txt
    Data (average for 6 basins from Jan 1856 to Aug 2006 http://data.climateaudit.org/data/hurricane/basin.average.kaplan.txt

    Here’s a plot of the 6 basin averages

  2. Steve McIntyre
    Posted Oct 5, 2006 at 3:57 PM | Permalink

    Here’s the Atlantic PDI data as collated by Willis including David Smith’s update for 2005-1006, as well as SST up to 2004 (I’ll update this.)
    Data http://data.climateaudit.org/data/hurricane/pdi.atlantic.txt
    See: http://www.climateaudit.org/?p=803#comment-46813
    bender adds: The “PDI” values are from Emanuel (2005) and so are the “SST” values (=”RawHadCRUT_SST”). The SSTs must be anomalies since they are in the range [-1,1]. HadISST are sometimes used in these analyses (e.g. that’s what Landsea has used) and these appear to be actual temperatures as
    they are in the range [26,30]

    Here’s a plot including 2006

  3. Steve McIntyre
    Posted Oct 5, 2006 at 4:20 PM | Permalink

    Here’s a compilation of Basin Averages of MSU data downloaded in Sept 2006
    Data http://data.climateaudit.org/data/hurricane/basin.average.msu.txt
    Script http://data.climateaudit.org/scripts/hurricane/make.msu.average.txt

    Here’s a plot. The similarity between basins in the satellite data is remarkable. I’ll post up similar data for the CRU and it’s nothing like this.

  4. Pat Frank
    Posted Oct 5, 2006 at 8:50 PM | Permalink

    Well, all the Kaplan SST data sets look very normal in a Histogram plot, and so do the residuals of a linear fit (unsurprisingly). I tested the NATL, NIO, and SWP sets for autocorrelation, and they all show ~0.9 at lag 1 (p

  5. bender
    Posted Oct 5, 2006 at 8:52 PM | Permalink

    tripped up by the ol’ &lt

  6. Pat Frank
    Posted Oct 5, 2006 at 9:06 PM | Permalink

    Darn. :-) I’ll do the analysis again tomorrow. It just showed deep auto- and intercorrelation in the Kaplan SST’s, even across data sets representing physically distant locales. If that’s not interesting, let me know.

  7. Steve McIntyre
    Posted Oct 5, 2006 at 10:44 PM | Permalink

    No posts on this thread without contributing a script or data !!! You may post discussion points on one of the other hurricane threads. This message will self-destruct.

  8. Steve McIntyre
    Posted Oct 5, 2006 at 11:02 PM | Permalink

    Here are the basin averages using CRU (a fairly recent version, but not the most recent version).

    Data http://data.climateaudit.org/data/hurricana/basin.average.hadcrut2.txt

  9. David Smith
    Posted Oct 7, 2006 at 10:50 AM | Permalink

    Here is a link to a program that creates timeseries of many atmospheric variables, as well as of sea surface temperature.

    Here is a link to a program that creates correlations of atmospheric variables.

    Here is a link to the US National Hurricane Center’s hurricane archives.

  10. Steve McIntyre
    Posted Oct 7, 2006 at 11:49 AM | Permalink

    Here is the script for bender’s calculations. http://data.climateaudit.org/data/hurricane/hurricane_R.txt

    Here are data files:
    Landfall http://data.climateaudit.org/data/hurricane/landfall.txt
    Emanuel PDI http://data.climateaudit.org/data/hurricane/emanualPDI.txt

    http://data.climateaudit.org/data/hurricane/allstorms.txt

  11. Steve McIntyre
    Posted Oct 7, 2006 at 11:56 AM | Permalink

    Here is description of PDI index by David Smith
    http://data.climateaudit.org/data/hurricane/Power Dissipation Index.doc

  12. Steve McIntyre
    Posted Oct 7, 2006 at 12:58 PM | Permalink

    Webster et al in their Response to Comment (Science 311, March 2006) stated that Chan Science 2006 and Emanuel 2005 used the following data sets to construct their PDI:

    http://www.weather.unisys.com/hurricane/index.html

    which provides information for individual storms in 6 basins for the following periods:
    ATL 1851-2005
    EPAC – 1949-2005
    WPAC 1945-2005
    SPAC only 2000-2005 as far as I can tell so far
    SIO 1949-2003
    NIO 1945-2003

    Emanuel 2005 says:

    Owing to changes in measuring and reporting practices since systematic observations of tropical cyclones began in the mid-1940s, there are systematic biases in reported tropical cyclone wind speeds that must be accounted for in analysing trends. The sources of these biases and corrections made to account for them are described in Supplementary Methods.

    Emanuel’s SI describes the adjustment but does not present the unadjested data. Landsea 2006 presents different results for Atlantic PDI. Is this data archived anywhere?

    What collations of the data are available other than what Willis obtained in his email from Emanuel?

  13. Willis Eschenbach
    Posted Oct 7, 2006 at 1:36 PM | Permalink

    Steve, Emanuel has data on hurricane tracks in NetCDF here, but I haven’t had time to look at it yet.

    w.

  14. Steve McIntyre
    Posted Oct 8, 2006 at 8:45 AM | Permalink

    Willis’ calculation of Atlantic PDI is here http://homepage.mac.com/williseschenbach/.Public/Atlantic_Hurricane_PDI.xls

  15. L. David Cooke
    Posted Oct 8, 2006 at 1:12 PM | Permalink

    Steve;

    Here are a few of the links I contributed to UKweatherworld just searching around NASA. The last link though is not a technical paper; however, it provides a pretty good description of the tropical geology of SE USA from about 7K years ago. It also has some strong references WRT geological and population characteristics and these are reinforced by strong; but, aged studies. If these are of use feel free to add if not it is not an issue.

    Dave Cooke

    http://climate.gsfc.nasa.gov/publications/fulltext/cp-2-79-2006.pdf

    http://earthobservatory.nasa.gov/Study/BorealMigration/boreal_migration3.html

    http://globalchange.nasa.gov/KeywordSearch/Metadata.do?Portal=GCMD&KeywordPath=&EntryId=NOAA_NCDC_PALEO_2005-015&MetadataView=Brief&MetadataType=0&homepg

    http://globalchange.nasa.gov/KeywordSearch/Keywords.do?Portal=GCMD&KeywordPath=Parameters%7CPALEOCLIMATE&MetadataType=0&lbnode=gcmd3a

    http://igloo.gsfc.nasa.gov/wais/abstracts06/Ackert.pdf

    http://quest.nasa.gov/projects/spacewardbound/docs/sdarticle.pdf

    http://igloo.gsfc.nasa.gov/wais/pastmeetings/abstracts04/Das.htm

    http://igloo.gsfc.nasa.gov/wais/pastmeetings/abstracts99/das.html

    http://neptune.gsfc.nasa.gov/csb/personnel/pdf/Parkinson/2003_Vaughan_Etal_Antarctic.pdf

    http://search.nasa.gov/search/search?as_sitesearch=igloo.gsfc.nasa.gov/wais/pastmeetings/PPT03&access=p&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=nasa_production&actionType=searchIndex&q=Holocene+Climate&numgm=5&site=nasa_collection&oe=UTF-8&proxystylesheet=nasa_production&ip=209.170.95.133

    http://search.nasa.gov/search/search?as_sitesearch=igloo.gsfc.nasa.gov/wais/pastmeetings/abstracts05&access=p&output=xml_no_dtd&sort=date%3AD%3AL%3Ad1&ie=UTF-8&client=nasa_production&actionType=searchIndex&q=Holocene+Climate&numgm=5&site=nasa_collection&oe=UTF-8&proxystylesheet=nasa_production&ip=209.170.95.133

    http://nai.arc.nasa.gov/ec/ec/Ulu_revised80105_3amend_2.pdf

    http://www-ss.ksc.nasa.gov/exploration%20park/Reference%20Library/Dynamac%20EIS%20Final%20pdf/APPENDIX_K_final_Jan2004.pdf

  16. L. David Cooke
    Posted Oct 8, 2006 at 1:31 PM | Permalink

    Steve,

    Sorry, my bad you were looking to post Hurricane data.
    My humble apologies…

    Dave Cooke

  17. L. David Cooke
    Posted Oct 8, 2006 at 3:41 PM | Permalink

    Steve;

    I would like to submit these links as sources of data that can help explain recent water vapor research that participates in hurricane formation. I did my best to make sure to include links that should terminate in professional studies from a .gov site.

    http://ams.confex.com/ams/pdfpapers/83490.pdf

    http://www.eol.ucar.edu/projects/refractt/ document link

    http://cosmic-io.cosmic.ucar.edu/cdaac/doc/index.html

    http://camex.nsstc.nasa.gov/camex4/CAMEX-5_White_Paper.pdf

    http://www.timed.jhuapl.edu/WWW/science/scienceHighlights/index.php#SunCoupling

    Dave Cooke

  18. Wolfgang Flamme
    Posted Oct 8, 2006 at 4:35 PM | Permalink

    (pls delete)

    #3 ->should be …/script*s*/…
    #10 ->bender’s calculations should move: …/*scripts*/hurricane/hurricane_R.txt

    Steve, your scripts turn out to be a valuable resource for R-novices like myself … wish I could browse that folder.

    Thank you!

  19. L. David Cooke
    Posted Oct 8, 2006 at 10:32 PM | Permalink

    REF #16

    Steve;

    Please correct the 3rd attached link from my list for the COSMIC system, it should have been: http://www.cosmic.ucar.edu/~braunj/ The microwave measurement referenced under this link is specifically: http://www.cosmic.ucar.edu/~braunj/front_range/mwrp.html

    Note that this is specific to the Denver CO area; however, the characteristics on 7/26, 8/14, 9/21-22, 10/1 2006 seem to be indicitive of an unusual characteristic of a dual direction of freedom from a single phenomon. (It has to be related to the depth of measeurement penetration rather then be a dual vector of change from a single phenomon.)

    Thanks,
    Dave Cooke

  20. Posted Oct 9, 2006 at 7:08 PM | Permalink

    I’m not sure how relevant this is to the Hurricane research papers you are examining, however it is sometimes worthwhile using completely independent data sets to make comparisons, so here are links to the complete Australian Bureau of Meteorology (BoM) Tropical Cyclone Best Track database which covers from 1906 to 2005:
    ftp://ftp.bom.gov.au/anon2/home/ncc/cyclone/cyclones_newformat.zip (350 KB ZIP file)
    and the corrections and amendments notes:
    ftp://ftp.bom.gov.au/anon2/home/ncc/cyclone/amends.txt

    The ZIP file contains the full database as an ASCII text file, and also contains a text file explaining the format.

    The information is in lines of text for 6 hourly data that includes the BoM season and cyclone number, cyclone name, data and time, lat and lon, and various meteorological factors including central pressure, maximum winds, wind radii, etc.. A spreadsheet can easily be used to separate out whatever parts you may be interested in.

    Researchers should note that the BoM uses the WMO standard 10-min-av winds instead of the 1-min-av winds used in the US, and this issue will need to be accounted for if comparing to say US best track data for the same cyclones – this can also be an issue when using independent data for other basins as well.

    Further to this, those unfamiliar with the BoM need to know that when making direct comparisons, much of the modern US best track material is largely satellite based, while in addition to satellite data the BoM uses a wide range of material from both land based instrumentation including radar (where appropriate) and shipping reports, and that extensive post-storm analysis is done for many landfalling systems which may all be taken into account in ‘best track’ data.

    The total number of cyclones in any season contained in the US and BoM data sets can be different due to both the amount and type of data taken into account, and different definitions of what a cyclone is – the BoM uses 10-min-av winds greater than 33 knots in all 4 quadrants, while the US uses 1-min-av winds greater than 33 knots in any quadrant.

    The cyclone intensity information can vary significantly between the BoM and US data – in most cases the BoM data is more accurate due to both local experience and the sheer volume of data used in the analysis.

    The full BoM database was undergoing a quality control revision in 1999-2000, however I understand the process is still incomplete (due to funding cuts, if my memory serves me correctly).

    I hope this is of use to some of you.

  21. bender
    Posted Oct 9, 2006 at 10:59 PM | Permalink

    Steve M, See corrections in #18. [Pls delete this post]

  22. Steve McIntyre
    Posted Oct 10, 2006 at 10:07 AM | Permalink

    Judith, can you explain some nomenclature to me in this hurricane data base http://www.npmoc.navy.mil/jtwc/best_tracks/shindex.html

    In the following years 1986, 1997, 1998 and 1999, there are out of sequence storm numbers 1986 has storms 1 to 33 and then 80; similarly for the other three years. Do you know why this is?

  23. Posted Oct 10, 2006 at 9:27 PM | Permalink

    Re: #22 – Steve, any JTWC 80+ numbers in best tracks data are those systems that were not identified as storms in real time, but were later found to be storms when re-analysing the data.

    This happens from time to time with marginal systems, mainly due to the pressures of getting information out at regular intervals with only a small number of mets working at any particular time, as the poor satellite imagery presentation of many maginal systems combined with lack of other supporting data means it can take a pro-met many hours of careful analysis of multiple satellite images before making a judgement that it is in fact a storm, and time may not be available to do this process properly until after the season when the best tracks are being put together.

    Most marginal systems that fall between the cracks in real time are a long way from inhabited areas where other data can help making such a judgement, and in real time they are usually being tracked as ‘invest’ systems that have potential to develop and are assigned 90+ numbers, so if anyone in a boat or plane is intending to be in the vicinity they can usually get enough info to see that the weather could deteriorate in the area.

  24. Steve McIntyre
    Posted Oct 11, 2006 at 2:08 PM | Permalink

    Details and script for collating hurricane and storm track data is here http://www.climateaudit.org/?p=855.

  25. Judith Curry
    Posted Oct 11, 2006 at 2:34 PM | Permalink

    Re the hurricane data quality issue. I am becoming increasingly convinced that the data outside the NATL is problematic. I suggest a two pronged approach. Prong 1: use exactly the same data as Emanuel to assess what inferences can actually be made from the time series of SST and PDI in NATL and WPAC from 1949-2005 (and ignore data quality issues, just focus on auditing the statistical methods). Prong 2: use the NATL data record (HURDAT) since 1851 to see what can be determined from the longer time series with varying levels of uncertainty as you go back in time. If you agree with this approach, then the data issues become: making sure you have the same data as used by Emanuel, and then sorting through the NATL data quality issues. sorting through multiple datasets, many of dubious quality, could be huge time sink and not really get us anywhere. Lets have some discussion about what data sets we are going to look at, before digging into the various warts associated with a number of different data sets.

  26. Steve McIntyre
    Posted Oct 11, 2006 at 2:58 PM | Permalink

    #25. I’m not planning to get into hurricante track QC issues – I don’t have the time; I’m already spending more time on this than I should without previously finishing some proxy papers. I’m just going to work through some of the calculations and see if I can repeat them.

  27. Steve McIntyre
    Posted Oct 15, 2006 at 11:14 AM | Permalink

    Judith Curry has archived track data as used here http://www.eas.gatech.edu/research/data.htm. Excellent.

    A discussion of the topic is here http://www.wunderground.com/education/webster.asp

  28. Steve McIntyre
    Posted Oct 16, 2006 at 4:45 PM | Permalink

    There is extensive information on individual West Pacific storms from old annual reports of the Joint Typhoon operation, now online. An example is here https://metocph.nmci.navy.mil/jtwc/atcr/1959atcr/pdf/section5.pdf. There’s a lot of data on individual storm tracks that could be easily digitized with a little clerical work.

  29. David Smith
    Posted Nov 18, 2006 at 11:14 AM | Permalink

    A good general reference on meteorology, including tropical weather, is
    here .

    It is slightly technical but well-written and understandable.

  30. David Smith
    Posted Nov 22, 2006 at 8:54 PM | Permalink

    A well-written article on tropical atmospheric circulation is given here . These Chinese researchers studied Hadley cells and associated features. Nice cross-sections of the ITCZ and circulation cells.

    My opinion is that some of the mystery about climate shifts will disappear as we learn more about the behavior of these massive heat removal systems. The Southern and Northern circulations have the same general features but differ in ways (see diagrams) that might affect their ability to remove heat. Perhaps at times one hemisphere’s Hadleys and Ferrels do a better job of removing heat than does the other.

    As a side note, if I remember correctly, Tim Ball mentioned that the GCMs do a rather poor job of modeling these important circulations.

  31. David Smith
    Posted Nov 27, 2006 at 8:46 PM | Permalink

    A paper on Hadley Cell and tropical/subtropical circulation changes since 1950 is given here .

  32. dtngwyfr
    Posted Sep 19, 2010 at 11:27 PM | Permalink

    Sweet site , I hadn’t noticed climateaudit.org before till my friend told me about it.
    Keep up the great work! I will be contributing more at climateaudit.org

    Looking forward to some good new friends here at climateaudit.org
    :-)

Follow

Get every new post delivered to your Inbox.

Join 3,381 other followers

%d bloggers like this: