Why does CRU have a confidentiality agreement with Germany?

Today we ponder why many National Meteorological Services (NMSs), including the NMS of Germany, have provided data to CRU under confidentiality agreements which supposedly prevent Phil Jones from disclosing the identity of these stations.

The issue came up recently as one of Jones’ reasons for not providing a list of stations used in HadCRU3. As reported here , CRU said that these were governed by confidentiality agreements:

The remaining 2% of data that is not in the websites consists of data CRU has collected from National Met Services (NMSs) in many countries of the world. In gaining access to these NMS data, we have signed agreements with many NMSs not to pass on the raw station data, but the NMSs concerned are happy for us to use the data in our gridding, and these station data are included in our gridded products, which are available from the CRU web site. These NMS-supplied data may only form a very small percentage of the database, but we have to respect their wishes and therefore this information would be exempt from disclosure under FOIA pursuant to s.41. The World Meteorological Organization has a list of all NMSs.

Doug Keenan, a CA reader was intrigued with this, and wrote to Phil Jones, asking him to provide a list of countries with which CRU had confidentiality agreements that prevented CRU providing data or disclosing station identity. The correspondence is at Doug’s website .

On May 25, 2007, Doug wrote to Phil Jones as follows:

According to Steve McIntyre, some of your raw station data was obtained from National Met Services who asked you to not disclose that data. Is this correct? If so, will you send me a list of the countries with which CRU/UEA has such non-disclosure agreements? Or, if it is easier, direct me to an appropriate person to ask for this?

On May 31, 2007, Phil Jones replied as follows:

Dear Doug,
I have done some searching in files – all from the period 1990-1998. This is the time when we were in contact with a number of NMSs. We have also got datasets from fellow scientists and other institutes around the world. All supplied data (eventually and sometimes at cost), but we were asked not to pass on the raw data to third parties, but we could use the data to develop products (our datasets) and use the data in scientific papers. It is likely that some of the NMSs and Institutes have changed their policies now – and that the people we were corresponding with (all by regular mail or fax) are no longer there or are in different sections. The lists below don’t refer to all the stations within these countries, nor to all periods, but to some of the data for some of the time.

The NMSs

Germany, Bahrain, Oman, Algeria, Japan, Slovakia and Syria

Scientists/Institutes (data for these countries)

Mali, India, Pakistan, Poland, Indonesia, Democratic Republic of the Congo (was Zaire), Sudan and some Caribbean Islands.

These are the only ones I can find evidence for. I’m sure there were a few others during the 1980s, but we have moved buildings twice since 1980.

Not sure how you will use this data.

Phil Jones

By the way, the World Meteorological Organization is one of the co-sponsors of IPCC and has adopted the following data availability principle:

As a fundamental principle of the World Meteorological Organization (WMO), and in consonance with the expanding requirements for its scientific and technical expertise, WMO commits itself to broadening and enhancing the free and unrestricted* international exchange of meteorological and related data and products;

So can someone explain to me why Germany, which seems to be a strong IPCC proponent, would place confidentiality restrictions on the temperature records supplied to CRU? OR, for that matter, India, the home of IPCC President Pachauri? Or Japan? Or Poland?

I wonder why Jones didn’t list the Caribbean islands that he has confidentiality agreements with. Was this just casualness in his response or is it that he only remembers that he has confidentiality agreements with “some Caribbean islands” and isn’t sure which ones.

I’m also not sure what Jones’ position is with respect to the confidentiality agreements signed in the old buildings that he doesn’t have information on or recall who they were with, but thinks may exist. Is he suggesting that his inability to remember the contracts or locate evidence of the agreements is a reason for not identifying station data i.e. he might be breaching a confidentiality agreement somewhere that he’s forgotten about? The mind boggles. Or does the present list photograph the exemptions that he’s relying on?

If we’re dealing with the present list, then we’re talking about 14 named countries and “some Caribbean islands”. Maybe CA readers living in any of the listed countries could write to their NMS (national meteorological service) and ask them to send notice to Phil Jones expressly waiving any confidentiality clauses with Phil Jones. I can’t think of any valid reason why any member of IPCC should at the same time prevent CRU from identifying the stations used in their analyses or providing the data to third parties. If Jones thinks that the policies in these countries may have changed, he should have asked them to waive confidentiality long ago. But he hasn’t done so and there’s no evidence that he plans to do so. If Jones’ claims about such confidentiality agreements are correct, then such practices by Germany, Japan, India and others should be categorically condemned not just at climateaudit, but elsewhere.

95 Comments

  1. woodentop
    Posted May 31, 2007 at 5:15 PM | Permalink

    Could it be a hangover from the origins of (some) national meteorology services as branches of the military? That’s not to excuse data withholding now.

  2. Reid
    Posted May 31, 2007 at 5:18 PM | Permalink

    Steve asks “The issue came up recently as one of Jones’ excuses for not providing a list of stations used in HadCRU3… So can someone explain to me why Germany, which seems to be a strong IPCC proponent, would place confidentiality restrictions on the temperature records supplied to CRU? OR, for that matter, India, the home of IPCC President Pachauri? Or Japan? Or Poland?”

    They don’t want people like Anthony Watts investigating their UHI corrupted data. Just like the IPCC doesn’t want anyone viewing the 2,500 expert scientist commentary.

  3. John Nicklin
    Posted May 31, 2007 at 5:46 PM | Permalink

    Sounds just like the Church of Scientology not allowing its guiding principles of faith to be published.

    The records are their secret stuff, paid for by governments. If the wrong people get their hands on it just think what havok could ensue. CA is definitely the wrong people, anyone who searches for thruth is the wrong people.

    If you wrote a book about all of this cloak and dagger stuff, it would have to be classed as fiction, nobody would believe its fact.

  4. Posted May 31, 2007 at 5:48 PM | Permalink

    Re #1: Ooh, that seems like a good guess! If some fraction of the sites are at military installations, the data reveals where the installations are. Which some countries would automatically consider sensitive information, especially during the Cold War in the days before Google Earth. Ask ‘em again today and you’d likely get a different answer.

  5. Posted May 31, 2007 at 5:51 PM | Permalink

    Could someone give a summary of the whole issue of the Jones/Hansen withholding of data? Exactly what data is being withheld and what data is not being withheld? I see lots of plots of surface data, so this is apparently not the data that is being withheld.

  6. Jim Johnson
    Posted May 31, 2007 at 6:14 PM | Permalink

    While were giving summaries … how about list and consise description of the datasets that comprise the ‘instrumental record(s)’.

    How many estimates of ‘measured global temperature’ are in use, (GISS, HAD …?) and what are they made of? (How are the HCNs related? CRU?)

  7. Jim O'Toole
    Posted May 31, 2007 at 6:57 PM | Permalink

    Similarly to #5 and #6,

    Here in the U.S., as I’m sure is the case in many parts of the world, I doubt that many representatives or senators are familiar with the specific literature that the IPCC relies on, but doubtlessly many have heard of the IPCC’s assessments. Likely they are of the opinion that reproducibility and a cast iron paper trail are a matter of course in the production of these assessments. However, if these representatives are familiar with the lack of record accessibility and stonewalling that Steve has run up against, they haven’t done much about it. Why? Probably because Stephen McIntyre is only one man.

    I would like to write a letter to my representative and senators asking them to pursue the issue of data/methods production and accessibility as it relates to the IPCC and the bodies it relies on, and to bring this up to the committees or agencies within their scope of influence. To this end, I would provide them with a list of the unavailable methods/data you desire, any reasons for the unavailability, the studies that rely on them, and how the IPCC employs them in their findings for their assessments. Something along the lines of: A is not available, which is relied upon by B in its findings, and the IPCC used in section C to state D. I’m a pretty regular reader here, but I don’t recall ever having seen such an organized listing on the blog. Steve, if there isn’t, in the interest of exactness and familiarity, would you please post one? You’re probably sick of going over it again, but I would pass on any replies to you. If any action is taken in tracking the information down, a prior accompanying history of who-said-what-was-where-to-Steve would help as well so that the same doesn’t happen to their staffers.

    Thank you,
    Jim

  8. Steve McIntyre
    Posted May 31, 2007 at 7:14 PM | Permalink

    #5. Yes, there’s lots of data, as evidenced by the plots. But we don’t know what sites were used in the CRU temperature calculation, which is more widely used than GISS, and we don’t know which version of these sites and what impact, if any, CRU adjustments have made to this data.

  9. Steve McIntyre
    Posted May 31, 2007 at 7:17 PM | Permalink

    A partial list of data requested and not provided is discussed here http://www.climateaudit.org/?p=819 – follow link to letter. I’ll make up a page on this when I get a change.

  10. Posted May 31, 2007 at 7:55 PM | Permalink

    Steve:
    When you do your page could you include references to authors and papers used by the IPCC reports so a direct link to them can be seen. I have been writing to politicians here in NZ for years with no response but am quite happy to write to them ALL regarding the unavailability of data etc. I really don’t think most of them have a clue and just toe the line in ignorance. However, just one politician could light a fire that might just keep on burning. As they say you don’t eat an elephant in one bite.

  11. Jonathan Schafer
    Posted May 31, 2007 at 8:58 PM | Permalink

    #1,

    Anything is possible, but I would doubt it. Otherwise, why would they hand the data over to Jones in the first place, confidentiality agreement or not. That kind of thing usually happens by bureaucratic blunder, forgetting to redact some confidential information or accidentally posting confidential government documents on a publicly accessible web site.

  12. Edouard
    Posted Jun 1, 2007 at 1:55 AM | Permalink

    Hello,

    Here you can find some data of European stations:

    http://www.wetteronline.de/framesets/rueckblick/rueckblick_download.htm

    You must enter the name of the town here “Rückblick-Download für PLZ/Ort:” and push the button on ten right.

    http://www.wetteronline.de/cgi-bin/regframe?3&LANG=de&WMO=07149&PRG=klimavar&ZUGRIFF=Download

    In the left window you can choose the kind of data you need (highest temperature, wind, rain etc…)

    You can alos get the data as Excel sheet:

    On the last link you must klick the link “datendwonload Paris/Orly”. It should be en Excel-sheet, but I don’t know how it works :-(

    Ok, you must pay for it. So it is useless for you. But the graphics are free ;-)

    Best regards

    Edouard

  13. Edouard
    Posted Jun 1, 2007 at 2:21 AM | Permalink

    Heureka,

    Here are some station data available (ascii), even in english ;-)

    http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/index.htm

    It seems to be free!

    Best regards

    Edouard

  14. Hans Erren
    Posted Jun 1, 2007 at 3:40 AM | Permalink

    WMO resolution 40 full text

    http://www.nws.noaa.gov/im/wmor40.htm

    Members should also provide the additional data and products which are required to sustain WMO Programmes at the global, regional, and national levels and, further, as agreed, to assist other Members in the provision of meteorological services in their countries. While increasing the volume of data and products available to all Members by providing these additional data and products, it is understood that WMO Members may be justified in placing conditions on their re-export for commercial purposes outside of the receiving country or group of countries forming a single economic group, for reasons such as national laws or costs of production;

    Jones is hiding behind this paragraph.

  15. Steve Milesworthy
    Posted Jun 1, 2007 at 4:33 AM | Permalink

    #1

    Some meteorological operations are still owned by the military, who are enormously paranoid, and whose default answer to any request is “No”. Typically they are also massively bureaucratic. Scientists and people who want to do real work do the minimum they can to get what they need, and certainly won’t be interested in keeping paperwork.

    The sort of freedom of information rights in the US have come to Europe only recently. Recently, (possibly still now?), independent european weather companies got their european data via the US (the europeans don’t publish it, but share it with the US. But the US then go and publish it).

  16. MarkW
    Posted Jun 1, 2007 at 4:57 AM | Permalink

    A general principle of science is that any study that can’t be replicated, is an invalid study.
    That principle still applies if the study can’t be replicated because the author refuses to release the data behind the study.
    Regardless of the reason for that refusal.

  17. Steve Milesworthy
    Posted Jun 1, 2007 at 6:34 AM | Permalink

    #16

    If the author has no valid reason for releasing the data, then yes the study deserves to be questioned.

    But in the real world data is sometimes protected. Breaking that protection opens one up to being sued. It doesn’t stop someone else going to the source of the data and purchasing it themselves. So the study could be replicated.

    Data is sometimes hard to retrieve – climate databases are so huge even the scientists who own it have trouble getting access to what they want because of cost and technical limitations; scientists make pragmatic decisions about what to look after and what to allow gather dust; tapes fail; tape robots drop tapes and run them over…

  18. Jeff Norman
    Posted Jun 1, 2007 at 6:40 AM | Permalink

    ??? Caribbean Islands depend upon tourism and are concerned that potential tourists might get ahold of their temperature records and make decisions not favourable to potential host islands ???

    Cannot
    Understand
    Bureaucrat’s
    Availability?

  19. izarrenhautsa
    Posted Jun 1, 2007 at 6:56 AM | Permalink

    Re (14)

    I wouldn’t say Jones is hiding behind WMO resolution 40. I think that there exists a clear and definitive difference in the way meteorological data are handled in Europe and US, and probably readers from US are not able to grasp that. It is a fact that those working with European NMSs face this problem each day. Perhaps you like it or you don’t like it, but it is a real problem in Europe (as far as I know, at least in Germany and Spain it works that way). You can easily get soundings from the server at the university of wyoming, but not from our local NWS, at least, not data useful for a forecast. You can get soundings from day-3 (day-5) after a LONG bureacratic process which includes signing a contract with the NWS with specific commitment for NO dissemination of the raw information to third parties.

    I can run a mesoscale model fed by BCs from GFS (US) which are available in Internet and I can make a forecast of weather for tomorrow, but I will never be able to do so using the output from the ECMWF global model, since I will not be allowed to get such data by my national WS.

    It is a fact. Ask your colleagues in Europe, I am not lying.

  20. Hans Erren
    Posted Jun 1, 2007 at 8:11 AM | Permalink

    re 19:

    A few years ago I requested the homogenised historic data from KMI Uccle Belgium, I received a reply that I could get the data for a licence fee of approx EUR 200, but I wasn’t allowed to disseminate the raw data to third parties.

    Last year I requested the station data for Hohenheim Stuttgart, I received the data under the restriction that I wasn’t allowed to disseminate the raw data to third parties.

    This is wat resolution 40 is all about.

  21. Steve McIntyre
    Posted Jun 1, 2007 at 8:19 AM | Permalink

    #19,20. Well, that may be what the NMSs are doing, but that doesn’t mean that these are appropriate policies when climate change is a policy issue.

    IPCC should have reported on this situation and spoken out against it. This is outrageous.

  22. MarkW
    Posted Jun 1, 2007 at 8:27 AM | Permalink

    #17,

    If the author can’t release the data, then the study should not be released.
    Any study that can’t be replicated is inherently faulty and should not be relied upon.

    Period.

  23. Peter Hartley
    Posted Jun 1, 2007 at 8:42 AM | Permalink

    #19, 20 This looks to me like the weather services in these countries are protecting their monopolies on weather forecasting. If the raw data were released, private firms could get into the weather forecasting business in competition with the government outfits. I doubt that it is any conspiracy to prevent people from getting data needed to assess claims of climate change. I agree with Steve in #21, however, that the situation is unacceptable when climate change has become such a critical policy issue. Is there any chance of using EU competition laws to challenge the weather forecasting monoplies of the national weather services? Might some of the private weather forecasting firms in the US like to take on the government bodies in Europe to make a market for themselves?

  24. Jaye
    Posted Jun 1, 2007 at 8:54 AM | Permalink

    Scientists and people who want to do real work do the minimum they can to get what they need, and certainly won’t be interested in keeping paperwork.

    Only an amateur would say something like that. If your business depends on data, then you archive so you can reproduce stuff later, do regression testing, etc. Do you think Boeing has thrown away any wind tunnel data from any test on the F-22 or 787? What kind of configuration management do you think people who do “real work” have to maintain?

    A Few/Some/Many scientist in academia do not typically do “real work” they play around with stuff and write papers. Which is appropriate, imo. This iterative process involves several groups all playing around with related stuff, write papers, argue, test each other results and then finally come up with something useful. The problem with climate science as it relates to AGW is that certain groups want to use preliminary results, with respect to this process, with which to obligate trillions of dollars in questionable activities.

    If a military aircraft was built like AGW is being “built”, then the test pilots would be pulling the ejection cord early in the flight.

  25. Jaye
    Posted Jun 1, 2007 at 8:58 AM | Permalink

    And another thing, if a grant for some sort of research covers collecting some sort of data then the data belongs to the granting agency (unless some contractual language says otherwise) not the grantee.

  26. Steve McIntyre
    Posted Jun 1, 2007 at 9:16 AM | Permalink

    Scientists and people who want to do real work do the minimum they can to get what they need, and certainly won’t be interested in keeping paperwork.

    Most people who do “real work” keep paperwork.

  27. Steve Milesworthy
    Posted Jun 1, 2007 at 9:17 AM | Permalink

    #24

    I was referring to the bureaucracy of intellectual property rights and contracts, and I didn’t say it was ideal, only that it wasn’t surprising.
    Where I work, the configuration management is very good, and 10 year-old model runs could be reproduced reasonably easily (older ones are harder due to changes in compilers and computers).

  28. Gerald Machnee
    Posted Jun 1, 2007 at 9:43 AM | Permalink

    Environment Canada places high hourly rates (about 90 dollars per hour) for requests for Severe Weather Reports and Severe Weather Events as well as other readily available material, likely to discourage requests.

  29. Steve Sadlov
    Posted Jun 1, 2007 at 9:47 AM | Permalink

    Meteo data are right up there with ICBM sub SIOPs and secret bunker locations …. I mean, my gosh, if they got into the wrong hands! /s

  30. Jeff Norman
    Posted Jun 1, 2007 at 10:38 AM | Permalink

    Re: #29 Steve Sadlov

    if they got into the wrong hands!

    Exactly, you could change government policy!

  31. MarkW
    Posted Jun 1, 2007 at 11:50 AM | Permalink

    #23,

    Having access to current data would enable competing forecasting firms. Agreed.

    I don’t see how that concern would affect historical data though.

  32. Posted Jun 1, 2007 at 2:21 PM | Permalink

    re 23:
    The Hohenheim dataset was from a University researcher.

    Volker Wulfmeyer and Ingeborg Henning-Müller, 2005, The climate station of the University of Hohenheim: Analysis of Air Temperature and Precipitation Time Series Since 1878 Int. J. Climatol. 26: 113’€”138 (2006) Published online in Wiley InterScience (www.interscience.wiley.com). DOI: 10.1002/joc.1240

    The licence fee for the Uccle data was in 2003 EUR 164,16. I don’t know if in the meantime the policy or fee has changed. I digitised the homogenised annual average of Uccle from a graph in:

    R. Sneyers, 1990, On the statistical analysis of series of observations, Technical note World Meteorological Organization no. 143, Geneva, World Meteorological Organization, 192 pp ISBN 92-63-10415-8

  33. Wolfgang Flamme
    Posted Jun 1, 2007 at 3:39 PM | Permalink

    Here’s a list of free available German station data. Here’s how to obtain it.
    From the 41 german stations listed, 26 seem to have made it into the GISS station list. However, looking at when the series from the Deutsche Wetterdienst started and the period of time that finally was incorporated in the GISS station data, there are remarkable deviations.

    E.g. to begin with the Helgoland island in the list, there is DWD data available since 1952 until present. However, GISS chose to use the 1977-1990 data segment only. Guess what … the rate of warming over the whole period 1951-2006 is almost zero (0.02°C/decade). You might want to crosscheck Helgoland island’s global warming contribution according to GISS by typing ‘Helgoland’ in the input field left to the search button here. It’s 0.12°C/decade – 6 times as much.

  34. Jim Johnson
    Posted Jun 1, 2007 at 11:18 PM | Permalink

    Re #6:

    I wasnt kidding about that request for a concise explaination of the ‘instrumental record(s)’.

    Anyone?

  35. klaus brakebusch
    Posted Jun 2, 2007 at 3:26 AM | Permalink

    #33

    there are mostly only a few years in the datasets.

    More (monthly) is here:

    http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/ausgabe_monatswerte.htm

    Daylies are here:

    http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/ausgabe_tageswerte.htm

    some more monthly sets (Germany and Europe) are here:

    http://www.wetterzentrale.de/klima/index.html

  36. Boris
    Posted Jun 2, 2007 at 7:33 AM | Permalink

    Hans Erren said:

    Jones is hiding behind this paragraph.

    How dare he uphold agreements he has made!

    Anyway, the whole confidentiality thing should be done away with.

  37. Stan Palmer
    Posted Jun 2, 2007 at 9:02 AM | Permalink

    re 36

    How dare he uphold agreements he has made!

    With the political importance of the IPCC in Europe, it would seem likley that a request from it for publiic availability of this information would be immediately granted. How could any government agency resist the politicla power of the AGE lobby.

    Of course the opposite could be quite true as well. If the AGW lobby does not want this information released then there would be thousands of Sir Humphreys in the bureaucracy to prevent it.

  38. Edouard
    Posted Jun 2, 2007 at 10:13 AM | Permalink

    I’ve posted this link for german stations. They are free and I’ve posted them in post in Nr. 13.

    http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/index.htm

    As far as I can see these datasets are complete from 1800 on ….

    Does nobody read the posts???

    Best regards

  39. Tim Ball
    Posted Jun 2, 2007 at 10:40 AM | Permalink

    Maybe I missed it, if so apologies, but do we know which stations and what adjustments are being made by Hansen with the GISS data? Has anyone asked for this information? Has this been made available? Which set did IPCC use, GISS or HadCRU3or some convenient combination? Has anyone questionned how there is a 0.16°C difference between GISS and HadCRU for the first 4 months of this year? This amount is large when compared to Jones claim of 0.6°C increase over approximately 130 years.

  40. Wolfgang Flamme
    Posted Jun 2, 2007 at 1:44 PM | Permalink

    #38

    Edouard, it depends on which link you choose from there – that’s why I pointed to where the most extensive data can be found IMO.

  41. Larry Huldén
    Posted Jun 2, 2007 at 2:17 PM | Permalink

    Edouard says:
    June 2nd, 2007 at 10:13 am

    http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/index.htm

    As far as I can see these datasets are complete from 1800 on ….
    Does nobody read the posts???

    These data on monthly level starts only from 1991 or later. They are useless for the current discussion.

  42. Larry Huldén
    Posted Jun 2, 2007 at 2:20 PM | Permalink

    Edouard says:
    June 2nd, 2007 at 10:13 am
    > http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/index.htm
    >
    > As far as I can see these datasets are complete from 1800 on ….
    > Does nobody read the posts???

    Unfortunately these data starts on monthly level only from 1991 or later.
    They are useless for the current discussion.

  43. Brooks Hurd
    Posted Jun 2, 2007 at 9:30 PM | Permalink

    Edouard:

    Thank you for the link.

    The only data that is pre 1991 is the monthly means from 1961 to 1990.

    This tells us nothing concerning the issues raised in this thread.

  44. Posted Jun 3, 2007 at 12:53 AM | Permalink

    Steve, I am not an scientist in this field, but it’s quit intriguing for me –
    is there any data set containing temeperature series collected from rural sites stations. I have read in
    Christy’s and Spensers’ paper that temperatures on High Sierra stations in CA didn’t rise at all in XX Century..
    What about some network of rural stations worldwide. If we collect those data and they show that
    warming is much smaller than officially reported or even absent, that would be fatal blow in official theory of 0.6 degrees increase and conclusive proof of UHI?

    I am living in Serbia, and we have just a few purely rural stations. On those station there is no
    warming trend in XX Century!

  45. Edouard
    Posted Jun 3, 2007 at 3:25 AM | Permalink

    Hello again,

    I don’t know which data you need, but here you get everything(?):

    http://www.dwd.de/en/FundE/Klima/KLIS/daten/online/nat/index_standardformat.htm

    Noone can say that someone doesn’t want other people to collect the data. If you get diurnal data back to 1800 you can reconstruct all the average data, as far as I understand climate.

    Even if I don’t know, who uses what kind of data, noone can say that german stations keep their data secret.

    Best regards

    Edouard

  46. Ivan
    Posted Jun 3, 2007 at 9:07 AM | Permalink

    I am not interested in German station data, but in worldwide rural station data. I have indicated that
    Some ten or so rural stations in High Sierra in USA show no temperature increase in XX Century. Rural stations on Grenland and Antarctica show no temperature increase in XX Century. Two rural stations in Serbia Rimski Sancevi i Sjenica also don’t show any temperature increase in the XX Century. It is quite interesting isn’t it? I wonder if there is any set of rural station data worldwide, because if we collect some 100 or 200 purely rural or wilderness station data and they show no warming or little warming, that would be reliable signal that official number of 0.6 degrees C of temperature increase in XX century is false, i.e. an artifact of UHI. I just wonder how to obtain more data. That’s my point, not any kind of conspiracy theory.

  47. Posted Jun 3, 2007 at 2:46 PM | Permalink

    re 46:
    I identified Rimski Sancevi as 13168 Novi Sad Rimski Sancevi, Sjenica is not in the list, however
    Serbian forecastrs can be found on the internet.
    Do you have the archived data? I surmise they are not well kept stations.

  48. Ken Fritsch
    Posted Jun 3, 2007 at 4:22 PM | Permalink

    Re: #45

    That latest link gets us back to 1952. You should let Phil Jones know that Germany does not keep its station data secret as it was he who implied that and gave it as a reason and example for not being able to provide raw temperature data on request.

  49. MarkW
    Posted Jun 3, 2007 at 5:43 PM | Permalink

    #46,

    Are there any non-rural stations in Antarctica?

  50. Posted Jun 4, 2007 at 12:05 AM | Permalink

    Not, in my knowledge.

  51. Andrey Levin
    Posted Jun 4, 2007 at 3:33 AM | Permalink

    Ivan:

    For any serious purposes the only reliable source of data for local, regional, or global atmospheric temperatures from 1978 is direct satellite measurements. For period from 1958 onward it is data from weather balloon measurements (in good agreement with satellite measurements from 1978). Unfortunately, for older temperature data we have to rely on vastly imprecise weather stations data. If you want to play (I do) with all these sources, go to

    http://www.co2science.org/scripts/CO2ScienceB2C/data/temperatures/temps.jsp

    There are other sources of conveniently graphed data.

  52. David Smith
    Posted Jun 4, 2007 at 4:59 AM | Permalink

    RE #51 Weather balloon (radiosonde) historical data has been culled and adjusted in recent years. The culled/adjusted balloon data looks a lot like the surface record.

    Those culling and adjusting practices are ripe for an audit.

  53. Posted Jun 4, 2007 at 11:30 AM | Permalink

    How does the culled and adjusted balloon data compare the the MSU data? By the way, what happened to the Spencer-Cristy “MSU Science Homepage” at http://wwwghcc.msfc.nasa.gov/MSU/msusci.html ? I know that there were some differences between the graphs from Spencer-Cristy-UAH and the RSS interpretation of the satellite data.

  54. David Smith
    Posted Jun 4, 2007 at 12:28 PM | Permalink

    Re #53 Steven Milloy maintains a nice webpage showing various temperature plots, located here . It includes both UAH and RSS satellite-derived temperatures.

    The US NCDC report included (at least in parts of 2006) a culled and adjusted thing called the RATPAC radiosonde time series, shown here in comparison with surface temperatures. But, I now (late 2006 onwards) see a University of Washington time series, shown here , which looks more consistent with the satellite-derived temperatures. And, no RATPAC. I will check on this change in the NCDC report, as I had not noticed it before today.

  55. David Smith
    Posted Jun 4, 2007 at 12:34 PM | Permalink

    RE #54 I may need to correct myself – on closer visual examination the two plots look about the same. I’ll plot the radiosonde points vs satellite, same color, so as to get rid of that optical difficulty on the RATPAC plot.

  56. Posted Jun 4, 2007 at 3:27 PM | Permalink

    53
    It is interesting that comparison between the two data sets of surface temperatures and satelite data set from 1979 onward show strong discrepancy, contrary to GCM predictions. Satelite data show 0.13 C trend per decade, while GHCN and HadCRUt show 0.26 and 0.19 C respectively. Isn’t this problem (troposphere should warm even faster than near surface)?

  57. Posted Jun 4, 2007 at 10:01 PM | Permalink

    It is a long established principle that you cannot give third parties data you obtained from others without specific permission. It belongs to the parties who gathered the data. You should, obviously say who you got the data from and the third party is free to ask for it directly.

    On another point. I find it hard to believe that Helgoland data only goes back to 1952

  58. Nicholas
    Posted Jun 4, 2007 at 11:17 PM | Permalink

    It is a long established principle that scientific data is available to anyone for any reason. This is because independent replication of results is a REQUIREMENT of scientific research.

    So, they can choose to follow one of two approaches. The scientific approach, or the one they are taking. Their choice.

  59. Gaudenz Mischol
    Posted Jun 5, 2007 at 12:11 AM | Permalink

    @53

    That’s what I observed to. In my bookmarked internet adress for MSU I only get an 404 error. And I didn’t find another homepage with their graphs of satellite data. Makes me wonder why. I first thougt of an update coming soon, but it’s now more than 4 weeks. Another example of hiding inconvenient data?

  60. Posted Jun 5, 2007 at 12:46 AM | Permalink

    You can find their uptated satelite data here: http://vortex.nsstc.uah.edu/data/msu/t2lt/uahncdc.lt
    Trend is 0.14 still, so nothing had change, large gap remained.

  61. Wolfgang Flamme
    Posted Jun 5, 2007 at 1:05 AM | Permalink

    #57

    Eli, you might be right. However the “Hell go land” people were not allowed to return until 1952 to what was sort of left of the island after operation “Big Bang”…

  62. Rick
    Posted Jun 5, 2007 at 3:38 AM | Permalink

    #57, #58

    Eli is correct. Not all data is necessarily available to anyone for any purpose. For instance, I believe that data recorded by satellite missions is frequently restricted to the PI’s group for a year or so in order that they can publish first. Other data arising from studies funded by commercial organisations may also be restricted. I don’t imagine you would be successful if you contacted a major pharmaceutical company and asked to have all their data on all their scientific studies. University research groups will also often hold on to their data until they can deliver on the science promised in the associated research grant. There would be little point in going through the hassle of writing a research grant, making the measurements and gathering the data if someone could come along at that stage, take your data and beat you to publication. The total data sets in these cases may well become available at a later time, but the need to deliver scientific output (publications) in return for the funding is a major factor.

  63. Nicholas
    Posted Jun 5, 2007 at 4:45 AM | Permalink

    Rick,

    “The total data sets in these cases may well become available at a later time, but the need to deliver scientific output (publications) in return for the funding is a major factor.”

    The problem is you have a contradiction here. You say it “may well become available at a later time”, yet you call it “scientific output”.

    It isn’t scientific output if it can’t be replicated, and how can it be replicated without the relevant data sets being made available? It depends on the type of science of course… but there’s no way to get around the fact that the paper MUST contain enough information for it to be replicated by independent third parties if it’s to be replicated. In the case of statistical studies, that typically means data and either a very good description of what was done, or the code to do it.

    I’ll re-iterate. You can release a scientific statistical paper, or you can withhold the data. You can’t do both.

  64. MarkW
    Posted Jun 5, 2007 at 4:53 AM | Permalink

    I can understand the reluctance to release your data until you are ready to publish.

    That isn’t the issue here.

    The studies have already been published. In some cases years ago.

    Your data should be released concurrent with publishing.

  65. Nicholas
    Posted Jun 5, 2007 at 5:38 AM | Permalink

    Yes, I agree, while I think I personally would share data at any time if asked, I won’t fault others for not doing this.

    However, I will fault people for relying on studies which are not audited or replicated to set important policy, and those who defend the practices which make this auditing and replication difficult.

    The countries which fund the IPCC should insist that it only includes the results of proper scientific inquiry in its reports, and in turn the IPCC should insist that all the papers whose conclusions it relies on are able to be audited, even if they haven’t actually made sure such an audit has been performed. (They probably should do that, too).

  66. Steve Milesworthy
    Posted Jun 5, 2007 at 6:14 AM | Permalink

    #65

    Yes, let’s get 10,000 of the world’s top statisticians to start poring through the thousands of references in the IPCC report. Bureaucrats would love that.

    Similar standards elsewhere would have put a stop to clean air acts, public health improvements, the space race, you name it…all important government policy guided by “poorly-audited” scientific advice.

  67. MarkW
    Posted Jun 5, 2007 at 6:22 AM | Permalink

    Compared to the cost of Kyoto, hiring 10,000 statisticians is nothing.

  68. Stan Palmer
    Posted Jun 5, 2007 at 7:03 AM | Permalink

    re 66

    public health improvements, the space race,

    Public health improvements — a most dramatic case of improvement of public health came with the removal of the handle of the Broad Street water pump in Soho London by Dr. John Snow. This ended a cholera outbreak and taught authorities how to prevent future outbreaks. This was accomplished using systematic statistical techniques. Anyone who thinks that public health is not served by the exacting use of statistics is wrong.

    Space race — after the Apollo 1 accident, the investigation revealed that there were seven (7) transistors, the loss of any one of which would mean the loss of the spacecraft. This was only a small part what was found wrong with the Apollo technology. One could also mention Challenger and Columbia and the loss of life there as well.

    Numerous other missions have been lost due to careless errors in the design of the mission. Hubble is a noted example of this. A very large and very expensive mirror using the most modern technology to grind it most accurately to the wrong shape. Nobody told the learned astronomers to remove the packing washers. How would they have known? Nobody peer-reviewed the packing washers.

  69. Brooks Hurd
    Posted Jun 5, 2007 at 7:12 AM | Permalink

    Re: 66
    What we would like is for the IPCC to demand that authors upon whom the IPCC relies must provide a way for their data to be scrutinized. It the data can be scrutinized, then the study can not be replicated, and the IPCC should not be claiming that such studies are scientifit, since they are not.

  70. Stan Palmer
    Posted Jun 5, 2007 at 7:26 AM | Permalink

    re 66

    The statement

    Similar standards elsewhere would have put a stop to clean air acts, public health improvements, the space race, you name it…all important government policy guided by “poorly-audited” scientific advice.

    is so outrageous that it deserves many more examples. Professional engineers are certified so as to ensure public safety. Building designs are audited to ensure that buildings do not collapse. The 19th century was replete with examples of bridge collapses and boiler explosions before systematic auditing of engineering designs was mandated. Even the 20th century had its examples of disasters caused by inadequate design. The aerial walkway of the Hyatt Regency hotel in Kansas City collapsed in 1980 killing 114 people. The design had the entire weight of the walkway and the people on it supported by a single nut.

    All engineering work is audited to strict standards. It is impossible to sell anything without adherence to numerous performance and safety standards.

    Audits do not inhibit innovation or utility. They enable it.

  71. Steve Milesworthy
    Posted Jun 5, 2007 at 7:53 AM | Permalink

    #68

    If you wish to audit John Snow’s “Mode of communication of cholera” you’ll find it sadly lacking. The users and owners of the Broad Street water pump would have every right to object to removal of its handle as the evidence is anecdotal, often turns out to be wrong, and there is no mention of control studies such as villages with refuse in their water supply who don’t get cholera. There is even an example when a woman gets cholera before the purported source floods her house.

    #70 – bridge collapses, boiler explosions? It’s clear that no modern heavy objects such as new-fangled steam locomotives should have been allowed near bridges, and no boilers should have been built, till the science of metallurgy and knowledge of the properties of steam had been fully determined.

    #68: Challenger, Columbia, Apollo? Would we have even made it into space yet?

    Of course audits are important. But they do inhibit innovation when they try to unreasonably reduce risk, or I might add, they are introduced with the deliberate purpose of inhibiting the innovation.

  72. Stan Palmer
    Posted Jun 5, 2007 at 8:30 AM | Permalink

    re 71

    Snow’s work was not anecdotal but exact and scientific.

    You’ll have to do a much better job supporting your argument. Juvenile sarcasm goes only so far.

  73. Steve Milesworthy
    Posted Jun 5, 2007 at 9:38 AM | Permalink

    #72 the sarcasm is being used ironically. I can’t read any more of Snow’s document because my stomach is churning at the images it inspires :-(

    The (collective) allegations here appear to be that because the data is not open to the world, it is invalid to present the science, and further, the auditing is too limited; and because the auditing is limited, the results must be false. Further, that the scientists are deliberately skewing the results because they know their results will never be audited.

    I think the allegations are wrong and unfair, and any auditor who comes to the table believing these allegations cannot be constructive.

  74. MarkW
    Posted Jun 5, 2007 at 9:55 AM | Permalink

    Is it sarcasm, or simply the inability to read plain english.

    Your characterization of the attitudes towards science of the posters here may buy you accolades at RC, but it doesn’t pass muster at any site that understands and appreciates how science actually works.

    Any study that can’t be replicated, is not science. It’s a personal assertion at best, little more than arm waving at worst.
    That’s not just a concept we’ve invented, it’s been the core of the scientific method for hundreds of years.

    Nobody has said that the results must be false because the authors refuse to release their data and methods, though such a conclusion is not unreasonalble. To make such a claim shows either extreme bad faith, or extreme incompetance at reading on your part.

    As to the attitude that an auditor brings to the table, I take it you have never been through an IRS audit.

  75. Steve Sadlov
    Posted Jun 5, 2007 at 10:27 AM | Permalink

    The audit concept has got your goat Mr. Milesworthy. I’m going to keep picking at that scab. Like Lt. Columbo going after a suspect, your area of sensitivity is revealed.

  76. Steve Milesworthy
    Posted Jun 5, 2007 at 11:00 AM | Permalink

    I do recognise that many contributors to this blog have a genuine scientific interest. My characterisation is based on comments like these ones on this page (italics on the particularly relevant flowery language). If I’ve misinterpreted any one of them I apologies to the particular author, but I’ve not misinterpreted them all:

    Weather balloon…historical data has been culled and adjusted in recent years. The culled/adjusted balloon data looks a lot like the surface record. Those culling and adjusting practices are ripe for an audit.

    If the AGW lobby does not want this information released then there would be thousands of Sir Humphreys in the bureaucracy to prevent it.

    Meteo data are right up there with ICBM sub SIOPs and secret bunker locations. I mean, my gosh, if they got into the wrong hands!

    certain groups want to use preliminary results, with respect to this process, with which to obligate trillions of dollars in questionable activities.

    Jones is hiding behind this [WMO] paragraph.

    Sounds just like the Church of Scientology not allowing its guiding principles of faith to be published.

    As to replication of results, I agree the results should be replicable, but there isn’t a golden right of all to immediate and free access to all data. Jones got his data from someone, so presumably someone else could get the data too and hence the study is replicable (or falsifiable).

    Apart from the bureaucratic hurdles which I doubt the IPCC have any control over whatsoever, what do we do with model data which is huge in extent? Some of the projects I know about are going to great lengths to improve openness and accessibility of large datasets. But if the current datasets are big, what about access to AR5 or AR6 data. Will Steve McIntyre’s inability to access a large global dataset also invalidate these studies?

  77. MarkW
    Posted Jun 5, 2007 at 11:20 AM | Permalink

    #76,

    My you are sensitive.
    I see nothing wrong with the quote you provide.
    There is much to be suspicious of in regards to the way climate data has been manipulated.
    The refusal, usually on the most spurious of grounds to provide access to the data behind the studies is also telling.

    There is not a golden rule granting access to other people’s data.
    Neither is there a golden rule requiring other people to take your work seriously.
    There is a golden rule that if you want to be taken seriously, you WILL release your data.

    Regarding Jones, how can one get the same data, when Jones won’t even reveal which stations he used for his studies.
    Nor will he reveal what adjustments he used to make sure the data he did use had the trends he was looking for.

    Jones is a remarkably poor example for you to use, what with him declaring that he won’t release his data to anyone who’se motivation is finding problems with Jone’s work.

    Model data is not huge in extent. Even a few gigabytes of data is trivial these days.

    Your willingness to excuse, even defend the hiding of data is very telling.

  78. jae
    Posted Jun 5, 2007 at 11:24 AM | Permalink

    Will Steve McIntyre’s inability to access a large global dataset also invalidate these studies?

    No, but without audits, how do we KNOW? You seem to be missing the point here. In science, the hypotheses have to be falsifiable. If studies supporting the hypothesis can’t be replicated, the hypothesis is false. It’s that simple.

  79. Steve Milesworthy
    Posted Jun 5, 2007 at 12:03 PM | Permalink

    #78

    But you are assuming there are no audits, and assuming the data is available to nobody else. (Have to admit I haven’t yet worked out the Jones story yet – there aren’t many links and track-backs to earlier posts here). Science doesn’t progress by everyone repeating each other’s work, it progresses by finding different ways of testing things, and rejecting or refining based on the results.

    #77

    Model data are not small – we’re talking terabytes and petabytes. As I understand it there will inevitably be bandwidth restrictions to raw AR5 model data except between the largest scientific organisations. An awful lot of money is currently being spent on the metadata – so you don’t spend all night downloading the wrong stuff.

  80. jae
    Posted Jun 5, 2007 at 12:15 PM | Permalink

    #79

    Science doesn’t progress by everyone repeating each other’s work,

    Often it does. Think about the cold fusion fiasco. Nobody else could get it to work, so the theory was declared bogus. For a good example of how an attempt at replicating indicated faulty methods, read some of the threads concerning MM’s work on Mann’s famous Hockey Stick (see left pane for links).

  81. Steve McIntyre
    Posted Jun 5, 2007 at 12:18 PM | Permalink

    #79. Raising the spectre of models is irrelevant to the paleoclimate situation where the data sets are small and can be decided on their own merits. I think that it would be an excellent idea for to have an independent engineering-quality analysis of at least one climate model. That would be a big job and require a substantial budget and is well beyond my resources. I do not exclude the possibility that the models may pass with flying colors. If you were building something “small” like a refinery, you’d spend millions on engineering, feasibility studies and third party due diligence, instead of merely quoting and re-quoting little readers-digest version academic articles in Nature and Science.

  82. Steve Sadlov
    Posted Jun 5, 2007 at 12:19 PM | Permalink

    Don’t feed trolls who are attempting to sow doubt and distraction.

  83. MarkW
    Posted Jun 5, 2007 at 1:07 PM | Permalink

    Nobody’s talking about model data.
    We’ve been specifically references climatological data. Temperature data and that kind of stuff.
    Most of those are sub megabyte, and are the type of data that compresses very well. (ASCII strings, repeating within a narrow band.)

  84. MarkW
    Posted Jun 5, 2007 at 1:08 PM | Permalink

    #82,

    That’s what the RC guys say about us.

  85. Stan Palmer
    Posted Jun 5, 2007 at 1:24 PM | Permalink

    re 81

    If you were building something “small” like a refinery, you’d spend millions on engineering, feasibility studies and third party due diligence, instead of merely quoting and re-quoting little readers-digest version academic articles in Nature and Science.

    And the professional engineers who signed off on the design would be doing so under penalty of law. If somehting went wrong, it would not mean that their paper would be rejected by Science; it would be prison.

  86. Stan Palmer
    Posted Jun 5, 2007 at 1:29 PM | Permalink

    re 79

    Model data are not small – we’re talking terabytes and petabytes. As I understand it there will inevitably be bandwidth restrictions to raw AR5 model data except between the largest scientific organisations. An awful lot of money is currently being spent on the metadata – so you don’t spend all night downloading the wrong stuff.

    Before the Internet, large files were placed on tape and sent by courier. This is the so called “sneakernet”. There is no reason why this could not be employed with modern storage technology.

  87. MarkW
    Posted Jun 5, 2007 at 2:35 PM | Permalink

    Commercially available thumb drives hold multi-gigabytes and can be bought for $20-$40.

    Can be mailed a lot cheaper than a tape reel.

  88. Steve Milesworthy
    Posted Jun 6, 2007 at 4:42 AM | Permalink

    Don’t feed trolls who are attempting to sow doubt and distraction.

    Do they have mirrors where you’re from :-) More seriously, the only way of getting a sense of this blog is by engaging fully. As I’ve said elsewhere, I do work with some climate scientists and mostly they’re interested in getting the science done well.

    #85 This means I’ve now been threatened by both the IRS and prison in this blog.

    #86 #87 For info, a 100 year run of a current model is typically 1.5 terabytes. A typical data archive rate of a climate research institute is about 1 terabyte per day. This should quadruple by the time we get to AR5.

  89. MarkW
    Posted Jun 6, 2007 at 5:04 AM | Permalink

    Steve,

    You’ve been told three times now that we are not talking about the output of the climate models.
    How many times do you have to be hit over the head with reality before it sinks in?

  90. Jeff Norman
    Posted Jun 6, 2007 at 5:54 AM | Permalink

    Re:#88 Steve Milesworthy,

    #85 This means I’ve now been threatened by both the IRS and prison in this blog.

    How have you been threatened by (with?) prison? Are you a professional engineer who designs faulty refineries? I would guess no (but one never knows).

    I believe Stan Palmer was demonstrating that some professions have more than just a duty to perform their work in a way that is diligent and auditable. They have a legal obligation. SteveM has made the same point numerous times in referring to assays.

    Several of us who labour under these kinds of obligations are frequently amazed by the apparently haphazard approaches used by those seeking to prosecute the case of anthropogenic global warming/climate change.

    I say apparently haphazard approaches because this is how it appears to us. Attempts to follow down paths of cause and effect following bread crumb trails of data that disappear and reappear without telling a cohesive story lead to frustration and doubt. These things make us skeptical of the allegations being made against modern industrial societies, allegations made with absolutely no accounting of the myriad advantages.

    What is it that makes you believe that AGW/CC is a clear a present danger that needs to be addressed as a priority, a priority that should not be diffused by attempts to clarify the issue?

    In my opinion one of the greatest weaknesses of the IPCC and their assessment reports is the lack of direction for improvements in data collection aimed at improving the knowledge base. For example they could say temperature trend data is missing in X location and 50 years of data here would be beneficial for assessing the impacts of AGW/CC and the measures taken to mitigate these impacts. Imagine a world body actually trying to improve our knowledge of the world.

    This missing direction suggests they believe they have all the data they need to get the job done. Since they do not have all the data they need to close the case against AGW/CC then perhaps this is not really the job they are trying to get done.

  91. Steve Sadlov
    Posted Jun 6, 2007 at 10:23 AM | Permalink

    Don’t feed the troll. Topic focus, topic focus …..

  92. Steve Milesworthy
    Posted Jun 6, 2007 at 11:01 AM | Permalink

    #89
    OK my last word. I do not believe that difficulty in accessing data necessarily invalidates the results presented, which is the allegation made in comments here. If it did, it would invalidate an awful lot of science including model results. (Clearly there is history between t’other Steve M and this Jones person on which I can’t comment due to lack of information).
    #90
    I wasn’t taking offence – I am involved in software engineering of models.

  93. jae
    Posted Jun 6, 2007 at 12:17 PM | Permalink

    88:

    As I’ve said elsewhere, I do work with some climate scientists and mostly they’re interested in getting the science done well.

    For the record, I don’t doubt that this is true. But there are a few famous ones out there that are spoiling things for the majority.

  94. Joe Bowles
    Posted Jun 9, 2007 at 11:03 AM | Permalink

    I am not a scientist, but it seems to me that most of the issues raised on this website fall under first principles:
    1. Establishing that the raw data from which everything else follows is available for scrutiny without any alteration;
    2. Establishing the data used has been collected in a consistent fashion so that the data is in fact proper for use. (The problem with weather stations comes to mind)
    3. Establishment of criteria by which research is either accepted or rejected, i.e. the methodology by which the data is processes is defined, the statistics verified by statisticians, and peer review is extended to the areas of expertise used in the study, not just to other peers in a subspecialty.
    I am an agnostic on global warming. I think it is reasonable to conclude it is happening, but the system is too poorly understood to draw the conclusion that it is man-driven, especially when the alternative hypotheses do not appear to have been adequately researched.
    I do find it hard to understand why we have jumped to building alarmist reports almost daily before we tackle the issues of the quality of our data. I also have a hard time understanding why so many in the climate world work so hard to avoid having independent analysis. If they believe their conclusions, having third party support would strengthen their position. If their position falls, then, it would either rule out poor areas of research or perhaps point to more fruitful areas to pursue.

  95. Steve McIntyre
    Posted Jun 9, 2007 at 11:25 AM | Permalink

    #94. I only partly agree. Obviously I think that the data and methods need to be scrutinized. And I think that the funding agencies should require non-compliant scientists to archive data and methods in a proefessional manner or lose their funding.

    Having said that, as I’ve said many times, I think that it’s reasonable for politicians to proceed on the information that they have, imperfect as it may be. Decision-makers do that all the time. If I had a big policy job, I would not be guided by very minority views but by the views expressed by organizations such as IPCC. But that’s not a reason to closely examine all the steps in the logic. It is surprising just how much hair there tends to be on the individual steps in the logical argument though. OF course, realclimate and their ilk are quick to say that problems in any individual step don’t “matter” because either they’ve “moved on” or more often because of GCM results. An engineering level analysis of GCMs is beyond my resources and there doesn’t seem to be any interest in a properly funded (multi-million dollar) engineering-calibre study of the “best” GCM.

One Trackback

  1. [...] but that has taken another 6 months. In connection with the supposed confidentiality agreements, as reported previously, Doug Keenan asked for the countries with which there were confidentiality agreements that [...]

Follow

Get every new post delivered to your Inbox.

Join 3,299 other followers

%d bloggers like this: