CA Mirror

The performance of the CA server is ridiculously slow. Last March, I considered moving to a wordpress account, but was persuaded that there were so many embedded links to CA and to CA graphics to move. Thus, we continued on using our own server, which has now collapsed under a load much more modest than WUWT volume, which is effortlessly handled by WordPress.

On an interim basis, I’ll post at camirror.wordpress.com (the logical name having been hijacked by someone else.)

I’ve done a post on a very recent refusal of my FOI appeal by the University of East Anglia. It may seem like a minor event given the tumultuous events of the past few days, but take a look at the chronology.

[The blog appears to be ok again this morning, 22 Nov]


42 Comments

  1. mpaul
    Posted Nov 21, 2009 at 1:41 PM | Permalink

    Steve, it breaks my heart to see CA sidelined during this period. As a Typepad blogger, I can tell you that they do a fantastic job dealing with spikes in traffic — as, I image, do WordPress and Google Blogger as well. Blogger has the advantage that you can map your domain (in other words, you can still use climateaudit [dot] org. I think the migration issues can be handled by a savvy web guy.

  2. Jeff Id
    Posted Nov 21, 2009 at 1:59 PM | Permalink

    Steve, Comments are set to moderation on the new blog. I doubt that was intentional. I’ve sent an email but am unsure if you are simply swamped with them.

  3. Arthur Hastings
    Posted Nov 21, 2009 at 2:25 PM | Permalink

    Steve, you can map your domain to a wordpress blog also.

    Do your homework regarding Blogger. I seem to remember that during the 2008 US Presidential campaign, a lot of blogs were constantly being disabled by Google because someone reported them as hate content or being untrue or something. Instead of guilty until proven innocent, they would immediately be turned off for a while only to be turned back on later. Being a controversial site, that wouldn’t be good for you.

  4. Sparkey
    Posted Nov 21, 2009 at 2:26 PM | Permalink

    John Hinderaker of the Powerline Blog has some good analysis of the CRU emails here and here. The second article will be of special interest to CA readers as it reviews the emails on the “team’s” efforts at damage control over the Yamal affair. John’s assessment on the emails can be summed up with this quote:

    The emails show beyond any reasonable doubt that these individuals are engaged in politics, not science.

  5. George Orlov
    Posted Nov 21, 2009 at 2:38 PM | Permalink

    I have used Drupal for a blogging platform, and there are a number of things one can do to make it scale very, very high. You can probably do some of these on WordPress, since it also uses PHP and MySQL.
    If you have not tries these, please do (and ask me if you don’t know how).

    1) PHP OpCache (I use eaccelerator).
    2) MySQL query cache on (other MySQL tuning would be helpful, depending on your loads)
    3) MemCache, block caching, anonymous user caching, etc
    4) Javascript and CSS aggregation/gzipping
    5) lot’s more stuff, depending on your setup.

    If you do migrate, you can maintain all your deep links to keep your pagerank up. That’s just using either the new application or apache mod_redirect.

    Good luck, George

  6. George Orlov
    Posted Nov 21, 2009 at 2:40 PM | Permalink

    Also, check your server logs, to make sure no one is purposely loading your server to make it perform poorly.

  7. Jimmy Haigh
    Posted Nov 22, 2009 at 12:28 AM | Permalink

    Whatever you do, Steve, keep up the good work! Your contribution will soon be seen to have been pivotal in bringing the world back to its senses on this issue.

    I’ll be dropping something in the tip jar later today so you can have a dram on me!

  8. Geoff Sherrington
    Posted Nov 22, 2009 at 2:51 AM | Permalink

    Steve,

    Do you want regulars to privately email a (very) occasional CRU email#/report for you (as I have just done) on the grounds that hard-to-explain actions at the time have now become clearer? Or would you rather select your own meal from the menu? Geoff.

  9. P Gosselin
    Posted Nov 22, 2009 at 4:06 AM | Permalink

    Why not post a thread at WUWT in the meantime. It’s a bummer not getting your commentary as this unfolds.

  10. sean
    Posted Nov 22, 2009 at 4:10 AM | Permalink

    The nature of climate audit is sober reflection, reportage of record. Being away from the centre of this storm is probably desirable. There may still be legal action, and or a leak investigation. If anyone can generate the slightest suspicion you have non-public knowledge about the leak, your emails and computers may be seized.
    You have enough enemies both inside the team, and in several governments for someone to try a little fishing.
    You have, anti -hacking, anti-terrorist, and anti-piracy powers which can be applied, and they are international, and have low standards of suspicion required. I think you will find, that climate comes under national interest , and spying laws may apply.

    If you equipment is seized, I would guess the contents would leak. Encrypting your data would not be a complete solution as in many countries you can be required to decrypt it.

    Now would be a good time to remove from your computers anything you do not need. You could do this as part of your server move.

    Please be clear I am not suggesting you have anything you should not, but I suspect people have talked to you off the record, and may have supplied grey data sets. Even the most minor lack of team spirit would have consequences for them.

  11. P Gosselin
    Posted Nov 22, 2009 at 4:13 AM | Permalink

    I’ve made visits to your tip jar in the past – very modest ones admittedly.
    I’d like to see a tip jar for a legal fund to pursue the faulty works at CRU. This to me has moved well beyond science (was CRU ever about science?) and moved into a legal arena, at least that’s where it ought to be now. Taxpayers, citizens, businesses, scientists, etc. have been damaged financially – some quite substantially.
    As I mentioned at other sites, I’m prepared to tip much more into a LEGAL TIP JAR.

    • AlanB
      Posted Nov 22, 2009 at 5:59 AM | Permalink

      Re: P Gosselin (#11),

      This … has moved well beyond science …into a legal arena, at least that’s where it ought to be now

      Please Steve, Don’t get sidetracked. I know you will Stick to the Science. Your time is precious…

    • Joe Crawford
      Posted Nov 22, 2009 at 8:47 AM | Permalink

      Re: P Gosselin (#11), If anyone knows how to contact him Lord Monckton might have the pull to get the British government to investigate CRU. He has show a bit of interest in the subject reciently.

      Here $50 for the tip jar.

      • Daryl M
        Posted Nov 22, 2009 at 1:07 PM | Permalink

        Re: Joe Crawford (#20),

        If anyone knows how to contact him Lord Monckton might have the pull to get the British government to investigate CRU. He has show a bit of interest in the subject reciently.

        Lord Monckton already knows about this and you can be sure he is more than a “bit” interested. I notified a person I know in the UK about the CRU files on Thursday night and within minutes and he forwarded it to a long list of contacts including MoB. He (my contact) has been very outspoken on AGW, having written well publicised letters to the House of Commons, Royal Society, BBC, etc. People in the UK are all over this issue.

        • Posted Nov 23, 2009 at 3:21 AM | Permalink

          Re: Daryl M (#26),
          Daryl and Joe, Lord Lawson was on the BBC Today Programme this morning (about 7.35 if those in the UK want to look it up on iplayer) calling for an independent inquiry. He was quite fair about the leaked emails – he said he maintained an open mind. He also managed to get in the point about temperatures not rising this decade.

          Re: Kazinski (#28),
          I like this bit,
          “And don’t leave stuff lying around on ftp sites”
          – isn’t that exactly what they did themselves?

        • TAG
          Posted Nov 23, 2009 at 11:24 AM | Permalink

          Re: PaulM (#31),

          http://news.bbc.co.uk/today/hi/today/newsid_8373000/8373594.stm

          The BBC report may be heard at this URL. The specifc URL is at 7:35 about halfway down the page.

  12. TerryS
    Posted Nov 22, 2009 at 5:09 AM | Permalink

    Steve, it seems that you need a better setup. Perhaps you should consider multiple servers instead of just one. I know this costs money so I’ve hit your tip jar with $50 to help with the costs.

  13. MrPete
    Posted Nov 22, 2009 at 7:24 AM | Permalink

    Seems we’re ok again here…

  14. Jeff Id
    Posted Nov 22, 2009 at 7:27 AM | Permalink

    The blog is back. Perhaps the new posts can be copied here again.

  15. Posted Nov 22, 2009 at 8:14 AM | Permalink

    All Your Emails Are Belong To Us

  16. Pistolus
    Posted Nov 22, 2009 at 8:33 AM | Permalink

    Phil Jones was told not to delete emails:

    “I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable!”

    http://www.anelegantchaos.org/cru/emails.php?eid=943&filename=1228922050.txt

  17. Ron Cram
    Posted Nov 22, 2009 at 8:41 AM | Permalink

    Steve, I remember you writing about William Connolley asking where a particular figure came from. It seemed an odd thing to me at the time. Does the CRU email here relate to the same figure?

  18. Pistolus
    Posted Nov 22, 2009 at 8:43 AM | Permalink

    Phil Jones on GISS land surface temps:

    “GISS is inferior – not just because it doesn’t use back data. They also impose some
    urbanization adjustment which is based on population/night lights which I don’t think is very good.”

    http://www.anelegantchaos.org/cru/emails.php?eid=1042&filename=1254850534.txt

  19. Fred
    Posted Nov 22, 2009 at 9:19 AM | Permalink

    And I bet your phone is ringing off the hook with the Globe & Mail, The Star and the CBC all lining up to interview you. Right ?

    Ahhhh sorry Steve, too funny to think the Canadian media want to cover this story. Day 4+ and the only thing I can find is a G&M link to the Revkin piece in the NYT. These giys have been so in the bag for the whole global warming hysteria thingy for so long they’ll look like fools now if they cover the biggest story in decades. Trillions of dollars, an entire global economy and international governance changes all over a false premise and manipulated data.

    Nice of Revkin not to tell everyone that he is a contributor in the emails as well, a contribution that may wonder about his journalistic agenda.

  20. Pistolus
    Posted Nov 22, 2009 at 9:27 AM | Permalink

    Phil Jones explains to Michael Mann how wants to coordinate peer reviews of a paper submitted to Science in order to ensure that it is not published, but he also doesn’t want any fingerprints on the coordination, as that would be unethical/unseemly/counter to how science is supposed to be done:

    “Can I ask you something in CONFIDENCE – don’t email around, especially not to Keith and Tim here. Have you reviewed any papers recently for Science that say that
    MBH98 and MJ03 have underestimated variability in the millennial record – from models or from some low-freq proxy data. Just a yes or no will do. Tim is reviewing them – I want to make sure he takes my comments on board, but he wants to be squeaky clean with discussing them with others. So forget this email when you reply.”

    http://www.anelegantchaos.org/cru/emails.php?eid=402&filename=1077829152.txt

    Presumably the paper they are discussing is Von Storch/Zorita, but could be another

  21. P Gosselin
    Posted Nov 22, 2009 at 9:30 AM | Permalink

    Lord Mockton is well up to date on these issues, and I do hope he uses his considerable influence to get a legal case going.
    Clearly many have been seriously damaged by the CRU and IPCC sheningans. They are all entitled to compensation.
    Steve has done the brunt of the heavy science lifting needed to blow this wide open. Now it’s time to move this to the legal arena. So I prefer hitting a legal warchest. The tip jars have served their purpose, unless of course some explains otherwise.

  22. Pistolus
    Posted Nov 22, 2009 at 9:34 AM | Permalink

    Keith Briffa on Bristlecone pine series:

    “I still believe the “Western US” series and its interpretation in terms of Hemispheric mean temperature is perhaps a “Pandora’s box” that we might open at our peril!”

    http://www.anelegantchaos.org/cru/emails.php?eid=759&filename=1164120712.txt

  23. Paul Coppin
    Posted Nov 22, 2009 at 10:17 AM | Permalink

    Blogger is frightfully slow (I have several blogger and wordpress accounts with nowhere near anybody’s traffic levels). I doubt it can handle your volume. WordPress generally can. You might enquire with WordPress as to whether your site name was highjacked for obfustication. If so, since they seem generally sympathetic, you might be able shake it free. I ran into this problem on Blogger – hijacking – someone monitored a thread I was in where their was a discussion about setting up a blogger account, and logged up the name that same day. They never actually used it and refused all contact. The CA.wp one hasn’t been used since it was set up – get WP to free it up.

    Steve: we’re hoping that John A claimed it in 2007 when we moved servers and considered going to wordpress. well find out soon.

  24. ianl8888
    Posted Nov 22, 2009 at 2:39 PM | Permalink

    In Aus, apart from an uninformed and inaccurate report in the Daily Telegraph ( a gutter tabloid), the mainstream “meeja” have studiously avoided any mention of this

    Unsurprising and quite predictable, since the MSM here have long pushed the AGW barrow and legislation is pending in a heated Parliamentary debate

  25. Kazinski
    Posted Nov 22, 2009 at 3:01 PM | Permalink

    Steve,
    I had a specific question about one of the emails that you are referenced in, at least I assume that MM is McKitrick and McIntyre:

    At 09:41 AM 2/2/2005, Phil Jones wrote:

    Mike [“Michael E. Mann”],
    I presume congratulations are in order – so congrats etc !
    Just sent loads of station data to Scott. Make sure he documents everything better
    this time ! And don’t leave stuff lying around on ftp sites – you never know who is
    trawling them. The two MMs have been after the CRU station data for years. If they ever hear
    there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than
    send to anyone.

    Is this the same data, that Jones just “sent load of” that he claimed in August of this year had been lost, back in the 80’s? The same data that he threatens to delete rather that release it to you? That seems awfully damning to me. I’d hope that you’d have a better idea than most of whether he is talking about the same data, in both 2005 and 2009.

  26. Die Rechte Ecke
    Posted Nov 22, 2009 at 9:33 PM | Permalink

    And the denials begin – look at who commented on the downplay article and added to it – see the bottom of Wired’s comments: none other than Michael Mann
    Ha!! They’re scrabling now that we know outright that they’re manipulating.

    http://www.wired.com/threatlevel/2009/11/climate-hack

  27. Posted Nov 23, 2009 at 8:58 AM | Permalink

    Steve –

    You’ve got a classical situation that demands mirrored websites. Generally these are placed in server farms, never stand alone server.

    Yes some wordpress tweaks can improve response time, such as the add ins to convert php pages to static html. As an example, a closed thread can be converted to straight html – why not, it’s closed. It’s nuts to use php to generate that page, and it is one looooonnnggg page, when there are 750 or 1500 posts in it.

    But that is not going to help a lot in these classical bottleneck situations due to inbound client requests one or two orders of magnitude higher than normal.

    Is it a valid concern or question, whether some hosting service would shut you down at politically sensitive moments? I don’t think so necessarily – an hour or two of work and inquiry asking where (as an example) right wing US websites are hosted should show some hosting services that don’t play these silly games. That is just one example of solution method.

    Large server farms route incoming traffic to a selected or random box, example, Amazon, ebay. At the simplest level of homebrew, an initial page could be used for index.htm, this is what initially displays on typing the URL into search bar:

    Box #1

    Climateaudit.org – select your site

    1. Click here for primary climateaudit.org (redirects to Box #2, “climateaudit500.org”)
    2. Click here for read only mirror #1, backup as of midnight (redirects to essentially, offshore mirror #1, “climateaudit501.org”)
    3. Click here for read only mirror #2, backup as of midnight (redirects to offshore mirror #2, “climateaudit502.org”)

    In a practical sense, this particular index.htm would not have to be used except in periods of high traffic.

    If you want to discuss further, email me.

  28. TAG
    Posted Nov 23, 2009 at 12:35 PM | Permalink

    Reading and listening to the rationalizations of climate scientists about the current controvery, I thought of Talleyrand’s observation about the French ancien regieme ” They have learned nothing and they have forgotten nothing”. Looking up Telleyrand, I found an even more apt quotation “”Tout ce qui est exagéré est insignifiant.”: (“All that is exaggerated is insignificant.”)” . Climate scientists would do well with a deep study of Talleyrand.

  29. Pooh
    Posted Nov 23, 2009 at 1:18 PM | Permalink

    Denial Of Service (DOS) attacks as a cause of slow response time? Just a thought.

  30. Posted Nov 23, 2009 at 3:22 PM | Permalink

    Hi Steve,

    Keep up the fantastic work! Open Science with Auditing is the wave of the Future of Science.

    WordPress allows you to have your own domain name used for any web site you host there. You should be able to set up a third level sub domain name under your own domain “_____.ClimateAudit.org” where you could have for example “mirror.climateaudit.org” or even just “www2.climateaudit.org”. This way you can keep your “climate audit identity” while moving to faster servers.

    Just a thought.

    All the best,

    pwl

  31. Tom Sullivan
    Posted Nov 23, 2009 at 5:13 PM | Permalink

    Steve,
    In order to reduce the load being placed on your site at this crucial time, the best solution (for WordPress) is WP-Cache.

    http://wordpress.org/extend/plugins/wp-cache/

  32. Daryl M
    Posted Nov 23, 2009 at 11:45 PM | Permalink

    Here is the first written word I’ve seen from Lord Monckton since the leak: They Are Criminals.

    Also, Lord Lawson announced the formation of Global Warming Policy Foundation (GWPF) in the House of Lords.

  33. Jeff M
    Posted Nov 24, 2009 at 5:48 AM | Permalink

    Steve,

    I think you could get away with some simple optimizations, this one may have a lot of bang for the buck:

    http://tantannoodles.com/toolkit/wordpress-s3/

    Moving static image files off of this server will also help, I assume your server spends more time sending image data than it does rendering page content, however if it is spending more time rendering page content, a caching proxy server would be the next step, especially for the home page content.

  34. Posted Nov 24, 2009 at 10:34 PM | Permalink

    Climateaudit is largely text based, where there are images they are simple jpegs such as are embedded into posts or comments. There are no issues such as media file uploading or downloading which slows it down.

    There are large, long page renderings by the php server. On CA these can be 1500 posts. That’s a big job. For one example, “Mike’s Nature…” thread is now 300 comments +. It appears as one long page when clicked.

    This can be fixed by Dashboard, Settings, Discussion, Other Comment Settings, then check “Break comments into sections” and pick a number like 50. I personally like the 1500 comment method, but that requires a correspondingly higher CPU workload for that one mouseclick.

    If a page averages 300 and this is set to 50, then server load goes down for page retrievals by a factor of 300/50 = 6, which is substantial.

    Cheers.

  35. Posted Nov 24, 2009 at 11:44 PM | Permalink

    Jeff M:

    is a map of CA, realclimate and Watts for the last year.

    But now Rush Limbaugh and Glenn Beck are mentioning Steve. That’s going to go on for a while and cause wild swings in traffic. Oh, and various attacks
    of course.

    I like the addin you referenced, but it wouldn’t help this site. Caching pages would, though. But it’s still not sufficient to handle 100 or 1000:1 traffic swings, that requires either homebrew directing traffic to separate boxes as I described in previous email or server farm doing so automatically.

    And replacing php pages with static html, and reducing the complexity and length of php burden on rendering a post.

  36. Dave
    Posted Nov 27, 2009 at 9:15 PM | Permalink

    Coordination suggestion:

    Note — I am assuming that the high-traffic situation is still a problem since a good deal of the new content is being posted at the mirror site. With that in mind:

    May I suggest that during this period of high traffic, all new content be posted at the mirror site and that a prominent notice be placed at the top of this site that directs visitors to the new site to read the new content.

    Just a suggestion.

Follow

Get every new post delivered to your Inbox.

Join 3,421 other followers

%d bloggers like this: