CA struggles under load #94

For the umpteenth time (as long time CA readers will know) the blog is under heavy assault from people whose only wish is to read the latest from Steve McIntyre unfiltered.

As a consequence, the MySQL db has caused lots of timeouts as the number of readers hits the maximum number of connections (which were probably the defaults) and the maximum numbers of lots of other parameters.

Mr Pete and Anthony have both been on the case and have added more hamsters onto the treadmill to keep CA running. They do ask for your patience – if CA appears to be down, give it 30 minutes and then try again – they are working on this issue.

You can see from our stats, the graph looks eerily familiar…

56 Comments

  1. Urederra
    Posted Feb 4, 2009 at 5:48 PM | Permalink

    Misleading graph: the Y-axis should start from 0. 😛

  2. Posted Feb 4, 2009 at 6:03 PM | Permalink

    Thanks for doing so much hard work it is appreciated.

  3. Mark M
    Posted Feb 4, 2009 at 6:05 PM | Permalink

    Being that this blog is sometimes about climate, you may want to place it in the “cloud”. Seriously – seriously consider moving this to EC2 with your mysql db on EBS. Easy to do if you have not already done so! 🙂

  4. Geoff Sherrington
    Posted Feb 4, 2009 at 6:43 PM | Permalink

    Please don’t lose that humour.

    Surely, your graph Y axis above would be inverted to be eerie?

    Re: Mark M (#3),

    Have VERY mixed reports about ‘cloud’ work for computer alternatives. I’d be prudently letting it settle down to let its timing coordination problems sort.

  5. Luis Dias
    Posted Feb 4, 2009 at 6:44 PM | Permalink

    Hmmm, does this mean that CA is the real primary driver of the climate? 🙂

  6. Larry Sheldon
    Posted Feb 4, 2009 at 7:03 PM | Permalink

    “Misleading graph: the…”

    It is a hockey stick. The science is settled.

    By definition it can not be “misleading”.

    • DJ
      Posted Feb 4, 2009 at 7:08 PM | Permalink

      Re: Larry Sheldon (#7), Yes! Definitely, the shape of a “Hockey Stick”. Just can’t get rid of it can we…:)

  7. subscriber
    Posted Feb 4, 2009 at 7:05 PM | Permalink

    Sirs,
    On the evidence of your graph I’ve just hit the tip jar to keep you on the air. I’ve no tv licence and your recent mystery serial has given me the best entertainment of the year – even surpassing last year’s contribution from “Knowles et al”. If, on subsequent analysis, the graphic proves to be unsound/misleading/incorrectly derived/statistically insignificant/based on corrupt data/an artwork, or whatever, then ….

    er,…

    I will never know.?.

    oh,..er:
    Can someone audit this place please?
    Yours etc etc
    🙂

  8. Posted Feb 4, 2009 at 7:10 PM | Permalink

    Unprecendented!

    • AnonyMoose
      Posted Feb 4, 2009 at 9:38 PM | Permalink

      Re: lucia (#10),
      Yes, it does not look like teeth should.

  9. Posted Feb 4, 2009 at 7:10 PM | Permalink

    Dang these typos!

    • Posted Feb 4, 2009 at 7:14 PM | Permalink

      Re: lucia (#11),

      This problem has been audited and definitely originates between keyboard and chair 🙂

  10. Billy Ruff'n
    Posted Feb 4, 2009 at 7:16 PM | Permalink

    OK, I’ve had my fun. I’ll get off this joy ride and let someone else have a turn! Bonne nuit….au demain.

  11. Bob Mc
    Posted Feb 4, 2009 at 7:19 PM | Permalink

    You can see from our stats, the graph looks eerily familiar…

    I smell a preference for Bristlecone averaging.

    To CA hosts: Don’t forget to archive the data that supports your graph.

  12. sprats
    Posted Feb 4, 2009 at 7:22 PM | Permalink

    Not bad for a site that came second in the Weblogs awards. Must be a concensus of some sort.

  13. Bob Mc
    Posted Feb 4, 2009 at 7:24 PM | Permalink

    Oh, are you guys in for it now.

  14. Paul Zrimsek
    Posted Feb 4, 2009 at 7:34 PM | Permalink

    As a consequence, the MySQL db has caused lots of timeouts as the number of readers hits the maximum number of teleconnections

    Fixed.

  15. Kohl Piersen
    Posted Feb 4, 2009 at 8:02 PM | Permalink

    “You can see from our stats, the graph looks eerily familiar…”

    Ah well…. didn’t somebody notice that any data at all through the Mannomatic gives ….

  16. Shallow Climate
    Posted Feb 4, 2009 at 8:05 PM | Permalink

    The graph is obviously a Mannian ploy, probably a mole here at CA. Extreme filtering, no error bars: Butterworth filter, most likely. After some time at this blog I can spot ’em in an instant. Country-of-origin: USA. Able to narrow that down to Sheep Mountain, CA.

  17. Patrick M.
    Posted Feb 4, 2009 at 8:24 PM | Permalink

    Don’t be surprised when you see CA#94 referenced in the proxy data of MBH09.

  18. MrPete
    Posted Feb 4, 2009 at 8:52 PM | Permalink

    ‘twould appear that some tweaking has significantly helped. Who woulda thunk we’d hit close to 200 simultaneous hits at a time?!!

  19. Don.W
    Posted Feb 4, 2009 at 9:12 PM | Permalink

    All,

    Being a long time lurker here I just wanted to thank everyone for their efforts with keeping the blog up to the task and can report that there have been no problems here in Northern, NV! Of course I’ve been laughing so long and so hard in between posts that I’ve probably been hitting the server just at the right time when I finally do refresh. The last couple of days have been truly entertaining!

    I came upon this site some time ago and my wife and I were able to contribute to Anthony’s efforts at his Surfacestations.org project while on vacation a year or more ago. What an education!

    As a layman, this site gives us faith that over time and through the work of you people and your contributors the science will at least be understood (I dare not say settled as at the very least we’ve all learned that!) and most of all it will be right. And I must confess that if things keep going for the next couple of days I do look forward to the return of “THE MOSH PIT!” Love your stuff Steve Mosher… keep it comin!

    Thanks to all again and best regards to Steve and his granddaughter!

    Don.W

    • DJ
      Posted Feb 4, 2009 at 9:36 PM | Permalink

      Re: Don.W (#22), Here, Here with what you’ve said Don. I too found Ca last year after I decided to go on Temporary Retirement. So I’ve gotten into this AGW thing and this post even though over my Head ALWAYS gives me hope! Thanks to all of those who contribute and those who watch from the Sidelines. Steve, I wish there were more like you out there in “Just doing Your Best”! I pray for you from other Grandpa to you that your Granddaughter gets well. For the “Holy Spirit” IS with you!

      Sincerely,
      DJ

  20. RevYJ
    Posted Feb 4, 2009 at 9:47 PM | Permalink

    I just dropped a hundred bucks into the tip jar. I know it’s not the same as a cool quarter mil from a certain catchup heiress, but we all have to make the best we can of these uncertain times. Steve, you’re the right man at the right time. History is on your side, so please don’t stop.

    • Posted Feb 5, 2009 at 2:52 AM | Permalink

      Re: RevYJ (#25),

      Blimey, Steve should have server problems more often!

      I’m sure he’s grateful.

      Meanwhile a statistically significant cartoon joke from XKCD:

      • Luis Dias
        Posted Feb 5, 2009 at 7:07 AM | Permalink

        Re: John A (#32),

        John A, you’re not trying to use somebody else’s work without due credit, now are you?

        The cartoon comes obviously from XKCD.

        John says: You’re quite right. The thing is that on XKCD, the link given doesn’t contain the name of the original comic. But I’ll add it in now.

      • AnonyMoose
        Posted Feb 5, 2009 at 10:30 AM | Permalink

        Re: John A (#32),
        XKCD comic is, as usual, even better with the mouseover popup.

  21. Posted Feb 4, 2009 at 10:02 PM | Permalink

    You really need WP Super Cache http://wordpress.org/extend/plugins/wp-super-cache/

    This plugin generates static html files from your dynamic WordPress blog. After a html file is generated your webserver will serve that file instead of processing the comparatively heavier and more expensive WordPress PHP scripts.

  22. Jedwards
    Posted Feb 4, 2009 at 10:31 PM | Permalink

    Surely this result is teleconnected to sudden level of obfuscation from from RC….

  23. Steve McIntyre
    Posted Feb 4, 2009 at 10:54 PM | Permalink

    Pete especially, John A and Anthony, thanks so much. I’ve been out for about 6 hours and the fur sure was flying while I was out. 🙂

  24. Dave Dardinger
    Posted Feb 4, 2009 at 11:16 PM | Permalink

    This may be similar to one I did before but anyways…

    Steven McIntyre
    A man, a plan, a blog site.
    It’s Climate Audit!

  25. Kalle
    Posted Feb 5, 2009 at 2:02 AM | Permalink

    Have you looked into using any wordpress cache plugins such as this one?:

    http://wordpress.org/extend/plugins/wp-cache/

    It should take a lot of load off the db. Many thanks to everyone involved in CA for the great work.

  26. Glynn
    Posted Feb 5, 2009 at 3:25 AM | Permalink

    I’d just like to echo the sentiments of comment #22, and add that here in snowy England I rarely have problems connecting to the site, and I have visited virtually every day for a long long time.

    Unfortunately i’m not bright enough to contribute to the usual comment threads, but I do get the gist of most of it and what fascinating reading it makes. Keep up the good work.

  27. kim
    Posted Feb 5, 2009 at 4:19 AM | Permalink

    A planet and its climate turns.
    Who cares?
    A child is ill.
    ==============

  28. MrPete
    Posted Feb 5, 2009 at 6:44 AM | Permalink

    Update: I’ve added a few performance enhancements that should both reduce load on the server and give you faster viewing.

    The first time you view a page may still be a bit slow, but future viewings should be much faster.

    (Note to interested tech folk: we already were using wp-cache. wp-super-cache won’t work because we avoid fancy permalinks. we also use php_speedy_wp now.)

  29. Mike Bryant
    Posted Feb 5, 2009 at 7:24 AM | Permalink

    Today’s Peanuts cartoon strip is about hockey players trying to take over the world… until their curved sticks are made illegal. Can’t find the URL, I saw it in a REAL paper this morning…

  30. John Knapp
    Posted Feb 5, 2009 at 8:26 AM | Permalink

    I’m one of those lurkers who are driving the readership up. Thanks for all your work. I have learned a lot reading your site as well as Real Climate. Though the difference in tone, objectivity, and seeking for veriable fact instead of trying to support previously held beliefs makes me prefer Climate Audit. I just trust you guys more.

    When I read Real Climate I am reminded of Richard Feynman’s statement that a scientest has the duty to try to disproove his/her own theory. By that definition they are not scientists.

    Gavin et al are obviously very smart people but they verifiy my own working definition of intelligence which is: A measure of a persons ability to rationalize their own prejudices.

    In any case, thanks for the education.

    ps. I have just made my drop into the tip jar to help you stay on the web. All you other lurkers….

  31. Independent Gav
    Posted Feb 5, 2009 at 8:55 AM | Permalink

    Whoar!

  32. Posted Feb 5, 2009 at 10:57 AM | Permalink

    Steve you just have to quit using those damn strip bark servers!

  33. Steve Burrows
    Posted Feb 5, 2009 at 11:19 AM | Permalink

    At long last, I made a donation, hope it helps.

    Please keep up the good work, much appreciated.

  34. Tim Wickstrom
    Posted Feb 5, 2009 at 11:25 AM | Permalink

    Add me to the list of long-time readers who’s just now made a donation to this site. The work done here is invaluable.

  35. AnonyMoose
    Posted Feb 5, 2009 at 12:12 PM | Permalink

    OT: A non-climate “scientist” being criticized for not releasing his methodology for Lancet Iraq death study.

    • cbone
      Posted Feb 5, 2009 at 1:32 PM | Permalink

      Re: AnonyMoose (#45),

      Its nice to see at least one field of science and University that is Mann enough to enforce their professional conduct standards. It is also interesting to note that the Journal involved is playing the typical game of ignoring the problem.

      I was especially impressed with this quote:”When researchers draw important conclusions and make public statements and arguments based on survey research, then subsequently refuse to answer even basic questions about how their research was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all survey and public opinion research.”

      With a few minor changes we get what (I believe) Steve has been saying all along: “When researchers draw important conclusions and make public statements and arguments based on statistical analysis, then subsequently refuse to answer even basic questions about how their analysis was conducted, this violates the fundamental standards of science, seriously undermines open public debate on critical issues, and undermines the credibility of all statistical and Climate Science research.”

  36. Son of a Pig and a Monkey
    Posted Feb 5, 2009 at 1:06 PM | Permalink

    You have to get of the readership bulge in the “Middle Ages”

  37. theoldhogger
    Posted Feb 5, 2009 at 2:07 PM | Permalink

    jon….eveyone’s adding to the tip jar. Don’t you think you should too?

  38. mortimer74
    Posted Feb 5, 2009 at 2:50 PM | Permalink

    Steve: Thank you for helping us get to the truth. Your service is invaluable, but worth at least the $20 I just put in the tip jar.
    I bailed out of the so-called environmental movement years ago due to the rash politicization of the issues. It just keeps getting worse, so we need to redouble our efforts to promote good science as the foundation for good policy.

  39. Posted Feb 5, 2009 at 3:02 PM | Permalink

    The problem wasn’t so much the caching, but the sheer numbers of connections to the MySQL database which caused the timeout. I’m glad to say that between them, Mr Pete and Anthony have certainly improved the performance under load.

  40. Max Lybbert
    Posted Feb 5, 2009 at 3:46 PM | Permalink

    For MySQL connection pooling, you *may* be interested in The New York Times DBSlayer ( http://code.nytimes.com/projects/dbslayer ). Some of the design is discussed at http://open.blogs.nytimes.com/2008/05/08/announcing-the-dbslayer-activerecord-adapter/ .

  41. Bob D
    Posted Feb 5, 2009 at 6:20 PM | Permalink

    I added a tip as well. $20 isn’t a huge amount, but if we all do it it’ll add up. Just wish I could afford more, the work Steve is doing here will be quoted quite a way into the future, I believe.

    For me, this whole saga, while not scientifically all that relevant (but politically very relevant), is showing two things:
    1. the whole argument of ‘only peer-reviewed papers are accurate/admissible’ is weak, or at least badly flawed; and
    2. many of the the scientific people behind AGW are being shown up to be activists – they are clearly not just dispassionate scientists only interested in the truth. That’s not to say some of them aren’t still good scientists – rather they spend a little too much time pushing an agenda.

  42. HarryG
    Posted Feb 5, 2009 at 10:11 PM | Permalink

    I refuse to believe the graph until I have been shown the UNCENSORED file contents.

  43. Posted Feb 5, 2009 at 11:12 PM | Permalink

    Steve

    Dropped my tip in the jar as well. Good job and keep at it!

  44. AGWBush
    Posted Feb 6, 2009 at 7:47 AM | Permalink

    Does that graph mean I can sail across the Arctic this decade after all?

  45. Jdubya
    Posted Feb 6, 2009 at 12:59 PM | Permalink

    Hey Mann,

    Nice hockey stick. Mind if I “borrow” the data? I have something I need to prove in order to get some funding.

  46. davidc
    Posted Feb 6, 2009 at 4:57 PM | Permalink

    “For the umpteenth time (as long time CA readers will know)…”

    My mother used to say that as a way of enumerating my serial misdemeanors. I used to think that umpteen = umpteen + 1 since the number umpteen stayed the same with each new misdemeanor. I now know that this property is claimed by a much larger number than my mother had in mind. Returning to this problem after many years I wonder if it is the solution to the equation

    umpteen =rounddown(umpteen + M)

    where M is a number just a little bit less than 1 for each new misdemeanor. That (M less than 1) as well as the use of rounddown rather than (say) roundup maybe shows a generosity that I missed at the time.

    But does someone else have a better definition of umpteen?

One Trackback

  1. […] in an instant. Country-of-origin: USA. Able to narrow that down to Sheep Mountain, CA.” . CA struggles under load #94 . _________________ . Al Gore has a new argument for why carbon dioxide is the global warming […]