Road Map and Site Rules

Opinions expressed on Climate Audit, other than those expressed by Stephen McIntyre personally, are those of the individual posters and do not necessarily reflect the opinions of Climate Audit or myself.

A Concern: Ken Fritsch makes the following comment:

While your efforts to avoid the implication of censoring of opposing views should be commended, I am not a little distracted by the noise levels that I find come from (a) personal debates that frequently do not add to the knowledge base of the specific topic at hand, (b) posters who seem to come to the discussion with the intent of having their feelings hurt or to uncover evidence of a bias towards them and/or people with their points of view, (c ) posters who raise to the bait of these posters and thus contribute to wasted space (ad hominem ad infinitum), (d) posters who merely seem to want to let skeptics and agnostics know at every opportunity that the circumstantial case is closed on AGW and only fools would question what they surmise to be an overwhelming and proven consensus from the climate scientists, (e) those who make their personal cases against AGW with little or no evidence to back it up and (f) those who seem to want to show that they can turn your efforts as a critic of some sometimes sloppy and vague climate science publishing back on you.

There are lots of places in the world where people can discuss general issues of AGW, but not many places where technical discussions of proxies can take place. I’m getting really tired of technical threads getting hijacked. If there’s a thread on Lago Paco Cocha or Quelccaya Plant Deposits or a technical topic, please do not hijack for general fuming. If anyone wants to vent, vent on the National Post Op Ed sort of thread and stay away from the technical threads. In order to encourage this, I am warning that I may start deleting off-topic posts on the technical threads. Yeah, yeah, I’m sure that somebody will claim that they are being censored, but I’m going to try it and see if the noisiness will reduce.


Caveat:
Third-party posts do not reflect my personal views or the views of this site.

Posting Suggestions: You can insert images into posts – see instructions here. For adventurous people, you can insert Latex commands for math formulae.

Some Site Rules: I have previously said that I have total contempt for the censoring of scientific comments at realclimate and do not do that here. However, light moderation opens the door for ad homs and taunting, which quickly involves everybody. I don’t have time to monitor everything so my handling of taunting has been inconsistent: sometimes I’ve let it go because the person is just making a fool of themself, sometimes I’ve got fed up and deleted it. A reader has written with the following suggested ground rules which are hereby adopted:

Blogs like this one provide a wonderful opportunity to people like me (a retired scientist) to get involved in an ongoing debate and it is very disappointing when the debate generates into one of these slanging matches. May I suggest some ground rules for posts:

1. Refrain from personal abuse and swearing,
2. Never attribute ulterior motives to another participant
3. Be patient with people who know less science or maths than you do yourself.

People who consistently break rule 1 and 2 should be issued with a yellow card by the moderator. If they continue they get a red card and are banned from the site.

While there’s a little politics from time to time, by and large, I would prefer that you don’t talk politics; there are plenty of other perfectly good places to do that. I don’t allow discussion of religion and will mark anything even close as spam.

New Posters: You sometimes get tripped up in our spam filter. Unfortunately in today’s world, a blog like this gets attacked by hundreds of spams a day and they are screened by a computer filter. Some of the things that the spam filter looks for is a sudden burst of activity from an unrecognized address; it may allow some posts through and then get triggered after a while and start rejecting posts. If one of your posts doesn’t go through, don’t keep sending them in; it just inflames the spam filter. If you have yahoo or hotmail address, the spam filter may also screen you. Sometimes people get filtered for reasons that I don’t understand. However, despite this, we are reliant on the spam filter. Contact us by email if you get caught up- see contact category at right.

Site Road Map
The main topic here are the multiproxy studies of millennial climate, which is what I work on, with some discussion of climate models. I want to keep the focus fairly narrow as there are plenty of other places to talk about things and I think that sticking to a niche is a good idea.

This site used to be pretty easy to follow through, but it’s now sprawled out with lots of little nooks and crannies. Here’s a roadmap to the site, which covers quite a bit more than our criticisms of MBH.

The Categories bar at the side is quite useful in reflecting what I think are the main themes here. Most posts that I wrote in the spring are just as topical (or untopical) now as they were then. Feel free to revive any of them. It’s also surprising what you can find on google. If you do “climateaudit” and any any other word, you can usually find an old post. You can find our articles on the right frame.

A recent exposition to the NAS panel is here.
(It’s surprising how high we get on google even on topics like “briffa climate” or “mann climate” or even other oddities like “preisendorfer”.)

Obviously, the main calling card is the critique of the Hockey Stick diagram of Mann, Bradley and Hughes (MBH), that was featured in the IPCC Third Assessment Report and many government publications.

If you go to the Articles sidebar, there are links to our formal publications. Ross has written an overview also linked there, that many people like.

My own short-form summary of our views on MBH98 is this. MBH98 made 5 main warranties: statistical skill, robustness, careful proxy selection, appropriate methodology and relatively even geographical balance. These warranties were fundamental to its acceptance. (My background is in business and I think in contract terms.) All their warranties have been breached. Their reconstruction failed critical cross-validation tests (we have publicized the R2 failure, but it fails others as well); it is not robust the presence/absence of bristlecone pines; the supposedly carefully proxies included bristlecone growth, which specialists say is contaminated by 20th century fertilization; their methodology includes a wildly biased “principal components” methodology (which is not actually a principal components method). The hockey stick is an imprint of bristlecone growth rate and reflects a non-temperature proxy from an isolated geographic region of the U.S.A. Again read through the articles and the exact language there should be preferred to this short re-statement.

There has been extensive coverage -see News and Commentary – the most notable of which are the profiles by Natuurwetenschap & Techniek (translated into English) and the front page coverage by the Wall Street Journal – but there has been extensive coverage elsewhere in Science, Nature, The Economist, National Post and European newspapers. Listings here are by no means complete. There have been two published Comments – one by von Storch and Zorita and one by Huybers, both of which we made detailed (and IMHO) complete Replies. realclimate has also criticized our critique on numerous occasions. If you go to the Category – MBH98, you’ll see some of our direct responses to realclimate at Errors Matter #1, # 2 and #3. The House Committee on Energy and Commerce (Barton Committee) has taken an interest in these matters and it has a Category as well.

One of the “so what”s sent our way is that the other multiproxy studies show the “same thing” and so, even if MBH is wrong, it “doesn’t matter”. I’m not convinced that these other studies are much good either. I’ve posting comments about these studies from time to time. Again go to the category Other Multiproxy Studies and there are subcategories for several of the major studies. There is a fantastic amount of overlap of authors and proxies, so that these other studies are not “independent” as ordinary people understand the term and their findings of the relative position of the Medieval Warm Period and the 20th century are very vulnerable to the bristlecones and Polar Urals series being unusable.

I’ve collected information on individual proxy series (see Category), which I’ve posted up from time to time e.g. on bristlecones, on Thompson’s ice cores, etc.

I’ve also started to make posts on statistical topics that I think are relevant: “spurious” regression as this is understood in econometrics (where there is a much more advanced understanding of autocorrelation than exists in paleoclimate); some posts on ARMA time series – I’m interested in ARMA(1,1) processes with AR1 coefficients >0.9, which are characteristic of many processes and have some odd statistical properties.

I have an ongoing campaign to improve standards of data archiving, disclosure and due diligence -(see Category) – which are independent of any particular substantive points on paleoclimate studies. I have no idea why the “Hockey Team”, as they styled themselves, have elected to withhold data and methods from scrutiny; it’s an unwinnable position, but they’ve done so and I’ll continue to criticize them on this point.

Sometimes I lapse into controversy, mostly after I’ve been slagged in print somewhere, but I try to stay cheerful.

As to your host, I’m pretty good at answering many questions, but have difficulty answering the question: what am I? No two public descriptions of my occupation are the same. I studied mathematics at university in a fine undergraduate program at the University of Toronto and was very competitive at it. My skills, as refreshed, are more than sufficient for what I’m doing. I’ve been in business nearly all my working life, most recently in financing and promoting mineral exploration projects. That gives you a lot of experience in the school of hard knocks and that counts for a lot in my opinion. (One of my underlying themes is that disclosure standards for climate scientists should be at least as high as that required of mining promoters.) One public mineral exploration company with which I was involved underwent a reverse takeover and became an oil exploration company (when I ceased to be an officer and director of the company.) I’ve done a very small amount of business consulting for it, but no energy consultant would call me an “energy consultant”, nor would I describe myself as one. In terms of occupation, right now, this is what I’m doing. No one’s paying me to do this and there is a substantial opportunity cost for me personally in doing this, but I enjoy it and can afford to do it for a while. (Given that our work has attracted enough interest that public funds have been employed to criticize it, I see no a priori reason why I should do it for nothing and make no long-term commitment to wear a hair shirt.)

I like the feedback. So look at the Categories to crosscut the sprawl here. I’m amazed at the number of hits that the blog receives. It seems to have found a niche and I’m amazed at some of the people who have found it. I particularly welcome the comments and feedback. Lots of hits are for that exchange rather than for me and, if I didn’t get the feedback, I wouldn’t keep up the blog.


63 Comments

  1. Steve McIntyre
    Posted Jan 18, 2007 at 8:43 AM | Permalink

    If you want to make comments on site policies, please do so on this thread. Please use the Unthreaded thread for topics that you wish to raise that do not relate to existing threads. There are also lots of old threads and sometimes they are worth reviving.

  2. Follow the Money
    Posted Jan 18, 2007 at 4:59 PM | Permalink

    (b) posters who seem to come to the discussion with the intent of having their feelings hurt or to uncover evidence of a bias towards them and/or people with their points of view, (c) posters who raise to the bait of these posters and thus contribute to wasted space (ad hominem ad infinitum),

    I don’t mind (b), I mind (c). Really, people should reply with sympathetic concern, short witty remarks, or not at all.

    As for peronal debates, I think they are unavoidable especially as the threads age. Best one can do is regular warnings to temper them down in heat and amount. Happens everywhere. A sociologist should study net threads and their devolutions. Lots of proxies to work with. :-)

    “I would prefer that you don’t talk politics; there are plenty of other perfectly good places to do that.” In general I agree with you, but as politics relate to AGW advocacy I can’t see how it’s not relevant – at least on the open thread, not on individual topic threads – a concept I recently violated, sorry! The reason for the debate about carbon-based AGW and solutions is politics, the money behind caron trading systems, aka “cap and trade,” “Kyoto-style solutions.” I don’t know where else this is being debated rationally, all other sites I’ve seen are basically peopled with Gore-huggers on one side and SUV huggers on the other.

    As for your role I sense some amazement on your part at your site’s importance. There are other examples of persons whose websites have become the locus of informed discussion, who otherwise have no financial or employment ties to the issue. For example a New York professor having no connection at all to Duke U. runs a site with the best level-headed analysis of the lacrosse case. But yours is by far the finest example. And your role for openness in your field is unparalled. Congrats and keep up the good work.

  3. Willis Eschenbach
    Posted Jan 18, 2007 at 5:42 PM | Permalink

    Steve, would it be possible to create an area where users (and yourself) could upload “R” scripts? I learn more from reading other people’s scripts than from any number of texts, and it would fit the spirit of the site to have the scripts archived.

    Thanks either way,

    w.

  4. Steve McIntyre
    Posted Jan 18, 2007 at 6:13 PM | Permalink

    I’ve got an area at http://www.climateaudit.org/scripts where I upload scripts from time to time.  This directory has always been there but John A has made it searchable now that we’ve got more control the VPS system. I’ll email you about establishing a willis subdirectory.

    I obviously agree with you on scripts.  I think that the main reason that most articles don’t have scripts is that the authors are embarrassed to show how trivial the analysis is.

  5. Posted Jan 19, 2007 at 7:34 PM | Permalink

    Steve:

    I agree with Willis, I have learned more about R reading and decoding your scripts, and other, than all the text books on R I have collected. Oh, I use the text books, but the core of my learning is seeing how you have used R to produce the telling graphics. By the way, thanks for going the extra mile for all of us who are still learning R, statistical analysis, and where the devil is hiding in the details.

  6. s243a
    Posted Jan 19, 2007 at 9:47 PM | Permalink

    Someone needs to put up some anti-kyoto stuff on youtube that has to do with Canada. Ronna Ambrose’s speech to the Untied nations isn’t even on youtube yet she endorsed Kyoto and still faced criticism form environmentalist.

    http://www.ec.gc.ca/minister/speeches/2006/061115_s_e.htm

    http://youtube.com/results?search_query=kyoto+canada&search=Search

  7. s243a
    Posted Jan 19, 2007 at 11:00 PM | Permalink

    Here is what I could find:

    Climate Catastrophe Cancelled

    Here is another vedio:

    INTERGOVERNMENTAL PANTOMIME OF CLIMATE CHAOS BLUES

    Ask a CEI Policy Expert: Iain Murray on the IPCC report leak

    Professor Tim Palmer on climate change

  8. Posted Jan 23, 2007 at 2:39 PM | Permalink

    All the posts on this site are your own views. The fact that they run counter to every reputable climatologist and most science should permit most readers to assess their value.

  9. Steve McIntyre
    Posted Jan 23, 2007 at 2:51 PM | Permalink

    #8. I am unaware of any views expressed here that run counter to “most science”. Most opinions that I express are statistical in nature and I try to be careful. If you find an errors in what I’ve said, I’d be quite happy to amend the comment.

    Our papers were considered recently by two senior committees. Wegman et al, chaired by the Chairman of the National Academy of Sciences Committee on Theoretical and Applied Statistics, supported our findings in categorical terms. The NAS panel chaired by North et al endorsed the specific points of our papers and did not reject any of our findings. If there’s anything specific that I’ve said that you disagree with, you are free to say so.

  10. Gerald Browning
    Posted Jan 23, 2007 at 8:24 PM | Permalink

    This is a test of LaTeX: $latex \frac{a}{b}[\tex]

  11. Gerald Browning
    Posted Jan 23, 2007 at 8:35 PM | Permalink

    John A.,

    Do I need quicktags for $latex and [\tex] in oder to use mime TeX?

  12. Steve McIntyre
    Posted Jan 23, 2007 at 9:28 PM | Permalink

    #11. Gerry, Latex isn’t working since the change to a VPS server for reasons that JOhn A has not been able to determine.

  13. Steve McIntyre
    Posted Jan 24, 2007 at 10:33 AM | Permalink

    I apologize for the repeated crashes during the last 2 weeks. Despite the upgrade to CPS, the site is still generating too much load and crashes the server. They say that another upgrade to a dedicated server from a virtual private server is required.

  14. Posted Jan 24, 2007 at 10:38 AM | Permalink

    Which means more money – correct?

  15. Gerald Browning
    Posted Jan 24, 2007 at 2:13 PM | Permalink

    The good news is the increasing number of people that are accessing your site and becoming aware of the observational data quality and methodology issues, the myriad climate model shortcomings, and the multitude of unanswered scientific issues swept under the rug by the IPCC.

  16. Tom Brogle
    Posted Jan 24, 2007 at 5:03 PM | Permalink

    [snip - Tom, please re-post in a more suitable thread or in Unthreaded]

  17. Ken Fritsch
    Posted Jan 25, 2007 at 12:12 PM | Permalink

    Which means more money – correct?

    John A, if that is what it takes, I would be willing to advance my future “dues” to CA. A step in the right direction would be to evaluate the costs involved and what it would buy.

  18. Arthur Dent
    Posted Jan 26, 2007 at 6:28 AM | Permalink

    I apologise if ths is in the wrong place but Ross may be interested that he is being lambasted by Jonathon hari in todays Independent newspaper for shilling for Exxon

    “She had lauded Ross McKittrick, funded by Exxon, for debunking” environmentalist graphs”

  19. Jim Barrett
    Posted Jan 26, 2007 at 5:45 PM | Permalink

    Steve: Yellow card for Jean S please. In thread “Southern Ocean Temperature Trends”:

    Posting 28: “….. please consult any (introductionary) text to time series before posting any more ramblings.”

    Posting 41: “….. study a bit more before posting more ramblings about things you obviously do not understand.”

    I’m getting pretty fed up with being accused of not providing substantial postings, and yet when I do attempt to post ideas clearly, thoughtfully and at some length, they are dismissed as “ramblings”.

  20. Posted Jan 27, 2007 at 1:40 AM | Permalink

    Steve: Yellow card for Jim Barrett please. Same thread:

    Your almost incomprehensible postings only suggest an aim to be perverse and obstructive, rather than to provide serious discussion. If you gave me some substantive argument, I would perhaps have something to respond to.

    (Not that I care, he can’t bring me down ;) )

  21. Mark T.
    Posted Jan 27, 2007 at 2:07 AM | Permalink

    My opinion is that Jim Barrett == Steve Bloom == Standard troll we’re all tired of explaining simple math to (the comments regarding “mean” = “trend” = “some form of weighted average” were the tip ‘o the hat).

    He got busted off this site trashing Steve, now he can’t post with his “normal” pseudonym without first issuing an apology. Not unexpected, of course.

    Mark

  22. Dave Dardinger
    Posted Jan 27, 2007 at 8:40 AM | Permalink

    Mark,

    The only trouble is that Jim Barrett has been around a while. I’m sure he and Steve Bloom overlap. And for that matter, I don’t remember any announcement that Steve Bloom has been booted from the site. (I wonder if there’s a list of “regulars” who aren’t allowed on because of bad behavior?)

  23. welikerocks
    Posted Jan 27, 2007 at 9:24 AM | Permalink

    re 22: see comment #180 in the Unthreaded topic. Steve Bloom hasn’t posted since SteveM saw the comments on that other blog. Bloom just crept back into the shadows for awhile I bet.

  24. Jean S
    Posted Jan 27, 2007 at 9:30 AM | Permalink

    re #19 (JB): Hah! I don’t know if my “crime” was the fact I consider your postings as ramblings with little content or the fact that I suggested that you should read some basic statistics texts. Whatever it was, from now on, I promise to stick with the site rule 3. In your case, it means that I won’t be answering any of your comments. In fact, I won’t even bother reading them anymore.

  25. Mark T.
    Posted Jan 27, 2007 at 4:25 PM | Permalink

    The only trouble is that Jim Barrett has been around a while. I’m sure he and Steve Bloom overlap.

    I realize that, but differentiating two trolls is difficult. They all look and sound alike.

    And for that matter, I don’t remember any announcement that Steve Bloom has been booted from the site. (I wonder if there’s a list of “regulars” who aren’t allowed on because of bad behavior?)

    Oh, I realize that, too. More of an off the cuff statement than anything. Bloom, however, has been strangely absetn since he trashed Steve and Jean on another site. He’s a coward, and a troll, and should be banned. Barrett is just another one that has very little idea what he’s talking about… c’mon, a mean and a linear fit are “fundamentally the same” said with such condescention as if he’s so bright? Sheesh…

    Mark

  26. Jim Barrett
    Posted Jan 27, 2007 at 11:02 PM | Permalink

    Mark T (postings 21 and 25):

    To summarise what I have been saying in the thread “Southern Ocean Temperature Trends” with statements such as:

    “mean = trend = some form of weighted average” or

    “a mean and a linear fit are `fundamentally the same'” (what I actually said was “there is no fundamental difference between the estimation of a trend and the estimation of a mean: they are both weighted averages over the input data” – you dispute this Mark?)

    only shows that you don’t understand (and probably don’t want to understand) the subtleties of what I explained at length in postings (54) and (61) of that thread. It really isn’t very hard – why don’t you either point out to me what is wrong in those postings – or perhaps get Steve to help you?

    Just saying “sheesh” is never a very good substitute for mathematical argument.

  27. Dave Dardinger
    Posted Jan 27, 2007 at 11:07 PM | Permalink

    re: #25 Mark,

    I know what you mean. I have a vision in my head of Jim madly thumbing through a text somewhere trying to find things which he can understand and will make it look to others like he has some expertese in statistics or whatever. I do this because it’s about where I’d have to do to go head to head with the stat guys around here.

    It’s like when I used to play chess. I could look at experts playing games and sometimes figure out what their plans were, but quite often they fooled me in what I thought was going to happen and then, perhaps, I’d see what had actually been going on. And on the few occasions where I got to play experts and above, I’d generally get rolled rather quickly; though I did manage to beat one expert once and draw with a master once; though mostly because he assured winning the tourney by doing so. Anyway, I knew I was mostly bluffing when appearing to have knowledge of what they were thinking but I wasn’t so foolish as to try telling them how they should have played the game in the post-mortem. Jim doesn’t appear to know what he’s facing here. But perhaps he is a big-shot statisticswise in the crowd he runs in.

  28. Mark T.
    Posted Jan 27, 2007 at 11:16 PM | Permalink

    only shows that you don’t understand (and probably don’t want to understand) the subtleties of what I explained at length in postings (54) and (61) of that thread.

    I fully understand what a linear trend is, and equating the fundamental relationship between a mean (i.e. a constant) and a trend (a line) is ludicrous. Making such statements invited others to question you first, btw… not just me.

    It really isn’t very hard – why don’t you either point out to me what is wrong in those postings – or perhaps get Steve to help you?

    A trend calculation is not fundamentally similar to the calculation of a mean, period. Certainly each of the points is weighted to create the linear coefficients, and I never argued once with your follow up description of the least squares algoritihm.

    Just saying “sheesh” is never a very good substitute for mathematical argument.

    It doesn’t take any mathematical argument to debate your statement, it is wrong not only by definition, but even by your own description of an LS calculation. Sheesh is simply amazement that you would think this way, and then continue to debate it even after demonstrating that you at least understand the concept.

    Mark

  29. Mark T.
    Posted Jan 27, 2007 at 11:17 PM | Permalink

    I know what you mean. I have a vision in my head of Jim madly thumbing through a text somewhere trying to find things which he can understand and will make it look to others like he has some expertese in statistics or whatever. I do this because it’s about where I’d have to do to go head to head with the stat guys around here.

    Wikipedia, actually. :)

    Mark

  30. fFreddy
    Posted Jan 28, 2007 at 4:33 AM | Permalink

    RE #27, Dave Dardinger

    I have a vision in my head of Jim madly thumbing through a text somewhere trying to find things which … will make it look to others like he has some expertese in statistics

    I think this is the key point here. He has already stated that his target audience is the large group of people who read this site without posting. I suspect that he is aiming for those people who do not have the mathematical background to distinguish between the real maths folk here and his own post-modernist nonsense. (Weighted average of a single time series, forsooth …)
    No doubt he will shortly be popping up on the credulist blogs bragging about how he creamed the amateurs at CA.

  31. Steve McIntyre
    Posted Jan 29, 2007 at 9:22 AM | Permalink

    The changeover to a VPS server has not done the job and the system is crashing. As it was explained to me, our daily traffic is about 1GB; the MySQL database is about 500 MB in size and the number of httpd requests (about 300 – per day? per hour? I don’t know) is causing cpu problems. The VPS server was a bump from $10 per month to $40 per month; but a Unix VPS server at Webserve.ca is $250 per month. I’ll ask John A about suggestions, but would welcome any bright ideas either on or offline.

  32. fFreddy
    Posted Jan 29, 2007 at 10:09 AM | Permalink

    Is it the number of requests or the volume of data transferred to service those requests ? If the latter, there might be some benefit from splitting some of the large current threads into two.
    “More Unthreaded”, for example, has nearly 400 responses and is nearly 3/4 MB in size.
    That price jump seems rather steep – they could buy you a dedicated box for that.

  33. Steve McIntyre
    Posted Jan 29, 2007 at 10:41 AM | Permalink

    #32. I haven’t been able to get a clear explanation of the problem, Here’s the sort of answer I get on their help line – which is very responsive.

    Deepak: yes, thats correct, but no of mysql connection and httpd request are too high to be handled
    Deepak: as this is ruinning main hardware node
    Deepak: you will need to make decision asap.
    Deepak: No of mysql connections are set up to unlimited
    Deepak: and no of httpd request for your single site is around 300
    Deepak: alsothogh now it shows less
    Deepak: but most of the time is more than 300
    Deepak: which is overusing the system
    Steve McIntyre: is that 300 per day? Doesn’t sond like very many?
    Deepak: and ruinning it

    Deepak: at a particular period of time it is 300
    Deepak: no per day
    Deepak: but that amount is per 15 minutes
    Deepak: but for particular span of time

    Steve McIntyre: how many httpd requests is normal? I don’t understand why my site would have unusual httpd requests – whatever they are.

    Steve McIntyre: I still don’t understand – 300 httpd requests per 15 minutes?? or per day?
    Deepak: 10 -30 are normal

    Steve McIntyre: what triggers a httpd request?
    Deepak: there can be several reasons
    Deepak: robots/search engines/ clients browsing your site
    Deepak: etc etc
    Steve McIntyre: I don’t know the etc etc, what sort of things

    Deepak: And it is not possible for me to tell you what exactly is triggering httpd requests
    Deepak: but the no of apache request are high
    Deepak: each httpd request coming to your site relates to mysql database
    Deepak: as pages are browsed
    Deepak: or if any client accesses wordpress part
    Deepak: several requests are made to site
    Deepak: the sie of your mysql databases is around 500 MB
    Deepak: each time when sigle request is processes
    Deepak: it takes a lot of cpu uage
    Deepak: and there by hampering hardware node
    Deepak: at this stage

  34. Chris H
    Posted Jan 29, 2007 at 11:05 AM | Permalink

    Here are some ideas.

    1. Instead of downloading the entire thread, split it into three downloads: the posting itself, comments made in the last 24 hours and older comments.

    2. Put all your static content, such as images on one web site and have your database on another.

    3. Ensure that the web server is making good use of the http expires header. This should be set to at least 24 hours for static content like the clouds picture. If you combine this with 1. above, you can set the expires header on both the posting and the older comments to 24 hours.

    4. Check to see how much the search function is being used and make sure it’s running at a low priority.

    5. Configure the web server to cache the threads in the Recent Comments List.

  35. fFreddy
    Posted Jan 29, 2007 at 11:28 AM | Permalink

    Deepak: each time when sigle request is processes
    Deepak: it takes a lot of cpu uage

    Question : Is this “a lot” by comparison with other blogs running on your service ?
    (If yes, this would imply some problem in the way this blog is set up.)

    Question : How does the number of Httpd requests change over time ?
    (If it correlates with time of day, it might be a large audience; if it is pretty constant for all 24 hours, that would imply some automated process is making the requests.)

    (Apologies for the guesses – I’m way out of my competence here.)

  36. Hans Erren
    Posted Jan 29, 2007 at 3:54 PM | Permalink

    sort the comments top to bottom and move the tail to other pages, then you don’t need to serve the entire thread at each view. or change to bbs type of style eg

    http://www.ukweatherworld.co.uk/forum/forums/forum-view.asp?fid=30

  37. Bob K
    Posted Jan 29, 2007 at 7:52 PM | Permalink

    I’d start by reducing the number of posts on the main page by at least half. Every visit that entire page gets downloaded. Some people may even have their browser reload the page when using the back button. Or on a short time interval.

    Limit comments to 25-50 per page, with a link to the individual pages located at the bottom of the post and comment page.

    I have a 4mb connection and some comment pages take quite a while to download. Like unthreaded. I think it would make browsing quicker and more enjoyable for all.

  38. Posted Jan 29, 2007 at 10:52 PM | Permalink

    Steve,

    What comments plugin are you using to show “recent comments” on the sidebar? Some are extremely inefficient because they make repeated calls to the Mysql database or call things in a very inefficient way. To experiment, change to a commens plugin that only calls the most recent comments without sorting by blog post. See if changing the call fixes the problem.

    If it does, then you can explore finding a plugin with the features you like.

  39. Posted Jan 30, 2007 at 2:48 AM | Permalink

    WP-Cache was suggested as a plug-in, but it won’t initialize because it claims that the wp-content directory is not writeable by the web server (which is not true).

    I think all of this is related to the issue of why LaTeX doesn’t work on this box. I think that somewhere, webserve have disabled the ability of the Apache system to execute commands or scripts.

    Maybe I should load up the conf files onto the scripts and then everyone can analyze what is causing this.

  40. StuartR
    Posted Jan 31, 2007 at 7:55 PM | Permalink

    I don’t know if it has already been mentioned, but In your road map I reckon you should have some sort of narrative that is punctuated by the validation of your work by the NAS report that in June 2006 (albeit reluctantly) vindicated all your critisicism of Mann so far. And will possibly let any stray visitor know that there is some hope that scientific sense may seep through over time.

  41. David Smith
    Posted Feb 1, 2007 at 5:58 AM | Permalink

    Steve M or John A, an odd event Wednesday – my “favorites” button seems to have somehow reset from this website, which starts with www) to an advertising website (climateaudit.org, which foes not start with www).

    I’m unsure how I did that. It happened during the website crash Wednesday evening.

  42. David Smith
    Posted Feb 1, 2007 at 6:04 AM | Permalink

    Re #41 Come to think of it, a screen appeared which had the left column of your website (the same list of topics, though not the same format). I did click on one of those links and it went to the advertising site. I can’t decide if I somehow did something in error or maybe that was a faux website that somehow changed my favorites button. Probably my error.

  43. Posted Feb 1, 2007 at 6:27 AM | Permalink

    The url climateaudit.com is an advertising site. Steve’s site is climateaudit.org. There’s very little Steve can do about this.

  44. David Smith
    Posted Feb 1, 2007 at 7:04 AM | Permalink

    What’s odd is how the faux site popped up via my favorites button during the crash. I must have made an inadvertent keystroke somewhere.

  45. Posted Feb 1, 2007 at 7:38 AM | Permalink

    Steve: I think he is trying to say that there are times where you have 300+ outstanding HTTP requests at a time. In other words there are 300+ processes running, trying to serve data to 300+ people at one time. This is likely due to the fact that many of these threads are very long, and generating the HTML takes a long time, so while the HTML is being generated for one person, many other requests come in and have to create extra processes to be serviced.

    John A: Have you tried client-side caching? If you can set up your HTTP headers (in Apache, or in WordPress) so that clients will cache a page for a few minutes, this might reduce the load on the server. It’s possible that you’ve already done that, I haven’t checked.

    Steve, John A: My experience with MySQL in the past has not been good. I switched to PostgreSQL and am much happier, although it does require some tuning, but I found it handles concurrent load a lot better. However, MySQL has been developed since I gave up on it, perhaps it isn’t too bad now. It’s probably non-trivial for you to switch database server software, but if you do a fresh install in future, you might want to try some alternatives. For one thing, my database shrunk quite a lot when I moved it from MySQL to PostgreSQL (it compresses data on the fly) and that helped with caching a lot. It (PostgreSQL) also has a concurrency model which doesn’t require locking, and locking was what slowed down MySQL accesses for me often.

  46. JP
    Posted Feb 1, 2007 at 2:16 PM | Permalink

    Steve,
    Have you ever considered a hosted solution like Typepad? I don’t know if they can support the kind of
    archival and graphic uploads you require, but it would cut down on the amount of work you must do
    to keep this site open.

  47. Posted Feb 1, 2007 at 3:17 PM | Permalink

    I’m going to ask again: What are you using to show recent comments on the side bar?

    There are some very poorly written scripts out there that make a bajillion httpd requests everytime the page loads– particularly for a site with lots of comments in the wp_comments table of the database. The code in one of them is so bad, calling the script is almost as cpu intensive as running a climate model.

    If that is your problem, using a more efficient script can make a big difference.

  48. Steve McIntyre
    Posted Feb 1, 2007 at 3:37 PM | Permalink

    #47. Margo, I don’t know. I’ll try to find out from John A.

  49. Posted Feb 1, 2007 at 4:30 PM | Permalink

    I thought to click the “TIGA” theme below. It looks like you used the plugin at freepress.

    I’m not sure, but I don’t think that’s a known bad plugin.

  50. Sidviscous
    Posted Feb 1, 2007 at 4:34 PM | Permalink

    Steve

    If your looking to move providers let me know. friend in Montreal is setting up his own co-location site. plenty of room for you and a big fat pipe. John A can keep administration obviously, and it will be in Canada, which I know is important to you, and a valid concern.

  51. Posted Feb 2, 2007 at 1:58 AM | Permalink

    The comments plugin is called Recent Comments v4. I have turned off WP-ShortStat to reduce the load on the VPS.

  52. richardT
    Posted Feb 2, 2007 at 7:15 AM | Permalink

    Last night I wrote a series of posts on the alternative IPCC summary coordinated by McKitrick which have subsequently been deleted. Since when has it been the policy of this site to censor posts that do not follow the party line? Perhaps the censors did not like the ease at which serious flaws in the alternative summary, written by one of their stars, could be found.

    Alternatively, is the censorship due to ethical consideration of discussing a leaked report? If so this is blatantly hypocritical as numerous posts have discussed leaks of the real IPCC report.

  53. Jaye
    Posted Feb 2, 2007 at 7:55 AM | Permalink

    Nicholas,

    Don’t know what your concurrency issues were but MySQL 5.0 use read/write table locking for large tables (instead of row or col. locking) for concurrent access. I suppose one could create a scheme were row locks were used but in large tables the trade off might be indexing to the appropriate row vs blocking access to the entire table until the write is completed.

  54. Jaye
    Posted Feb 2, 2007 at 7:59 AM | Permalink

    Nicholas,

    BTW, MySQL has two table models that don’t use locking at all. Although I’m at a loss as to how the db can maintain transaction integrity without some sort of locking scheme whether its in the table or blocking the client threads farther up stream.

  55. Dave Dardinger
    Posted Feb 2, 2007 at 8:26 AM | Permalink

    Richard,

    Are you a new poster? I don’t seem to recognize you. Anyway, when a new poster comes and starts posting many messages quickly they tend to get labeled as spam by the spam protection software. After a bit John A or Steve M will come along and sort things out for you.

  56. richardT
    Posted Feb 2, 2007 at 8:43 AM | Permalink

    #55
    Thanks Dave, but I’ve been posting for couple of months with no problem. Only the posts I made about the alternative summary were deleted, not my other posts, and replies to my posts by Jae were also deleted.

  57. Mark T.
    Posted Feb 2, 2007 at 9:10 AM | Permalink

    True, richardT, but you have not been posting very regularly for that month and you’ve likely upped your post count significantly in the past few days (from what I’ve seen, you have). SpamKarma takes a while to “recognize” a new user, and sudden jumps in post count results in deletions. Even if you’re a regular, and you stop posting for a while, then return, the same will happen.

    Mark

  58. Andrew
    Posted Feb 3, 2007 at 9:55 PM | Permalink

    The acronyms need explanations. Would it be possible to sticky a dictionary of acronyms to the main page at the top? Or to include in the style sheet of this page a full definition when you hover over the scronyms? (This would be the best solution) I am well versed in computer acronyms but like any other field if you never heard one used before you have no idea what it means until you look it up. Even still it can be hard to follow until you are used to them. Some examples but many more are used:

    IPCC
    TAR
    SAR
    AR4
    SPM
    NAS
    WG1
    ect…

    I realize this would take some work but it would help incredibly to get the word out to the lay person. The information you provide is invaluable but it is hard to compete with motion picture hysteria that is an “Incovient Truth” if the average person does not understand it.

    Also why do you not link to JunkScience on the left?

    On another note have you guys thought about producing your own documentary?

  59. Steve McIntyre
    Posted Feb 5, 2007 at 9:22 AM | Permalink

    Thanks for the various comments on how to increase blog operating efficiency.

    I can spec the operating problem more closely – it is CPU usage which frequently runs up to 100% of allocated CPU usage. Bandwidth is not an issue. Large CPU usage seems to be a problem with some WordPress installations – google CPU usage WordPRess. WP Cache seems to be one recommended fix. I’ve asked the server to install it on our site as our own installation attempts failed.

    Hans Erren suggested that the threads be broken into pieces a la UKWeatherworld. That’s a good idea. We’ll try to do it, but at this point, I don’t know whether you can do it in WordPress.

    I’m hopeful that WP Cache will mitigate the load and crash problems, but the problem might be trying to make WordPress operate a larger data base than it’s designed to do.

  60. Ron Cram
    Posted Feb 6, 2007 at 11:27 AM | Permalink

    re: 56

    Richard,

    I am sorry your posts got lost. I would chalk it up to the problems the website has experienced rather than to censorship. I’ve never known Steve or John A to delete posts because they were afraid of the science and I would not expect them to start now. I would like to read the alternative summar McKitrick is working on, as well as your comments about it. Can you post a link to the alternative summary and restate your comments here?

  61. Steve McIntyre
    Posted Feb 6, 2007 at 11:32 AM | Permalink

    We’ve disabled the sorting of Recent Comments in an effort to keep the website running.

  62. Posted Feb 6, 2007 at 11:46 AM | Permalink

    John A, Does the Apache configuration use “Rewrite” rules? (Any “Rewrite” statements in the httpd.conf or the htaccess files?

  63. Posted Feb 7, 2007 at 5:37 PM | Permalink

    Re #62

    This new server does. It’s essential for the running of WPMU which powers auditblogs.com

    The old server did not. It’s still a mystery as to why the performance was so poor.

Follow

Get every new post delivered to your Inbox.

Join 3,322 other followers

%d bloggers like this: