The performance of the CA server is ridiculously slow. Last March, I considered moving to a wordpress account, but was persuaded that there were so many embedded links to CA and to CA graphics to move. Thus, we continued on using our own server, which has now collapsed under a load much more modest than WUWT volume, which is effortlessly handled by WordPress.
On an interim basis, I’ll post at camirror.wordpress.com (the logical name having been hijacked by someone else.)
I’ve done a post on a very recent refusal of my FOI appeal by the University of East Anglia. It may seem like a minor event given the tumultuous events of the past few days, but take a look at the chronology.
[The blog appears to be ok again this morning, 22 Nov]
Climate Audit 
42 Comments
Steve, it breaks my heart to see CA sidelined during this period. As a Typepad blogger, I can tell you that they do a fantastic job dealing with spikes in traffic — as, I image, do WordPress and Google Blogger as well. Blogger has the advantage that you can map your domain (in other words, you can still use climateaudit [dot] org. I think the migration issues can be handled by a savvy web guy.
Steve, Comments are set to moderation on the new blog. I doubt that was intentional. I’ve sent an email but am unsure if you are simply swamped with them.
Steve, you can map your domain to a wordpress blog also.
Do your homework regarding Blogger. I seem to remember that during the 2008 US Presidential campaign, a lot of blogs were constantly being disabled by Google because someone reported them as hate content or being untrue or something. Instead of guilty until proven innocent, they would immediately be turned off for a while only to be turned back on later. Being a controversial site, that wouldn’t be good for you.
John Hinderaker of the Powerline Blog has some good analysis of the CRU emails here and here. The second article will be of special interest to CA readers as it reviews the emails on the “team’s” efforts at damage control over the Yamal affair. John’s assessment on the emails can be summed up with this quote:
I have used Drupal for a blogging platform, and there are a number of things one can do to make it scale very, very high. You can probably do some of these on WordPress, since it also uses PHP and MySQL.
If you have not tries these, please do (and ask me if you don’t know how).
1) PHP OpCache (I use eaccelerator).
2) MySQL query cache on (other MySQL tuning would be helpful, depending on your loads)
3) MemCache, block caching, anonymous user caching, etc
4) Javascript and CSS aggregation/gzipping
5) lot’s more stuff, depending on your setup.
If you do migrate, you can maintain all your deep links to keep your pagerank up. That’s just using either the new application or apache mod_redirect.
Good luck, George
Also, check your server logs, to make sure no one is purposely loading your server to make it perform poorly.
Whatever you do, Steve, keep up the good work! Your contribution will soon be seen to have been pivotal in bringing the world back to its senses on this issue.
I’ll be dropping something in the tip jar later today so you can have a dram on me!
Steve,
Do you want regulars to privately email a (very) occasional CRU email#/report for you (as I have just done) on the grounds that hard-to-explain actions at the time have now become clearer? Or would you rather select your own meal from the menu? Geoff.
Why not post a thread at WUWT in the meantime. It’s a bummer not getting your commentary as this unfolds.
The nature of climate audit is sober reflection, reportage of record. Being away from the centre of this storm is probably desirable. There may still be legal action, and or a leak investigation. If anyone can generate the slightest suspicion you have non-public knowledge about the leak, your emails and computers may be seized.
You have enough enemies both inside the team, and in several governments for someone to try a little fishing.
You have, anti -hacking, anti-terrorist, and anti-piracy powers which can be applied, and they are international, and have low standards of suspicion required. I think you will find, that climate comes under national interest , and spying laws may apply.
If you equipment is seized, I would guess the contents would leak. Encrypting your data would not be a complete solution as in many countries you can be required to decrypt it.
Now would be a good time to remove from your computers anything you do not need. You could do this as part of your server move.
Please be clear I am not suggesting you have anything you should not, but I suspect people have talked to you off the record, and may have supplied grey data sets. Even the most minor lack of team spirit would have consequences for them.
I’ve made visits to your tip jar in the past – very modest ones admittedly.
I’d like to see a tip jar for a legal fund to pursue the faulty works at CRU. This to me has moved well beyond science (was CRU ever about science?) and moved into a legal arena, at least that’s where it ought to be now. Taxpayers, citizens, businesses, scientists, etc. have been damaged financially – some quite substantially.
As I mentioned at other sites, I’m prepared to tip much more into a LEGAL TIP JAR.
Re: P Gosselin (#11),
Please Steve, Don’t get sidetracked. I know you will Stick to the Science. Your time is precious…
Re: P Gosselin (#11), If anyone knows how to contact him Lord Monckton might have the pull to get the British government to investigate CRU. He has show a bit of interest in the subject reciently.
Here $50 for the tip jar.
Re: Joe Crawford (#20),
Lord Monckton already knows about this and you can be sure he is more than a “bit” interested. I notified a person I know in the UK about the CRU files on Thursday night and within minutes and he forwarded it to a long list of contacts including MoB. He (my contact) has been very outspoken on AGW, having written well publicised letters to the House of Commons, Royal Society, BBC, etc. People in the UK are all over this issue.
Re: Daryl M (#26),
Daryl and Joe, Lord Lawson was on the BBC Today Programme this morning (about 7.35 if those in the UK want to look it up on iplayer) calling for an independent inquiry. He was quite fair about the leaked emails – he said he maintained an open mind. He also managed to get in the point about temperatures not rising this decade.
Re: Kazinski (#28),
I like this bit,
“And don’t leave stuff lying around on ftp sites”
– isn’t that exactly what they did themselves?
Re: PaulM (#31),
http://news.bbc.co.uk/today/hi/today/newsid_8373000/8373594.stm
The BBC report may be heard at this URL. The specifc URL is at 7:35 about halfway down the page.
Steve, it seems that you need a better setup. Perhaps you should consider multiple servers instead of just one. I know this costs money so I’ve hit your tip jar with $50 to help with the costs.
Seems we’re ok again here…
The blog is back. Perhaps the new posts can be copied here again.
All Your Emails Are Belong To Us
Phil Jones was told not to delete emails:
“I did get an email from the FOI person here early yesterday to tell me I shouldn’t be deleting emails – unless this was ‘normal’ deleting to keep emails manageable!”
http://www.anelegantchaos.org/cru/emails.php?eid=943&filename=1228922050.txt
Steve, I remember you writing about William Connolley asking where a particular figure came from. It seemed an odd thing to me at the time. Does the CRU email here relate to the same figure?
Phil Jones on GISS land surface temps:
“GISS is inferior – not just because it doesn’t use back data. They also impose some
urbanization adjustment which is based on population/night lights which I don’t think is very good.”
http://www.anelegantchaos.org/cru/emails.php?eid=1042&filename=1254850534.txt
And I bet your phone is ringing off the hook with the Globe & Mail, The Star and the CBC all lining up to interview you. Right ?
Ahhhh sorry Steve, too funny to think the Canadian media want to cover this story. Day 4+ and the only thing I can find is a G&M link to the Revkin piece in the NYT. These giys have been so in the bag for the whole global warming hysteria thingy for so long they’ll look like fools now if they cover the biggest story in decades. Trillions of dollars, an entire global economy and international governance changes all over a false premise and manipulated data.
Nice of Revkin not to tell everyone that he is a contributor in the emails as well, a contribution that may wonder about his journalistic agenda.
Phil Jones explains to Michael Mann how wants to coordinate peer reviews of a paper submitted to Science in order to ensure that it is not published, but he also doesn’t want any fingerprints on the coordination, as that would be unethical/unseemly/counter to how science is supposed to be done:
“Can I ask you something in CONFIDENCE – don’t email around, especially not to Keith and Tim here. Have you reviewed any papers recently for Science that say that
MBH98 and MJ03 have underestimated variability in the millennial record – from models or from some low-freq proxy data. Just a yes or no will do. Tim is reviewing them – I want to make sure he takes my comments on board, but he wants to be squeaky clean with discussing them with others. So forget this email when you reply.”
http://www.anelegantchaos.org/cru/emails.php?eid=402&filename=1077829152.txt
Presumably the paper they are discussing is Von Storch/Zorita, but could be another
Lord Mockton is well up to date on these issues, and I do hope he uses his considerable influence to get a legal case going.
Clearly many have been seriously damaged by the CRU and IPCC sheningans. They are all entitled to compensation.
Steve has done the brunt of the heavy science lifting needed to blow this wide open. Now it’s time to move this to the legal arena. So I prefer hitting a legal warchest. The tip jars have served their purpose, unless of course some explains otherwise.
Keith Briffa on Bristlecone pine series:
“I still believe the “Western US” series and its interpretation in terms of Hemispheric mean temperature is perhaps a “Pandora’s box” that we might open at our peril!”
http://www.anelegantchaos.org/cru/emails.php?eid=759&filename=1164120712.txt
Blogger is frightfully slow (I have several blogger and wordpress accounts with nowhere near anybody’s traffic levels). I doubt it can handle your volume. WordPress generally can. You might enquire with WordPress as to whether your site name was highjacked for obfustication. If so, since they seem generally sympathetic, you might be able shake it free. I ran into this problem on Blogger – hijacking – someone monitored a thread I was in where their was a discussion about setting up a blogger account, and logged up the name that same day. They never actually used it and refused all contact. The CA.wp one hasn’t been used since it was set up – get WP to free it up.
Steve: we’re hoping that John A claimed it in 2007 when we moved servers and considered going to wordpress. well find out soon.
In Aus, apart from an uninformed and inaccurate report in the Daily Telegraph ( a gutter tabloid), the mainstream “meeja” have studiously avoided any mention of this
Unsurprising and quite predictable, since the MSM here have long pushed the AGW barrow and legislation is pending in a heated Parliamentary debate
Re: ianl8888 (#27),
Andrew Bolt, HeraldSun
Steve,
I had a specific question about one of the emails that you are referenced in, at least I assume that MM is McKitrick and McIntyre:
Is this the same data, that Jones just “sent load of” that he claimed in August of this year had been lost, back in the 80’s? The same data that he threatens to delete rather that release it to you? That seems awfully damning to me. I’d hope that you’d have a better idea than most of whether he is talking about the same data, in both 2005 and 2009.
And the denials begin – look at who commented on the downplay article and added to it – see the bottom of Wired’s comments: none other than Michael Mann
Ha!! They’re scrabling now that we know outright that they’re manipulating.
http://www.wired.com/threatlevel/2009/11/climate-hack
Steve –
You’ve got a classical situation that demands mirrored websites. Generally these are placed in server farms, never stand alone server.
Yes some wordpress tweaks can improve response time, such as the add ins to convert php pages to static html. As an example, a closed thread can be converted to straight html – why not, it’s closed. It’s nuts to use php to generate that page, and it is one looooonnnggg page, when there are 750 or 1500 posts in it.
But that is not going to help a lot in these classical bottleneck situations due to inbound client requests one or two orders of magnitude higher than normal.
Is it a valid concern or question, whether some hosting service would shut you down at politically sensitive moments? I don’t think so necessarily – an hour or two of work and inquiry asking where (as an example) right wing US websites are hosted should show some hosting services that don’t play these silly games. That is just one example of solution method.
Large server farms route incoming traffic to a selected or random box, example, Amazon, ebay. At the simplest level of homebrew, an initial page could be used for index.htm, this is what initially displays on typing the URL into search bar:
Box #1
Climateaudit.org – select your site
1. Click here for primary climateaudit.org (redirects to Box #2, “climateaudit500.org”)
2. Click here for read only mirror #1, backup as of midnight (redirects to essentially, offshore mirror #1, “climateaudit501.org”)
3. Click here for read only mirror #2, backup as of midnight (redirects to offshore mirror #2, “climateaudit502.org”)
In a practical sense, this particular index.htm would not have to be used except in periods of high traffic.
If you want to discuss further, email me.
Reading and listening to the rationalizations of climate scientists about the current controvery, I thought of Talleyrand’s observation about the French ancien regieme ” They have learned nothing and they have forgotten nothing”. Looking up Telleyrand, I found an even more apt quotation “”Tout ce qui est exagéré est insignifiant.”: (“All that is exaggerated is insignificant.”)” . Climate scientists would do well with a deep study of Talleyrand.
Denial Of Service (DOS) attacks as a cause of slow response time? Just a thought.
Hi Steve,
Keep up the fantastic work! Open Science with Auditing is the wave of the Future of Science.
WordPress allows you to have your own domain name used for any web site you host there. You should be able to set up a third level sub domain name under your own domain “_____.ClimateAudit.org” where you could have for example “mirror.climateaudit.org” or even just “www2.climateaudit.org”. This way you can keep your “climate audit identity” while moving to faster servers.
Just a thought.
All the best,
pwl
Steve,
In order to reduce the load being placed on your site at this crucial time, the best solution (for WordPress) is WP-Cache.
http://wordpress.org/extend/plugins/wp-cache/
Here is the first written word I’ve seen from Lord Monckton since the leak: They Are Criminals.
Also, Lord Lawson announced the formation of Global Warming Policy Foundation (GWPF) in the House of Lords.
Steve,
I think you could get away with some simple optimizations, this one may have a lot of bang for the buck:
http://tantannoodles.com/toolkit/wordpress-s3/
Moving static image files off of this server will also help, I assume your server spends more time sending image data than it does rendering page content, however if it is spending more time rendering page content, a caching proxy server would be the next step, especially for the home page content.
Climateaudit is largely text based, where there are images they are simple jpegs such as are embedded into posts or comments. There are no issues such as media file uploading or downloading which slows it down.
There are large, long page renderings by the php server. On CA these can be 1500 posts. That’s a big job. For one example, “Mike’s Nature…” thread is now 300 comments +. It appears as one long page when clicked.
This can be fixed by Dashboard, Settings, Discussion, Other Comment Settings, then check “Break comments into sections” and pick a number like 50. I personally like the 1500 comment method, but that requires a correspondingly higher CPU workload for that one mouseclick.
If a page averages 300 and this is set to 50, then server load goes down for page retrievals by a factor of 300/50 = 6, which is substantial.
Cheers.
Jeff M:
is a map of CA, realclimate and Watts for the last year.
But now Rush Limbaugh and Glenn Beck are mentioning Steve. That’s going to go on for a while and cause wild swings in traffic. Oh, and various attacks
of course.
I like the addin you referenced, but it wouldn’t help this site. Caching pages would, though. But it’s still not sufficient to handle 100 or 1000:1 traffic swings, that requires either homebrew directing traffic to separate boxes as I described in previous email or server farm doing so automatically.
And replacing php pages with static html, and reducing the complexity and length of php burden on rendering a post.
Coordination suggestion:
Note — I am assuming that the high-traffic situation is still a problem since a good deal of the new content is being posted at the mirror site. With that in mind:
May I suggest that during this period of high traffic, all new content be posted at the mirror site and that a prominent notice be placed at the top of this site that directs visitors to the new site to read the new content.
Just a suggestion.