UPDATE: 3/19 – I have the server updated with the latest distro packages, and now configured as LAMP with Apache2, MySQL5, and PHP. Working on getting WordPress installed and operational next. After that, there is quite a bit of WordPress content, plugins, and configurations to move over as well as to recreate a number of folders and files from the original machine. It will be a few days yet before a move to the new hardware is ready to attempt. – Anthony
—
Thanks to the generous donations of many people here, I was able to purchase a new fully configured (and guaranteed for 3 years for parts) Linux based server to be the new home of CA. It is shown below after I first fired it up to make sure it survived the shipping:
For anyone interested, here are the specs:
Intel SR1500ALR 1U server platform
- Memory 2 GB – 2 x 1 GB DDR2 667 MHz ECC Registered Fully Buffered
- Operating System Ubuntu 8.10 Server Edition 32 bit
- Optical Drive CD-RW / DVD
- Power Supply 600 Watt Power Supply
- Processor Quad Core Intel Xeon E5410 2.33 GHz 1333 MHz FSB 12 MB L2 65 nm
- RAID Battery Backup no RAID Battery Backup
- Hot Swap 3ware Hardware RAID 5 – 3 x 250 GB Seagate Barracuda drives
Due to the number of fans, it sounds like a jet plane. Lots of air moving – a good thing. This machine should be able to handle a fair amount of load and at the same time be more reliable.
I’m now working on getting it configured and ready to put WordPress on. Once this system is up and running in parallel with full backup data from the original, I’ll drive to the CoLo and do the swap. The downtime should be minimal.
With the extra cash sent, I’m purchasing some extra cold spares and an external USB drive to use for off-chassis backup via a cron job. We’ll also setup an offsite backup as well.
Again, thanks to everyone that helped out!
Climate Audit 
54 Comments
Exxon Mobile spares no expense. Take that Fenton communications.
Re: jeez (#1),
LOL!
Anthony, thanks for the update…hope everything goes as planned….I’m sure Steve is really appreciating this also…
If that is correct, it is a mistake, IMO. Any power outage can lead to corruption on the RAID set.
I would have gone for more memory, as well. Even my home laptop has 3GB, but then I do software development.
Re: Richard Sharpe (#4),
I opted not to get this because there are already two levels of standby power at the CoLo. A humongous UPS and backup generator which kicks in in about 10 seconds. – Anthony
Re: Anthony Watts (#12),
OK, that is fine if there is already backup power. You are less likely to see such corruptions then.
What’s the carbon footprint on that thang?
Why 32-bit Linux and not 64-bit? It seems like a waste to buy a 64bit chip and then use a 32-bit OS.
Also I’d back the observations of Richard Sharpe: There needs to be a RAID battery backup and you should upgrade to 4GB of RAM.
Re: John A (#6),
Because I have reports of issues with WordPress on 64 bit Linux. Going for ease of install, support, and reliability – Anthony
It really doesn’t matter, 32-bit vs. 64-bit with the OS, in any Linux application (even Winders) I’ve ever used. The chips are almost all 64-bit now anyway (and have had more than 64-bit math for a very long time), so it is not unusual to be using 32-bit OS with 64-bit chip. The 32-bit OS probably has better support in the long run as well.
Mark
Re: Mark T (#7),
Actually, windows 2008 is the last 32-bit windows. Likewise, 32-bit linux is the redheaded stepchild .
I’d be tempted to run wordpress in a Vmware VM (using free vmware server under 64-bit linux). Then you can
move your site to another box simply by moving the vmx file.
Does anyone else find this comment window incredibly slow?
Re: scott lurndal (#13),
Immaterial to my point. The OS does not need to match the chip, i.e., there is no “waste” (the biggest impact is addressing, since the math has been 64-bit for many years). Eventually 64-bit will be the norm, but it is not now, and will not be for a while. I’m guessing at least 4-5 years before the majority are using some form of 64-bit OS (and in Linux land, there are servers still running 1.X versions simply because they work). In the mean time, there’s a much larger library of information out there in net-land for 32-bit OS apps.
Mark
No Anthony, thank YOU.
I see my 40€ looking splendid.
Great.
Miguel
Richard,
ECC memory can be a bit pricey. I do agree wit you that an UPS would is important. Perhaps thier web host provides an UPS. The only other thing I would have added is a simple USB external drive. A simple script set to run once a week at say, Friday midnight could back up the database and other mission critical data.
Data center scale UPSs have been known to fail spectacularly. (short circuits vaporizing large copper bus bars. that sort of thing). And backup generators sometimes fail to start. And transfer switches sometimes don’t, disconnecting the equipment from the redundant power.
8GB of ECC memory for an E5410-based system I just built cost me all of $131.96 before sales tax.
Test the window speed–seems ok so far.
Anthony: you are there making the decions, do what you think best.
(I’ve been wary [for more years than I want to relate} of stacke4d UPS due to the way UPS’s work and the harmonics questions. And if I were going to put in more meor, I’d make sure the banks are symmetrical. Or at least make sure I knew how the hardware assigns addresses. The term for the issue escapes me but it has to do with being able to access adjacent addresses simultaneously.)
You are doing fine, Mr. Watts.
Most the hits for things like the front page can be served from cache, and even with system with 0.5 – 1GB one could probably cache most or all of the fairly static pages on the site. For a blog, the access is not really random either, as there is a high degree of commonality to what every patron is viewing, which makes the set of things which need cached/accessed fairly small vs the total. A few users will wander off into the archives, but not many overall. For dealing with comments, searches, and other DB related chores, one would almost always get disk bound first. The short version is that it depends a lot on the application, but one doesn’t necessarily need a lot of RAM for many types of web serving, and having it, while it won’t hurt, also won’t necessarily lead to better performance. Only way to say for sure though is to benchmark and check for bottlenecks.
FWIW, 32b code will normally be faster than 64b code on a 64b processor, all else being equal, as it uses up less space in the processor’s cache. However, the compiler may take advantage of the 64b CPU’s extra registers and instructions when compiling for 64b, which, while technically available even in 32b mode, probably weren’t used for compiling the distro in order to ensure compatibility with earlier processors. Of course, if one intends to use tons of threads or address >= 2-3GB of memory per process, 64b is necessary.
And the window seems pretty snappy, actually.
Oh, btw, don’t stick your swap partition on that RAID5 array if there is anyway to avoid it. You might consider RAID 1 + hot spare (which, unlike the last system, the 3Ware should handle that properly) instead of RAID5 if you can’t avoid it otherwise.
My last comment though, as there are bikeshed qualities to talking about other people’s system setup.
Anthony:
Thanks for all you do. You made a good decision on the server and Linux. I have heard from other Linux enthusiasts that the problem with 64 bit Linux is not the operating system, but the libraries and some applications. Not sure which ones. They will eventually come around, but, the goal in this case was not raw processing speed, but, reliability. You found the combination.
I enjoy operating with Ubuntu Linux, also. I also use Linux on my Godaddy server.
bb
All of my browsing is on a pc running Kubuntu in dualboot with Windows. I am in Kubuntu 95% of the time but I keep windows as a backup to run a few windows programs for picture transfer/editing and for my wife and daughter to use when I have to work on their pc’s. I have stuck with 32 bit out of habit. I wonder how many here use Linux on their own pc?
Oh yes, thank you Anthony for your work and for the occasional glimpse “behind the curtain” to see how all this comes about. Very impressive.
Don’t forget to install it right next to the CRS. 😉
Sweet setup Anthony.
Glad it has lots of fans to assist in the launching and cool operations. Make sure to install a system health monitor to keep an eye on the temperatures of the components to collect the local weather into climate reports! It’d be cool to see the graphs online! 😉 No hockey sticks if you can avoid it.
All the best.
So, Anthony,
What’s the estimated cost for wattsupwiththat.com to have its own dedicated server ?
Please let us know so that we can contribute ASAP.
Re: Gardy LaRoche (#24),
About the same…perhaps just a little less. But that is a project for another day. – Anthony
PWL:
“Glad it has lots of fans to assist in the launching and cool operations. Make sure to install a system health monitor to keep an eye on the temperatures of the components to collect the local weather into climate reports!”
In my experience, the temperature information you’ll get from the onboard monitoring on a commodity motherboard is pretty useless. It may tell you that the temperature is going up or down some of the time, but nothing more. It’s probably better than what we can expect from the NOAA, but that’s setting the bar rather low I should think. If you really want to know what’s going on temperature-wise for a component in your computer room, someone walks down there with a handheld IR thermometer and points it at the back of the rack. I guess there’s a climate related analogy in here somewhere? 🙂
I know your post was tongue-in-cheek, but I just couldn’t resist 🙂
OA
I tried running 64 bit Linux for a while but got sick of broken/missing packages that work fine on 32 bit. Unfortunately the 64 bit ports just aren’t as well tested or maintained. Drivers were a problem too, often it takes months for 64 bit drivers to catch up with the 32 bit ones in terms of compatibility and such. It was much more of a hassle, and I didn’t think the benefits of 64 bit (which are more than just the additional address space) were really worth the stress. I have run some 64 bit servers at work with good success but they were only really running one thing (database) so there was less to go wrong. I think Anthony probably made a good choice here.
#29 Nicholas,
While I have seen the 64-bit problem with other distributions such as Red Hat, Ubuntu has a full 64-bit version for the mainline applications but some community supported apps might have only 32-bit versions available. Ubuntu is the best thing since rolled toilet paper … and that is sayin’ somethin’
I have experience with Suse and Cent/RH. Unless you need the precision and memory space, some of our simulations do have that requirement, life is a bit easier with a 32 bit install. Even some of the base libs can be a bit flaky in their 64 bit incarnations.
I’ve also had a lot less trouble with 3rd party software using 32-bit linux, there is no reason to use 64 bit unless you have > 4 gig RAM.
But which is cause and which is effect? Most people don’t configure 32-bit OS installs with more than 3GB (or possibly 4GB) because they know they can’t use it.
But anyway, there’s no particular rush it seems to me if app issues are still problematic. There’s always next year’s mid-life upgrade!
The burning question is What’s its Carbon Footprint?
adoucette:
Huge I hope!
With the cooling we have now, any warming would be welcome.
crosspatch : Funny you should mention that, I installed Ubuntu 8.10 64 bit and had lots of problems with it. For example, the compiler was buggy, audio didn’t work, it didn’t detect my video card properly, and some other things I forget. Once I had upgraded everything to the latest version (kinda hard without video, so I had to fix that first by going into safe mode), it started working better.
People keep saying things like “there is no reason to use 64 bit unless you have more than four gigabytes of RAM” so I might as well point out why this isn’t true. Firstly, any more than about 3GB and you can’t easily address it all because of the way virtual memory is used, so it can help if you have 4GB total. Secondly, in 64 bit mode the CPU has access to more registers and some other features which can make it faster. However, you have to balance that against the fact that pointers take up twice as much RAM and cache. So there are advantages to it aside from the extra address space, but IMO they are outweighed by the inconvenience of bugs in the software. When all those bugs are worked out then it may be worth switching.
Re: Nicholas (#35),
I’m not sure this is true. For every processor I’ve ever worked with, the register set is a given irrespective of the OS that is running behind the scenes. There are typically less than a few hundred registers anyway, so there’s no addressing issue.
The rest of what you said, however, is absolutely true.
Mark
Mark T, it is true.
Intel 32 bit registers: http://www.pouet.free.fr/spip/IMG/html/x86reg.html
(Nine, plus segment/page table registers)
AMD 64 registers: http://www.logix.cz/michal/devel/amd64-regs/
(Seventeen, plus segment/page table registers which are not shown)
It’s the new instruction format in 64 bit mode which allows the addressing of the additional registers. Keep in mind that several of the original nine have fixed uses (primary the stack pointer, base pointer and instruction pointer) so the extra eight can really help. x86 is a very register starved architecture, most have 16 or 32.
In other words, there are six registers that are generally useful in 32 bit mode (EAX, EBX, ECX, EDX, ESI and EDI) and those same six plus another eight in 64 bit mode.
Re: Nicholas (#37), Those are HW constraints, not OS constraints. The 64-bit ABI is accessible with a 32-bit OS, btw (n32 for Linux).
Mark
I also don’t understand why aren’t you going fully 64 bit.
Re: Luis Dias (#38),
Because the main application which runs Climate Audit, WordPress, reportedly has issues with 64 bit Linux. Also, we use a number of plugins that rely on a number of libraries, many of which also have issues with 64 bit. The decision to use 64 bit was considered, studied, and rejected for this reason.
Ease of setup, maintenance and reliability is the goal. Using 32 bit Linux for this application is the correct choice to ensure these things. If we were just running a file server 64 bit would probably be better, but we aren’t, we are running a series of dynamic applications including MySQL, PHP, WordPress and a boatload of plugins.
I really think you should do RAID 1 across all three volumes rather than RAID 5. It will write much faster, it will read much faster, it doesn’t “waste” a volume as a hot spare does, and you can always convert to RAID 5 if you need the extra 250GB storage later on (probably while the server is still running with the volume still live).
Um, n32 would be for the MIPS, which is admittedly, the processor I have the most experience with (though currently I’m working the PowerPC 8641).
Mark
Raid 5 will read faster than Raid 1, but will write slower. Since there are a lot more reads than writes at CA, Raid 5 is a better solution for serving web pages. Raid 5 is also more disk efficient, as you would need 4 Raid 1 disks to equal the storage of a three disk raid 5 setup (due to fully redundant stripe, Raid 1 needs even #’s of disks 2,4,6 etc). Thus Anthony’s set up in Raid 1 is 250 GB storage, vs. 500GB in Raid 5. The extra space will be helpful for storing database info for running scripts and the like. Both Raid 1 and Raid 5 are tolerant of single disk failures. Rebuilding the array in Raid 5 is more complex, but nothing that a hardware controller could not handle. Raid 5 is a completely acceptable configuration here and I expect this machine will run like stink. Don’t know if there are any extra memory slots on the MB, but another 2G wouldn’t hurt, and its a relatively cheap upgrade.
Re: hswiseman (#43),
Sorry to be a stick in the mud but RAID 5 does not read faster than RAID 1. RAID 5, which to read all n member disks and then check the parity to serve one page, cannot read faster than RAID 1, which is able to read an independent page off each mirror member and serve them simultaneously. You can build a RAID 1 with any number of disks greater than 1 and for n disks the read speed is n times the write speed. The reason for RAID 5 is because it is more cost efficient; but it is less reliable and less fast than RAID 1 — but do you really need 500GB for a blog? I admin a SQL server database with a 300 GB store (storing financial tick data) and that is a HUGE database.
There’s a nice write up on wikipedia.
Re: Stat Trader (#45),
I think the emphasis is on resilience to hardware failure rather than actual read access speed. I’d agree with you regarding the actual disk capacity is overkill (and then some). Steve stores a lot of documents, scripts, presentations and data which are not in the database but are offered as resources to the scientific community. I expect this will continue and grow in the future.
I just checked and the space taken by the Climate Audit www directory is 1.6GB
Thanks for all your hard work Anthony
Nice work Anthony! I probably should have suggested SliceHost rather than an EC2 instance. For your CoLo costs alone, a Slicehost “slice” would have given you the same capacity – hardware hassle free! Out of curiosity, how many dependencies does the CA WP instance require that you can’t rysnc the directories straight to your new server along with a mysqldump? BTW – I have hardware envy! 🙂
Regards.
The reason I’m plugging on with this is because of the reliability issue. With RAID5 on three disks, if you loose two disks you’re dead in the water. With a three way mirror (RAID1 on three disks), if you loose two disks you’re still live. A three way mirror will also read 50% faster than two way with a hotspare and over 50% faster than RAID5. The *only* reason for choosing RAID5 is for economy when you have to store a lot of data — which you don’t — and you’ve already purchased the disks. Plus, most modern RAID controllers should be able to reconfigure from RAID1 to RAID5 while the volume is still live — so if you end up needing the storage you can get it.
Re: Stat Trader (#48), I agree. RAID 1 on three disks will provide more reliability and better performance at the cost of usable disk space. Given the size of these drives I think that’s a reasonable exchange. I can’t see CA needing more than 500Gb over the next couple of years.
I appreciate the input, but the system is already configured for RAID5, and I’m loathe to go back to ground zero now.
Fair enough, but please think about it next time. I’ve had many issues with RAID5 sets going bad, and they are slooooow.
I design, direct, and signoff on the certification testing of some of the most advanced weapon system avionics and avionics laboratories on the planet.
And I say YOU SIR!, are such a geek 😉
Thanks Anthony, The efforts are VERY appreciated.
well, at least this is one post on CA that is troll free!