Tag: hosting

Posted on by — 21 Comments

What a Hosting Provider did Today

I found Dennis the Menace, he now has a job as system administrator for a hosting company. Scenario: client has a problem with a server becoming unavailable (cause unknown) and has it restarted. MySQL had some page corruption in the InnoDB tablespace.

The hosting provider, being really helpful, goes in as root and first deletes ib_logfile* then ib* in /var/lib/mysql. He later says “I am sorry if I deleted it. I thought I deleted the log only. Sorry again.”  Now this may appear nice, but people who know what they’re doing with MySQL will realise that deleting the iblogfiles actually destroys data also. MySQL of course screams loudly that while it has FRM files it can’t find the tables. No kidding!

Then, while he’s been told to not touch anything any more, and I’m trying to see if I can recover the deleted files on ext3 filesystem (yes there are tools for that), he goes in again and puts an ibdata1 file back. No, not the logfiles – but he had those somewhere else too. The files get restored and turn out to be two months old (no info on how they were made in the first place but that’s minor detail in this grand scheme). All the extra write activity on the partition would’ve also made potential deleted file recovery more difficult or impossible.

This story will still get a “happy” ending, using a recent mysqldump to load a new server at a different hosting provider. Really – some helpfulness is not what you want. Secondary lesson: pick your hosting provider with care. Feel free to ask us for recommendations as we know some excellent providers and have encountered plenty of poor ones.

Posted on by — 21 Comments
Posted on by

Decommissioning old servers, saving money…

Of course it’s not quite that simple. I’ve just decomissioned an old Red Hat 7.1 box (hosted dedicated server) that had been in service since 2002, so about 7 years. Specs? Celeron 1.3GHz, 512M, 60GB HD. Not too bad in the RAM and disk realm. It did a good job but goodness am I glad to be rid of it!

Not having that box online is safer for the planet, although it (perhaps amazingly considering the age of some of the externally facing software components) has never been compromised – I consider that mostly luck, by the way, I’m not naive about that. But it’s not easy to move off old servers, it’s generally (and also has been in this case) a lot of work.

Of course hosting has moved on since 2002, places like Linode offer more for less money/month. Of course they virtualise (Xen based in this case) and that’s not been my favourite (particularly for DB servers but depending on the use it really comes down to how you set up the whole infra). It’s a different environment, so different “rules” apply for the optimal setup. The feature/pricing model of the hosting(/cloud) provider actually has more than a little bit to do with that. Distributing tasks like MX relaying, DNS, moderate MySQL tasks, web server, across different virtual machines, with added redundancy across different data-centers, works very well for many use cases. And the funniest thing… more servers, with distributed redundancy, the net cost per month is actually lower than that one single server!

There a many aspects to consider, and I’m intending to write more about that in future posts. I just found it an interesting experience, dealing with this (personal, not even business) server. We handle with these technical environments all the time in our work, but it’s not quite the same perspective. It’s not all technical/financial issues, there’ more to it.

Posted on by
Posted on by — 6 Comments

Your opinion on EC2 and other cloud/hosting options

EC2 is nifty, but it doesn’t appear suitable for all needs, and that’s what this post is about.

For instance, a machine can just “disappear”. You can set things up to automatically start a new instance to replace it, but if you just committed a transaction it’s likely to be lost: MySQL replication is asynchronous, EBS which is slower if you commit your transactions on it, or EBS snapshots which are only periodic (you’d have to add foo on the application end). This adds complexity, and thus the question arises whether EC2 is the best solution for systems where this is a concern.

When pondering this, there are two important factors to consider: a database server needs cores, RAM and reasonably low-latency disk access, and application servers should be near their database server. This means you shouldn’t split app and db servers to different hosting/cloud providers.

We’d like to hear your thoughts on EC2 in this context, as well as options for other hosting providers – and their quirks. Thanks!

Posted on by — 6 Comments