Diary of a downed server
It’s 4:37 am. I’m awake. Why? Kept tossing in my “sleep” thinking about a couple of my sites being down due to a crashed server. Something about a hardware failure. A VPS server that had a hardware failure.
Let’s backup here just about 6 or so hours.
My wife and I had gone out for dinner around 6pm and did a little “window shopping” for houses in our area for some friends who were looking to move out our way. Dinner was good. The house hunting was OK. Reported in with our findings when we got home around 8pm.
We kicked back to watch some TV (I get to see what that is occasionally) and around 10:30 or so, after the news went off, I go to check my site(s), as I usually do periodically and find two of them down. Both are on the same VPS. Not good.
10:37 pm I fire off an email to VPS Support only to learn that my server is the one listed on the status blog as being down due to a hardware failure. Anything else but a hardware failure and I would most likely not be too worried. I’ve heard horror stories about this before so I’m keeping up with things to see how this turns out.
It’s now 4:50 am and the status blog has been updated at 4:30 am. It tells me that the restoration process is about 40% complete now.
Ordinarily, I wouldn’t be to concerned as one of the sites is a hobby site of mine. However, the other one is a client’s. See why I’m a bit concerned now?
I’ll update this post as I learn more.
Gonna be a long day….
Update: As of 7am CST, the restoration process is around 65% complete.
Update: As of 9:52am CST, no news. Status unchanged from the 7am update. 
Update: According to the status blog, at 10:00 AM CST: They are in the process of provisioning a new VPS node to migrate existing accounts not yet restored to. Still concerned.
Update: As of 4:41pm, still no further updates on the status blog. Server still down.
Update: Last one for today. No updates on the status blog since 7pm but after calling a Tech I was told “they’re still working on it”. Nuts!
Update: It’s now 6am Sunday morning. Still nothing and no further updates from the status blog about anything. Fired off another email to see if anyone is still working on things.
Update: Got an email from Support saying that the site is back up! w00t! Not! They toasted my database! No database, no blog. All 100+ posts plus content gone. If I still drank, I’d pop a cold one about now. <deep sigh> Gawd, I hate starting over.
Update: Hopefully, this is the final one. Turns out that MYSQL wasn’t running and once they restarted that things came back up as hoped for. I did lose some posts and some content due to the backup being from the 21st instead of a nightly back-up as told but that’s a lot easier to swallow than the entire thing being lost. <whew> Still looking at that cold one but won’t touch it.
Moral to this story?
First, always have a recent backup copy of your data on hand. Don’t rely on your host for this as it’s a crap-shoot whether they’ll have a recent one or one from a week ago. (I did have one but didn’t have to use it as replacing some images was far easier.)
Second, even a VPS server can go down. It doesn’t just happen to the shared servers. Any of them can crap out at anytime.
Lastly, it can even happen to us designers too. We’re not immune to a server failure although we hope that it doesn’t happen. 
The site I was referring to here is The Mystic Bird.
