1-2 March: Dead Servers Aren't Much Fun

There are several of you out there who host various web sites on my domains, and you might've noticed they're not available just now. They're all hosted on one server, you see, and it crashed at just before eight AM yesterday. Here's a picture:

Oops, I didn't take the picture before I left. Deal. I'll take it when I get back.

It had an unscheduled reboot, and Mike rushed over to the colocation facility where it'd been in exile since our move and pending creation of our firewall (a firewall wards your network against malicious use, basically). The fastest way to repair it would be to bring it home, so now it is.

I booted it up, and... the bootloader wouldn't even load. That's very bad; it means the operating system can't find itself. Seizing upon a rescue disk, I mounted one of Lorien's drives, the one that had held the core operating system. Mostly it was okay, except that /lib, a directory that holds many objects that make your computer go, was totally corrupted. The fastest route out of that was to rebuild the operating system... which we did.

You can tell by the fact the travelogue's still here that the web content survived pretty much intact, as did all users' home directories. I'm still putting the pieces back together, which has been quite complicated by the additional fact that we changed IP addresses in moving the server home from the colocation facility. The thing that's preventing me from a return to complete normal functionality is that the Powers That Be need to know that my server's moved -- for the technical, while I do run primary DNS for all domains except snugharbor.com, the root servers don't know that ns.snugharbor.com has changed IP addresses, and NSI is being their usual turgid selves about acheiving resolution. It's a waiting game at this point.

That's pretty much eaten my brain. It's so eaten my brain that while I was supposed to go to Laurel's and help cook for the Agincourt Victory Feast in San Mateo, and then go to same and cook more, it didn't quite turn out that way. See, three straight nights of short sleep, combined with the fact that things Weren't Quite Right Yet, pretty much made me unfit for non-geeky company (and, truth be known, even geeky company would only have received a recitation of my woes). I shucked fava beans at Laurel's for three hours, helped take stuff to the site, then went home and packed. Trust me, that was the safest recourse for everyone. And hey, I managed to make it to Sportmart and get a pillow and blanket for the train, too.