Unscheduled migration

Started: 2016-08-10 21:02:07

Submitted: 2016-08-10 21:04:19

Visibility: World-readable

In which the intrepid narrator begins the laborious process of rebuilding his server during an unscheduled migration

As you may have noticed, my site -- and everything else I serve -- went down on Monday. Not everything is back up; my website came first (mostly because I had the best backups of it), and I'm working on bringing everything else back up over the next few days. If it's later than Saturday, and your see something broken, let me know because I've probably forgotten about it.

The cause of this particular outage was an out-of-memory condition on my VM, which probably ran out of memory because of higher-than-average load to my Apache server that caused it to spawn too many instances and run out of the memory allocated to the VM. The kernel OOM-killer thrashed for some time until my ISP's monitoring picked it up. They rebooted the VM, but it wouldn't boot. It was running an ancient version of Ubuntu, so they provisioned a new VM for me running an up-to-date OS, gave it the same IP address, copied all of my old data into a backup directory, and gave me the keys Tuesday afternoon. I got my website back up and running from backups last night; I'll work more tonight.

I am aware of the irony that, in my day-job, I work as SRE, so I'm responsible for the care and feeding of a very large number of servers in production. My employer (and other cloud service providers) have many services that I might use to improve my scalability and redundancy, and I'm now considering precisely what I want to do in the long-term going forward, but right now the quickest course of action is to resurrect my server and services more-or-less as they were before, then figure out what to do from there.