Explosive Hydrogen

Started: 2005-07-29 20:06:18

Submitted: 2005-07-30 23:35:37

Visibility: World-readable

At this moment, I'm on seat 13F on an Airbus A320 (tail number N478UA) bound for Ontario International Airport, about sixty miles east of LAX. This is "United for Ted" flight 1441. There's not much room in sardine class for notebook computer use; the worst part is that the armrest is digging into my elbow. But putting it down fixes the problem, so at least some things are easy.

Kiesa and I are heading to the Greater Loma Linda Adventist Megalopolis, where her brother is getting married on Sunday. Then we're flying up to San Francisco for three days of paid time off. We have no real idea what we're going to see in San Francisco; odds are we'll read the tour books we checked out from Longmont Public Library on the plane from LAX to SFO.

Tuesday was a bad day to forget my mobile phone at home. Before leaving for work, I discovered that Hydrogen, the primary server for a certain web-hosting company we all know and ... cherish (it starts with an "x" and ends with a "13"), was down. My day involved showing off our latest flagship product to friendly customers who were significantly less antagonistic towards the fact that it wasn't yet ready for prime time than I thought they'd be, and a fascinating lunch at Tokyo Joe's which included a twenty-minute rant complaining that the western world should declare war on Islam in its entirety; every once and a while I noticed that Hydrogen was still down and wondered what Scott was doing about it.

It turned out that Scott was leaving messages on my mobile, which I discovered when I called Kiesa at 2100 after a business dinner at bd's Mongolian Barbecue in Boulder. I called Scott, who was quite glad to hear from me. He gave me the saga so far (Hydrogen's on-motherboard IDE controller apparently died during a software upgrade and reboot last night. Scott's attempts to fix the problem, assisted by two of his friends (including his PHP ninja) were unsuccessful. They attempted to plug the twin drives into a backup computer (featuring a faster processor and all sorts of goodness), which went ok except that the custom kernel didn't have drivers for the new machine's hardware, and their attempts to compile the appropriate drivers were failing horribly; they were stymied by kernel panics at every turn.

I downloaded and burnt the latest stable kernels in the 2.4 and 2.6 trees, raided two doorstops for PCI network cards in case I needed to swap one out, and headed down to Inflow's Denver 3 data center, ready for a late night of mercenary fun. The data center guy buzzed me in, exchanged my photo ID for a visitor badge, and took me back into the rows upon rows of little servers in big racks to where Scott, Logan, and Steve sat hunched around a little keyboard and monitor cart in the aisle. I took over the console and started playing. I quickly discovered that the modules were corrupted; blowing away the modules directory let me run lilo to install my new kernel, but for some reason my new kernel wouldn't boot until I wrote the MBR to /dev/hdc, and then it kernel panicked; I rebuilt the entire kernel and was able to reboot to my new kernel, complete with the network card driver support I needed.

At that point, it was 2300 and I had completed the hard part; the rest of the cleanup involved making sure the kernel had the appropriate drivers for the motherboard's IDE controller (without the drivers, it lacked DMA support, which wasn't good for load), convincing the software RAID to do what it was supposed to do, and verifying that the network aliases Hydrogen required were running. The cleanup took longer than the actual hard part; I finally declared victory and left Inflow at 0130.

I'm glad fixing servers isn't my day job, but it's fun every once and a while.