A cautionary tale of DNS servers

Started: 2010-08-20 08:13:26

Submitted: 2010-08-20 08:33:13

Visibility: World-readable

Last Thursday morning, while I was at work, Kiesa e-mailed me to ask whether I thought our webserver was down. I couldn't access it either, and began to wonder whether something I had done had crashed it. (This isn't out of the question; I run a virtual server from local Linux-friendly provider tummy.com, and while it's more stable now that they migrated me to a new server with more memory and CPU eighteen months ago, I was still able to bring my server to its knees by posting a link to Calvin's website to Twitter before I had loaded it myself to cache all of the smaller-sized images his site uses. A crowd of instant-search-engines, lead by One Riot and Google, pounced on my link and I discovered that my resize-images-on-the-fly algorithm doesn't withstand dozens of concurrent hits.) I found that I couldn't access my home server behind my DSL line either, which isn't altogether surprising given that my DSL modem keeps crashing on me, especially under high loads, and needs to be power-cycled. I didn't really think I could do much while still at work, but when I still couldn't access my server after a couple of hours I took a deeper look. I couldn't resolve DNS either, and dig +trace failed mysteriously after getting my nameservers from the .org servers without revealing why, only that it couldn't access any further nameservers. I dug through my e-mail to find my virtual server's IP address and discovered that my server was still up; I could access it by IP address but not by any of my DNS.

I quickly verified that bind was still running on my server, and it was accessible from the outside world, but it took me a few minutes to figure out why I still couldn't resolve names the normal way. I replicated the results dig +trace gave me, step by step, and discovered that the top-level .org servers were giving the wrong IP addresses for my nameservers. I had set up ns1.festing.org to point to Honor, my virtual server, and ns2.festing.org to point to Ziyal, my home server, but had forgotten to update the IP address of Honor when I migrated virtual servers eighteen months ago. Since then, Ziyal had been inadvertently functioning as my primary DNS server, and when my home DSL went down, my servers were essentially cut off from the Internet, despite still being connected itself. I presumably never noticed this before because I was either at home when my DSL went down, or I had already requested a DNS result and cached it in my local nameservers, wherever I happened to be. (With Kiesa working from home most days, she can power-cycle the modem whenever necessary, but on Thursdays she tends to work from a nearby library.)

With Kiesa unavailable to power-cycle my DSL modem until early afternoon, I figured out how to update the IP addresses of my nameservers in my name registrar's records and waited for the results to propagate across the Internet. By the time Kiesa brought Ziyal back up on the Internet, both my primary and secondary nameservers were up and all was well.