Re: wth is it with always dns?
It makes me feel like "it's always DNS" is like the folks who try to blame the network for every little problem when it's almost never the network either(speaking as someone who manages servers, storage, networking, apps, hypervisors etc so I have good visibility into most everything except in house apps).
I don't blame the network. DNS isn't the network, it's an app that allows wetware to make use of a network. If DNS is down, the Internet isn't, it's just those cat vids have gone into hiding.
But I have in the past had good cause to blame DNS. Usually when the pager went beep while I was dreaming of new ways to torment sysadmins. Mainly because they controlled the ping to beep software and decided that inability to ping the domain name of a DNS server meant it was a network problem. I'd ping the server by IP address, it'd respond and I'd get to try and wake a sysadmin. That particular issue was eventually solved by a combination of billing for overtime, and creating a shadow ping box that would ignore those, test for both network and app reachability and wake the right person. Sysadmins may like to think they run the network, but that's a neteng's job.
But I digress. DNS isn't my speciality, but curious about a couple of bits. Like why it would be necessary to restart servers for a DNS change. AFAIK that's still a thing if you want to change a server's IP address, but I'd have hoped that in the 21st Century, that could be done on the fly. Then the good'ol push vs pull isuse, like manglement not always understanding that DNS changes aren't pushed. So usual routine of dropping TTL ahead of changes, and hoping resolver/client caches play nicely. And why stuff fell over under load. By stuff, I've seen issues where anti-DDOS systems have caused problems when DNS activity increases due to TTL being lowered, but a decently configured DNS setup should have been able to cope. If not, bit of an oops in Salesforce's capacity management.
Sysadmins again.. :p
Biting the hand that feeds IT
