Akamai Edge DNS goes down, takes a chunk of the internet with it Akamai's Edge DNS service went down on Thursday morning, US West Coast time, knocking over its customers' websites as it fell. As of 0909 PDT (1609 UTC), the status page of Akamai – which sites around the world rely upon to deliver content among other services – said, "We are aware of an emerging issue with the Edge DNS …
This post has been deleted by its author
Ah, but if you don't have a functioning "U" key on your keyboard it would be inaccessible.
Imagine the scenario - yor day's already going badly because yor keyboard is acting p, everyone yo email thinks yo're American with yor talk of "neighbors" and "colors"... and to top it all off, yo can't even figre ot why yor favorite web RLs aren't fnctional...
...which led me to wonder: if a website goes down and there's no down detector to report it, does it make a noise?
Which, for reasons understood only by my inner idiot, led me in turn to: if a bear shits in the woods but there's no one there to see it, does it still make a smell?
Jeez, even _I_ think I'm weird.
Is that it? Rather than promising plans for vastly increased resilience, all we can expect is an "aw, shucks - what am I like!"
I'd prefer, as just one example, an explanation of what type of release and deployment method they're using. Because the frequency of these outages strongly suggests an unwise faith in the Continuous Deployment religion in the most fundamental services the net relies on.
"I'd prefer, as just one example, an explanation of what type of release and deployment method they're using. Because the frequency of these outages strongly suggests an unwise faith in the Continuous Deployment religion"
A friend was involved in setting up early infrastructure at Akamai. They built an architecture that was redundant and heterogeoous to a level I've never seen before or since.
It sounds like you know a lot about Continuous Development. If you identified a flawed deployment process, I'm interested to hear.
My experience is more with networking. At any layer, there is always at least one single point of failure. To the consumer, DNS is layer 3 (routing). To a network design, it's layer 7 (application). The actual implementation is probably something like AnyCast using BGP, which is wonderful stuff but also as complicated as it sounds.
tl;dr: I'm amazed that DNS works as well as it does.
If we consider how much of the internet uses these services and how infrequent these events are, despite the attention each one gets, you could come to the conclusion that the services are actually pretty good at keeping sufficient resilience such that you don't have to worry about them most of the time. I don't know if that's a positive or a negative though--if they were less resilient, then maybe people would have more than one and they could better withstand the failure of one such system.
Perhaps you missed this critical piece of information: "Cloudflare CEO Matthew Prince offered a "don't blame us" sympathy tweet"
The "Bad days happen to everyone" message was from a different company - not the one affected. I'm not sure what sort of information you expect one company to give you about a completely different company's procedures???
Because the frequency of these outages strongly suggests an unwise faith in the Continuous Deployment religion in the most fundamental services the net relies on.
I used to work in Telecom Billing. Those guys looked at their colleagues running the network core working on their CI/CD processes for evolving the cloud core and just said "No. If the network goes down for 24 hours that's certainly a pretty bad day. But if Billing goes down for 24 hours that's real money we can't recover!"
Some things are just too important.
As an affected customer it took our stuff out at about 8:34am pacific time. I checked their status page, everything looked fine but DNS was not after several manual attempts to query their systems. Tried to call support, queue was full. Tried to do support chat, was immediately disconnected (that surprised me I expected to be put in a queue even if I was #8590283 in line), tried to file a support ticket, internal server error. Once I saw that, I hung up the phone obviously others were reporting the issue to their support.
Given their support systems were overwhelmed I'm surprised they were unable to update the status page of their site to show an issue was going on.
They have a community support page, and that didn't get a post till about a half hour into the incident, and they didn't even get to email me that there was an issue until two minutes after it recovered(9:39am for us, email came in at 9:41am pacific time). Same with their status page the outage was going for about a half hour before it was updated.
Don't mind the outage, but would be nice if they could get their status page closer to real time status, should have it updated say within 5 minutes of a major disruption like this?
If companies really cared about a CDN provider going down because it does happen the obvious solution is multiple providers, but not many organizations are up to doing that. Though it's significantly easier than using multiple data centers or for those in public cloud multiple cloud providers. Same goes for DNS providers, nobody is forcing you to use a single provider. If it means that much to you then use a 2nd one(or a 3rd), again it's quite simple (but most orgs don't care enough to do it). I recall noticing Amazon was using Dynect about 11 years ago now for the first time(they were UltraDNS only before). And my Dyn rep at the time said they signed up one Q4 after UltraDNS had a big outage. Seems like today they still use both of those providers at least for their main domain. Meanwhile microsoft is bold enough to rely on their Azure DNS for their main domain.
What happened was someone in the break room unplugged the coffee pot to clean it. Shorted out the outlet, popped the breaker which killed the 4 port switch for the DNS server
Happens all the time, no problem. "Hey, bring that extension cord over here." so someone can trip over it next week.
\S only, don't have a shit fit!
It’s a pity that the robustness that made the earlier Internet so great is now disappearing. We now have multiple single points of failure that take entire chunks of the web down when they go TITSUP.
They’re not monopolies obviously but this much consolidation is clearly (at least to me) having a negative effect.
Don't know what is this "robustness" you're talking about. The early internet was way more unstable, starting with the last mile (phone lines, yay!) down to very little, if any, redundancy on the physical layer. The things is, the early internet was way, way smaller than it is now and thus incidents such as this one impacted very few people, services and businesses.
You're right in that the consolidation you mention has happened, but was unavoidable. In any mature market -all of them, not only telecomms- there is just not enough room for more than a few big players. It is safe to assume that if one of these players starts to under perform in some crucial dimension (reliability, performance, cost...) some other will quickly attempt to grab its place.
Acme:
>We [* * *] can confirm this was not a result of a cyber attack on the Akamai platform.
How would they know, unless they know exactly what caused it and aren't fessing up?
This response would be produced in either case; either they don't know or they do know. Such "confirmations" are meaningless and any reasonable engineer and even a few lawyers know that they can't prove a negative.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
