Arguably this sort of "bring the Internet to its knees" attack was pretty much inevitable.
And arguably that has been the case for most of a decade.
Right now a lot of companies must be quaking in their boots wondering what's next.
An extraordinary, focused attack on DNS provider Dyn continues to disrupt internet services for hundreds of companies, including online giants Twitter, Amazon, AirBnB, Spotify and others. The worldwide assault started at approximately 11am UTC on Friday. It was a massive denial-of-service blast that knocked Dyn's DNS anycast …
It seems a lot of people are talking about DNS in the moment ... but we must not forget who really is to blame - it's the stupid security programmed into may IoT's because of greed. The cause for this problem are not the inherent problems of the DNS system(s) but the stupidity of device manufactures like Dlink, Netgear, Avtech and so on.
Routing works just fine during a DNS outage; the problem is that you can't find the addresses that you want to route to. DDoS against DNS authoritative servers has always been scary. If only every ISP supported ingress filtering, as they're supposed to, tracking down and killing DDoS bots would be that much easier.
Well, yeah ...
Bottom line is that its the current DNS process, which is in place to ensure that the domain name issuers get their little piece of the pie, which make attacks like this possible. In the good old days the ENTIRE DNS data files/database was mirrored on a number of servers around the planet ... so a DOOS would have to hit them all to cause things to fall over this badly.
These days, all the hacker has to do is find out which commercial Domain Name provider provided which mega huge internet presence with its domain names ... and just hit that server. If anything this attack should result in a number of DYN clients going elsewhere (presumably to a less visible DNS provider)
What should happen is that ICANN points out that the current DNS verification and validation processes (which are only in place to protect the IP of the DNS provider) actually make it easier for the Ungodly ... and that perhaps total replication of the database across mulitple provider sand locations might be a good idea to revert to.
But that's unlikely - because nowadays ICANN represents the DNS providers.
"In the good old days the ENTIRE DNS data files/database was mirrored on a number of servers around the planet ... so a DOOS would have to hit them all to cause things to fall over this badly."
I was wondering the same thing. Where are the master root servers these days? I take it they either no longer exists or people like Dyn don't bother with them.
The Internet does route around damage, but this isn't an attack on routing. It's an attack on a network service. That's rather a different thing.
However, it's certainly true that far too little effort has been put into fundamentally hardening network services of all sorts against these sorts of attacks. Unfortunately far too many Internet protocols and services are built around assumptions of good behaviour.
"Routing" has many more meanings than "IP routing". The Internet was designed as a distributed systems - kill a node, it would still keep on working. Now we're turning it into a big centralized system where a few big data centers, the grandsons of mainframes, hold everything. And when they become the easy target of huge distributes attacks like this, they are kaputt... the old saying "when you put all your eggs in one basket...".
If all those DNS records were widely distributed, good luck to take down all of them.
Many ISP's - to name BT for one but probably most, simply do not respect published TTL's.
We've seen issues where we set low TTL's during a site migration but the ISP's simply don't go back for another lookup. You end up having to fanny about with DNS providers until enough time has passed for the providers DNS to re-do the lookup in its own time.
Good question, that commentard: "with really long TTLs, how do you manage regional or global load balancing".
https://en.wikipedia.org/wiki/Anycast - Anycast addresses. This is what Google's 18.104.22.168 and 22.214.171.124 and OpenDNS 126.96.36.199 and 188.8.131.52 use.
I suspect I've answered your question as posted but probably not what you intended or the full story.
Anycast is not suitable for TCP since TCP is dependent on all packets of a connection going to the same place. It might very well work for specific applications and short-lived connections but it's definitely not something you want to deploy on a website that's supposed to be reachable by 100% of all users.
You can even encounter scenarios where, for some subset of users, it works very poorly if at all - equal cost multipath for example, where every other packet ends up at a different anycast instance.
Except that it isn't a good question as the original comment did specify that a prerequisite would be that people didn't use DNS for load-balancing.
People are using something for something for which it wasn't designed to do and have found out that it's not really suitable for it? Who Could Possibly Have Seen That Coming?
...with really long TTLs, how do you manage regional or global load balancing? Failovers? Switching records in general? Migrations?
You can do it by not using DNS to switch between sites. Instead you have one or more load balancers with fixed IPs which you point the DNS at, and then redirect the traffic to the sites and servers as you want.
We do DR failover this way, as well as load balancing and migrations between hosting environments.
There is a slightly increased latency, obviously, but not enough to impact normal traffic.
TTL is nothing really to do with it. Sites would go offline under sustained attack sooner or later.
The main issue here is that these large companies are doing DNS wrong on a more fundamental level. We learned years ago that people were attacking DNS providers and this could be leveraged to take out fundamental infrastructure and sites of all sorts of sizes. The fix is obvious and it's something I've recently pointed out to github:
If you're an attack target do not just use a single DNS provider. Use 2.
If you do that it's much easier to not be caught in the crossfire. It's also much more difficult for adversaries to take you out via DNS - they have to take out two entirely separate networks to achieve that requiring double the attack assets.
The internet was designed very insecurely but they did build it in a way that made it easy to mitigate attacks like the one today and everybody running DNS services at the companies that were taken out look like complete clowns in retrospect. It's like the people who expect AWS zones to be up 100% of the time despite them not being designed to be survivable and Amazon giving people the tools to not do that.
Also fwiw using anycast to balance large sites is a really bad idea. If anycast was a solution to the problem we wouldn't be sitting here talking about anycasted dns providers being taken out.
"The internet was designed very insecurely"
Security wasn't a consideration at all in those days.
Fundamentally everything we have security-wise is a bodge. Ultimately no matter what security mechanism one contrives, it always boils down to the following. Machines are hopeless at identifying people.
The security thing wasn't really a complaint, just a fact of life. We can we rebuild, we have the technology - though I wasn't really arguing for that. I wouldn't mind burning UDP to the ground though but it's an entirely separate subject.
TCP is dependent on all packets of a connection going to the same place
TCP anycast is a thing (indeed it's how a lot of HTTP DDoS protection works). Doesn't mean it's a sensible use of resources when your DNS provider can do useful things for you; it's all cost/benefit - DNS providers are cheap, anycasted HTTP isn't. As I said it's not really a solution to the problem, not relying on your singular provider's servers in the case they get hit or plain just go down is.
Yes, it can be done, but it's not a general solution the way DNS anycast is.
When anycasting DNS, you can just plop down anycast instances in whatever locations will host you, with no special routing policies in place, no prepending/community games, and having it exported as widely as possible.
> If you're an attack target do not just use a single DNS provider. Use 2.
If *you* are an attack target, it is *your* infrastructure that is going to be targeted, not the DNS providers (they may throw that into the deal as well, but expect your own infrastructure to become suddenly popular with IP cameras and stuff).
it is *your* infrastructure that is going to be targeted, not the DNS providers
Yeah but TCP attacks the average toddler can deal with, they're blatant and they're easy to identify the source of and can be mitigated quite quickly. UDP attacks against DNS infrastructure are very difficult to deal with which is why they're popular for taking out large targets - and regardless of that "you" as the target can mean that you're one of many large US sites and the attacker would be happy to take you out as collateral.
"If *you* are an attack target, it is *your* infrastructure that is going to be targeted,"
For some values of "you". If "you" means the US internet business community then DNS is part of that infrastructure and, from what's happened, appears to be a single point of failure for quite a large portion of "you".
It's mainly an attack on port 53, right?
What if DNS servers were able to agree on a different, free port as part of the protocol? There would be no telling which port would be used, and an attacker would have to scan 65535 port to find it, right?
Any client scanning all ports would be easy to identify.
(This was a brain fart from someone who can hardly configure a Cisco router.)
It's not a question of scanning or whatever, the attack was against a "shared" DNS provider where all these sites were using common infrastructure. It's not as if you can "hide" your DNS server because resolvers have to be able to find them, so they have to be pointed at from the parent zone servers.
Hopefully, all DNS sites will start caching; I wish my computer would cache the IP address of sites I visit so that I wouldn't even notice a DNS failure - it could even warn me if an IP address changes, to help prevent IP spoofing.
Anyways, I switched my DNS to that given in your article, and I could connect to the RuneScape servers once more! Quite saved my morning. Also I was reading old issues of U&lc, and that too was restored.
Ayup... mid to late 90's with NT DNS(!) servers, then later with Solaris x86 Bind DNS servers, when setting up new websites, you always had to stop Internet Exploder, close Netscape, restart your DNS client, just to check to see if the scripts took...
I'm sorry, I don't know where I was going with this -> wife pops in with 'Oh look, a portable ski lodge!'... thought evaporated.
"DNS already does caching"
Not really - each record has a Time To Live (TTL) in seconds. Your DNS server should honour that but it can go a bit mad when people ignore the standards to fix things.
For example I've just looked up github.com via 2001:4860:4860::8888 (Google public DNS - IPv6) four times in quick succession and got the following TTLs: 117, 160, 144 and 18. I then looked up the NS records (AWS, four NS records) and looked them up there - now the A records round-robin between two IP addresses and with a TTL of 300.
The world can be a nasty place
Completely normal, expected, and perfectly TTL-obeying behavior.
DNS resolvers don't reply with the TTL as originally specified in the response from the authoritative DNS server (i.e. what the guy who set up the domain specified) - they respond with the time remaining until the record expires from their cache.
Your 4 queries ended up at 4 different servers. Google's DNS servers exist at multiple locations with the same IP address - this is what's known as anycast instancse. Each anycast instance then consists of multiple actual servers behind a load balancer. The load balancer just picked a server at random for each of your queries. Nothing mystical about any of this - everyone big does the same thing in almost exactly the same way, including the root servers.
These 4 servers had cached the record (i.e. received a query for it when it wasn't cached and subsequently looked it up and stored the result in the cache) at different times. Hence different times remaining until expiration in their caches.
> Hopefully, all DNS sites will start caching; I wish my computer would cache the IP address of sites I visit so that I wouldn't even notice a DNS failure - it could even warn me if an IP address changes, to help prevent IP spoofing.
I have local DNS server running in cache mode on all my computers - desktops and servers. Theses are all Linux machines. IDK if Windows has that capability, but I think the default configuration for Ubuntu is to run bind as a caching DNS server if it is turned on. So then I have my net config is using 127.0.0.1 as the DNS source, and my bind configuration uses 184.108.40.206 plus another one.
One additional benefit is that when I'm on a cable connection this bypasses the cable company's default DNS that it sets up in my cable modem's DHCP config, which they use for various nefarious purposes such as inserting their own ads in websites, selling my traffic info, and "fixing" domain name typos by routing to their own advertising sites. I've seen all of those tricks at various times when visiting people who use comcast or optimum.
You know, this is really enough of this crap. What's the point? It's like the assbags that went out and wrote viruses and sent them around to screw up computers of people they didn't even know. What was that for? And now why try to bugger up the whole Internet?
I'm not smart enough to figure out a solution (and there may not be one), but it seems to me that something should be possible.
Technically, we need some geniuses to figure out a way to trace this crap back to its source. Politically, we need international treaties which provide that anyone who screws up the Internet, regardless of where they are, will be arrested and tried for it. Once found guilty, give them a nice, LONG prison sentence. And maybe a permanent, non-dischargable judgment (for many millions) that follows them around for the rest of their life to make sure they're pauperized to the point they can't AFFORD a computer.
You raise a good point with: "What's the point?". Perhaps a live fire test of the botnets? A warning? Not sure from here.
With the IoT crap getting whipped into botnets, this could be a harbinger: "Remember this? we'll pay us or you're next."
Or a state actor group flexing it's muscles as a warning....?
I just don't think it's being done for fun.
ISPs could do a lot more to target bots. I've reported bots innumerable times with clear, comprehensive evidential data and had the reports totally ignored by big-name ISP's that were utterly clueless & didn't want to know so now I don't even bother... Time for large penalties for hosting bots, methinks.
If ISPs had proper egress filtering DDoS should be possible to handle.
Any site should be able to send a message to the net saying block all traffic from that IP to me. The ISP closest to offending node then applies this filtering for an hour or so.
Even if attacked by a million nodes, sending out a million block messages should be doable.
Doing strict uRPF everywhere is simply not possible. Asymmetric routing is far too common.
And there are perfectly legitimate, if somewhat odd, scenarios where you need the ability to send packets from locations different from where incoming traffic would be routed. Certain load balancer / geographically distributed setups, for example.
Was this attack even spoofed, by the way? Doesn't really sound like it from the reasonably specific host counts. And a lot of these hosts are probably on various lower-end connections, which is where spoof protection actually is frequently implemented. I remember some DDoS tools actually used to test each host for spoofability on installation and automagically have them do the "right" thing spoof-wise (fully spoofed sources / addresses from the subnet only / only real address).
If you are approaching a terabit of traffic it likely won't help defense much that it's totally unspoofed. Atleast not if you can't get long source address ACLs inserted far upstream.
The only thing spoof protection truly breaks is amplifier attacks, but for that all you need is one or a few originating hosts and that you'll always be able to find.
One big problem is it's often extremely difficult to trace the originators with a distributed DOS attack through compromised devices controlled by heavily disguised control systems which, themselves, can go through compromised devices. Often this can be triggered by anybody, anywhere using any old public network. Even when the controlling source (or the source of the compromising agent) can be identified, these are often residents of countries where the rule of western law doesn't hold, or even regimes where this sort of activity serves a purpose of the state (or even agencies in that state not under full control).
It might be that some really draconian action will be required on ISPs and network operators to manage the security on their devices. A can conceive of ISPs and network operators being compelled to police their own user base for illicit traffic on pain of having some of their service access cut off which means, by implication, they have to police their users the same way.
Perhaps also some penalties for manufacturers and suppliers of devices that can be compromised which don't fix security holes. This is one huge issue for the "Internet of Everything".
Ultimately, the whole infrastructure needs to be hardened, and especially core services, such that they are far more difficult to attack in this way.
Yeah. Law enforcement cannot stop this. "Cyberwar" counterattacks won't work either.
"Draconian self-policing" (throttling/disconnecting infected downstream users) won't work against botnets whose DDoS traffic is effectively indistinguishable from legitimate traffic. End users won't disconnect infected devices that appear to be functioning normally. Government "cybersecurity" regulations will be misguided and ineffectual. Nothing will be done until the internet is unusable.
What can be done is, 1) Cutting back on unnecessary technology, integration, services, features, etc. 2) Keys instead of passwords. 3) Standard binary data formats that are less susceptible to serialization attacks than oddball/proprietary formats and the "web soup" of text formats embedded in one another. 4) Not just open source, but simple and understandable open systems all the way down to the transistor level.
"It might be that some really draconian action will be required on ISPs and network operators to manage the security on their devices. A can conceive of ISPs and network operators being compelled to police their own user base for illicit traffic on pain of having some of their service access cut off which means, by implication, they have to police their users the same way."
I agree. And there's already precedent with email blacklists occasionally blocking a whole ISP for allowing outgoing spam. I can easily see interconnect companies and back-haul providers being the "police" in something like this. What about attacks against the various internet exchange hubs? Easy. Shut down connects with the ISPs with the top 3 or top 5 number of attack sources and tell them to sort it or find another interconnect.
Yeah, I know it's not really that simple, but all this talk of "big data", security services "black boxes" on ISP networks, ISPs own monitoring and records keeping of users data held for later analysis, "cloud" processing etc, you'd think it should be a piece of piss to track and block all this shite. Isn't this why all the data is being collected?
"ISPs and network operators being compelled to police their own user base for illicit traffic on pain of having some of their service access cut off which means, by implication, they have to police their users the same way."
If a large enough number of devices are involved the illicit traffic from any one device might not be easily discoverable. A better variation would be policing their user base for vulnerable internet-exposed devices. Where the device is an ISP-supplied router this would have the immediate effect of requiring the ISPs to be more careful in deciding what kit they supply.
Great. We'll put Doctor Syntax in charge of vetting routers for the ISPs. If this happens again, we'll put him in jail. ;)
Seriously, there's no point in punishing people for systemic problems dating back 25-50 years. Nobody's blameless, everyone's in over their head. The system in question is literally the sum total of every living software project that grew from a working prototype to a big ball of mud. BBOM^N.
"Politically, we need international treaties which provide that anyone who screws up the Internet, regardless of where they are, will be arrested and tried for it. "
Does this include the countless people / businesses / etc who cut every possible corner to produce cheap IoT style gadgets because they dont really give a toss about how they could be misused?
Yes, the skids who launched this attack need to be identified and punished, but then so do the people who fundamentally fucked everything up so much that a bored kid can take down the internet.
> I'm not smart enough to figure out a solution (and there may not be one), but it seems to me that something should be possible.
What I'd _like_ to suggest but is actually a bad idea would be when one of these hijacked devices is identified, that the victim server could be allowed to route back to the offending device, and reset it, erasing the bogus code and setting a new random password. Then the device would still run, but the owner would be locked out of the admin interface until they reset to factory specs again (and hopefully set the user /pass to something different). Needless to say, this is a bad idea.
But either from class action litigation liability forcing a recall, and/or legislation, requiring every device to have a different factory reset password and defaulting to not allow admin access from the WAN side would solve most of these problems. And I suspect you will see ISP / cable providers taking an active role and blocking devices that they determine are susceptible. They could do thus with a quick login test when a device is first seen by their routers by detecting the device type and trying the default login. If it wirks, they block traffic from that device (or port, if on a local NAT setup.)
David Gibson, VP of strategy at Varonis,
That should be at something else. Sound similar, but more apt. The guy is clueless.
There are multiple issues here, but if you look at the outage maps you will see that the most affected part is USA and especially the Eastern Seaboard.
The reason for the size of the outage is a combination of Dyn not being a Carrier in its own right and the way USA Internet works. USA Internet is built around a small set of private peerings between large Carriers and there is no public peering as such. If you are providing a "service" like Dyn you cannot peer - you have to buy transit. This limits your upstream diversity to a couple of links. Whack any one of them with a DOS and you go off the radar for a significant portion of the USA Internet. The reason it gets really ugly is that USA providers often have some seriously "funky" routing policies where they override BGP to force traffic not to leave their network. In normal days - fine. If you have your upstream links going up/down because you are being DOS-ed - not so much.
Compared to that a Carrier run DNS can be geographically distributed and located next to peering points resulting in significant levels of resilience. Similarly, in Europe peering rules are often waived for companies specializing in DNS and some DNS services are run by peering points themselves resulting in much higher resilience. There are also a number of decades old methods of providing load distribution and georgaphic resilience which work very well if you are a Carrier. If you are just a service provider with limited upstream diversity - not so much.
As far as OpenDNS Openly violating DNS semantics and disregarding TTL reported by source servers if it cannot reach them - no thanks. Protocols specs are designed in a specific way for a reason and if you do stupid sh*** like that all kinds of retarded things can happen.
You are fundamentally wrong. Lots of service/content providers peer. A lot.
See CloudFlare for an example. Or Akamai. Or Google. Or even entities that basically do only one thing, like Facebook and Netflix.
Peering is based on mutual benefit, and obviously there often is a benefit for ISPs to have content their users access (or DNS queries for them) go through peering instead of transit.
You are fundamentally wrong.
Tell that to any one of the USA Carrier oligopoly. They will tell you to contact their sales department to mutually benefit from your money as a paying customer.
You can do that argument over, on this size of the pond. That is how the Internet works in Europe. In USA - not so much.
So unless you are a part of the merry oligopoly gang anycast will not help you much. Sure - you have HA and resilience, but not upstream link diversity. So once your links start to be hit by 600G+ you are out of the game there and then.
While you are correct about Tier1s peering policy, you aren't correct about the implications.
The solution is simply to peer with´a lot of networks from "Tier2" and down.
Not a lot of networks have a Tier1 as their only upstream anyways - they tend not to provide good connectivity for a specific region, you want a national network for that.
Basically, you only have Tier1s as your primary transit when you have a lot of peering on your own. This is what the whole "Tier" model implies - peering with networks on the same "Tier" and buying transit from those on a lower.
The role of Tier1s is basically to provide international connectivity.
When doing anycast you ideally don't want your traffic hitting the network of a Tier1 ever (or atleast not be transported far by it) - that means you have some geographic region not covered by the anycast.
By the way, OVH has successfully weathered twice that and they are just a hosting provider.
CloudFlare certainly has weathered comparable attacks as well.
A Tier1 run DNS service (i.e. using their AS with their peering policies) would actually be WORSE - precisely because Tier1s don't peer except with other Tier1's.
Tier1's don't have end users (well, not a lot of them at least). As a DNS service, you want the best paths possible to end users. Therefore, you don't want a Tier1 to be the DNS service.
Makes no sense to be down. Everyone knows you never put your DNS servers on the same darn subnet, same thing would be to never host your DNS servers at one company. Unless you're looking to cut corners.
Solution is pretty simple: Run your own DNS servers. Of course that does require people.......
Just in case anyone needs it, OpenDNS provides IPv6 lookups at 2620:0:ccc::2 and 2620:0:ccd::2 (you'll find the relevant data and details on their website).
A pox on unsecured IoT kit providers, and prolonged percussive re-education for those abusing those weaknesses.
Our IDS/IPS automatically inserts a new firewall rule for every incoming hack. It then reports the offending IP address to the ISP owning it. It never sleeps.
During a DDoS attack on our website in 2014, we were attacked by approximately 7000 servers from almost every subnet in Brazil and Argentina, during an attack that lasted a week. Each attacking server was blocked after the first hack query and, over the course of the week, the attack tailed off, as each ISP took the offending IP address offline.
This may not be a perfect solution, but it deactivates each mindless parasite as it removes each attack endpoint. If everyone did this, and ISP's responded fast enough (Best: Brazil, Germany, Russia, USA, Indonesia, Israel. Worst: China, Mexico, France) the hackers would spend their lives constantly looking for new servers running vulnerable WordPress/Joomla installations, as the existing ones were neutralised.
If the bad guys didn't have free access to millions of open IOT and home routers, this would just be a minor annoyance. Fix the problem at the root and all the complex BS solutions discussed would just be that:
Doesn't mean that better managed DNS services wouldn't help. Doesn't mean that robust defenses and even offenses isn't a good idea. But building the castle walls higher when the effing barbarians are at the gate seems like too little, and to effing late.
If the public DNS servers algorithm was changed to continue to use the entries whose TTL had expired if it was not possible to get a reply from a master DNS server would that have any severe effects ?
(I am thinking of providing responses to users with a 60 second TTL and requerying the master DNS servers at 60 second intervals until a response is received.)
Biting the hand that feeds IT © 1998–2020