BGP super-blunder: How Verizon today sparked a 'cascading catastrophic failure' that knackered Cloudflare, Amazon, etc Verizon sent a big chunk of the internet down a black hole this morning – and caused outages at Cloudflare, Facebook, Amazon, and others – after it wrongly accepted a network misconfiguration from a small ISP in Pennsylvania, USA. For nearly three hours, web traffic that was supposed to go to some of the biggest names online …
This post has been deleted by its author
"tag all free peer and customer transit routes with "no-export". That would reduce the damage small peers could do."
That is not necessarily effective. All of networks strip all communities, including NO_EXPORT. It is pretty easy to configure a router to do that, which makes it pretty easy to do it accidentally.
It also wouldn't necessarily work. So no-export means the route wouldn't be exported to any of the peer AS's (e)BGP neighbours, so networks beyond Verizon's AS would be unreachable. Or large parts of Verizon given regional ASs. And like Tom says, communities rely on the upstream to act on community tags. They're generally used downstream to give customers some choices wrt traffic engineering.
The best approach is to enforce the use of route registries, and implement the correct filters to limit the damage.
Disclosure: Ex-Verizon
So I think best practice for signing up a BGP customer starts with the routes they want to advertise. Then making sure there's a route object in the relevant registry showing the routes that will be advertised by the customer AS and an origin showing the transit provider(s). In AS702-land, that used to be enforced, but sometimes required a bit of handholding and help to get the correct routing registry data filled out. Then when the session's configured at the provider end, there should be a max prefix set on the BGP peering config, and a filter to only accept the routes assigned to the customer.
In theory, that should stop these scenarios with either the prefix limit kicking in, or the route filters rejecting non-customer routes. In practice, it can be a bit of a ballache, especially in ARIN land where swamp routes often don't have route objects defined, or maintainers have long gone AWOL. RIPE users tend to be better behaved, and there's a bunch of software that will build filters based on the RIR data... But I sure as hell wouldn't trust a 'BGP optimiser' to do it for me, especially in a network as complex as Verizons.
"I remember, years ago, the internet was designed to stop people taking a big chunk out and stuffing it up for everyone else."
Years ago, the first octet of an IPv4 was supposed to represent the destination network and the second, the location inside that network.
Things grew, numberspace got crowded and that tidy setup got obliterated. IPv6 is _big_ and _sparse_ precisely to allow the tidy setup to be maintained without panicking and stuffing numbers into every available gap.
.......w*kers, stop pratting about and implement RPKI ROA validation. Follow the lead of the likes of AT&T and numerous other networks who understand how important the DFZ is as a shared worldwide resource and stop taking it for granted. There is no excuse for inaction on this, pull your fingers out your arse Verizon (and many others!)
The phrase you're looking for is: Iran is America's 'Great Satan'.
Pretty sure Trump is US's 'great satan'. Especially if you consider 'satan' means 'accuser'.
Perhaps in the run up to the next election he can can change his catch-phrase a little and make it more accurate...
"Make America 'Great Satan' again!"
This post has been deleted by its author
"Unless someone creates a 'project' and associated billing codes/budget to go with it, nothing gets done to fix BAU issues in large companies like this."
Or in other words, people should start billing Verizon for hijacking their routes - and then they _might_ start paying attention.
> To be fair, the magic beans worked just fine for young Jack
Jack is a moronically stupid (beans for a cow?), home-invading thief who does not think twice about stealing from a gentle giant; a persecuted minority, carefully minding his own business far away in the clouds, not harming anyone. And then, when the giant has the timidity to try and recover his own property, Jack saves his own skin by killing the giant and then engages in victim blaming by concocting a fairy story in which he is somehow the hero.
This week on the Jeremy Kyle show, we put Jack face-to-face with the Giant's wife to get her point of view.
;-)
Verizon "take our customers' safety very seriously"
They say that about customer privacy, too, with similar results.
With auto updates and potato like mono-cultures of OSes, how long before a couple of Friday evening patch releases take the entire Internet down for a week?
The problem is outsourcing and OVER RELIANCE on the Cloud.
Accidents will happen, more likely than a Cyber Pearl Harbour!
A BGP patch bug is suggested for one of the two reasons for failure of Internet in 'No Silver Lining' by Ray McCarthy
“If an in-house system fails, only one bank, or one retailer or one supplier is affected,” insisted Louise. “If everything is outsourced to the Cloud, even if it’s a hundred times more reliable it’s an apocalyptically bad event because you lose everything at once. There are too few cloud providers, who are too similar and too big.”
Once all retail POS, Wholesale re-ordering, services/Mobile billing etc is outsourced to the "Cloud" events like this will be more severe.
"...caused outages at Cloudflare, Facebook, Amazon, and others..."
Someone is owed one hell of a lot of beers! Now if you could've added Google and Bing, maybe that other bunch of Yahoos...
Very well done.
Oh - and those who clean up the messes as quickly and quietly as possible? Yeah we owe you big time as well (though next time this lot goes down maybe you can put the boot in a few times before you help them up? Perhaps 'accidentally' trip while trying to get them on their feet and drop a knee to the nutsack or something?)
That the set of people who know exactly how things are supposed to work, and can keep them working is:
1. Entirely unrelated to the size of the corporation or the amount of money that corporation has invested in the latest technology and/or processes.
2. Small.
3. Undervalued.
You will notice that there is never any key-man risk in HR or marketing.
This post has been deleted by its author
...would it be possible (as my network is mega basic) for a 'bad actor' to break into a local, small ISP to cause this. Then take the data dump from the small ISP? To make it all look like an internal fuck up instead of a state sponsored attack?
Just curious. I have no doubt this is an internal, American cockup, but would it also be possible to do the above? Be a sort of, slight of hand/misdirection type of hack?
Surely the ISP's employees would be suspicious if someone mangled the routing and nobody would be responsible. Wasn't even in that day. Haven't seen the router in years, guv, honest!
It's not as obviously malicious as redirecting Amazon's Cloud DNS and then grabbing the credentials from anyone trying to open their Cloud Crypto Wallet (TheRegister reported).
The global phone system is not in a better shape, either. Anyone with access to the network can redirect calls to mostly any number worldwide through their own equipment.
At least, BGP changes are monitored and logged by several institutions and traffic redirections can be investigated after the fact.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
