back to article ISP Level 3 goes TITSUP after giganto traffic routing blunder

ISP Level 3's customers have been left without internet access since this morning, after the provider seems to have leaked routes to a Tier 1 transit provider in Malaysia. An incident report from CloudFlare said that while "the Tier 1 transit provider of the ISP leaking routes appears to have stopped accepting these …

  1. This post has been deleted by its author

    1. Anonymous Coward
      Anonymous Coward

      Obligatory post-snowden comment.

      NSA accidentally screwed up their tap in Malaysia Telecom and called out the whole world rather than just SEA into their tap.

      1. Martin Summers Silver badge

        "NSA accidentally screwed up their tap in Malaysia Telecom and called out the whole world rather than just SEA into their tap."

        Yawn

    2. theloon

      If this was the issue, then Level 3 is totally at fault, since one of the core principals for BGP policy is only to accept what you are expecting, since this is the obvious outcome of not having that route policy correctly defined.

      This is such a basic error that the 'fat finger' has to have been involved.

      However it's another example of how policy needs to be defined elsewhere, and the limits of BGP policy configuration done via the current OSS model....

      Please god don't anyone say "hey we need another extended attribute for BGP that will solve this"..

      1. Anonymous Coward
        Anonymous Coward

        Thankfully there is the RPKI system already existing to verify exactly who is allowed to advertise what. Just needs network operators to actually use it..... hmmm.

        1. Sir Runcible Spoon

          FFS Tier-1 ISP's have been using AS filter paths for 20 years - how the hell is this still happening in 2015?

          1. Tom Samplonius

            "FFS Tier-1 ISP's have been using AS filter paths for 20 years - how the hell is this still happening in 2015?"

            Given that you must mean prefix lists, not a path fitler, as a path filter would have allowed this through. As far as why, your comment is a perfect example: people don't know the difference between prefix filters and AS path filters, among many other things.

        2. Yes Me Silver badge

          Hmm...

          Hmm... I wonder who on earth (literally) could ever be authorised to announce the default route in BGP4.

      2. Tom Samplonius

        "However it's another example of how policy needs to be defined elsewhere, and the limits of BGP policy configuration done via the current OSS model....

        Please god don't anyone say "hey we need another extended attribute for BGP that will solve this".."

        Umm ok, so you know about BGP attributes, but you don't know about routing registration databases? You are supposed to build routing policies based on routes registered in a registration database. Interestingly, Level 3 runs their own routing registration database (http://www.irr.net/docs/list.html). What Level 3 is supposed to be doing, is automatically building new route prefix lists and pushing them to their edge routers every day. And those prefix lists would contain all acceptable routes.

        Since a large ISP may have several thousand prefixes, automatically generating them from a database is the only way to go.

    3. Anonymous Coward
      Anonymous Coward

      Someone stipped on top of a sacred router

      Most likely explanation - it is the wrath of BGP after someone stripped on top of a sacred router.

      Me coat. The one next to the backpack :)

  2. Mike Pellatt

    It certainly killed telappliant for 30 mins+

  3. Dabooka Silver badge
    WTF?

    I know who I blame.....

    and it's nothing to do with levels, tiers and ISPs.

    Wrath of the Gods

  4. Alan J. Wylie

    #WhoBrokeTheInternet

    Hashtag #WhoBrokeTheInternet

    1. Destroy All Monsters Silver badge

      Re: #WhoBrokeTheInternet

      MALAYSIA STRONG!

      1. Anonymous Coward
        Anonymous Coward

        Re: #WhoBrokeTheInternet

        Kim Kardashian nudie pics?

  5. NogginTheNog

    Aha

    Is that why my usual public DNS resolvers (4.2.2.1 - 6) went all pants this morning?? I've been (reluctantly) using 8.8.8.8 since...

  6. Doctor Syntax Silver badge

    In other news...

    ...Freeparking have recovered from their TITSUP. They've resumed spamming me about renewing a domain which was transferred away from them.

  7. Smelly Socks

    Goooood morning Malaysia!

    https://twitter.com/TMCorp/status/609167065300271104

    catch it before it's removed

    -ss

  8. John Sanders
    Holmes

    Level 3

    Seems to suspiciously have lots of BGP problems lately...

    Just an observation.

    1. This post has been deleted by its author

  9. Antonymous Coward
    Mushroom

    Tolerate a nuclear attack my arse!

    Been wondering "who's gone and broken the internet now" for a couple of hours. Wasn't expecting it to be a single misconfigured server in Malaysia.

    1. Destroy All Monsters Silver badge

      Re: Tolerate a nuclear attack my arse!

      ProTip: The Internet was NOT made to tolerate a nuclear attack.

      See also: Paul Baran

      1. Lee D

        Re: Tolerate a nuclear attack my arse!

        And, even if it were, it certainly wasn't to do so immediately, with zero downtime or with zero human intervention worldwide.

        Else things like BGP would have been in the bin decades ago. I mean, seriously, just having routing tables hit certain sizes is enough to make many brands of high-end networking gear just fall over. BGP routing tables grow into the same kinds of fixed spaces. And, hell, BGP announcements do nothing to take account of CAPACITY of the system on either end (i.e. the preference of a particular route based on its response time etc.).

        The Internet won't invisibly and automatically survive any kind of attack. However, it will be not-so-difficult for even a small bunch of humans to cobble it back together even if that means throwing out DNS, BGP or similar in some fashion to allow it to do so.

      2. Anonymous Coward
        Anonymous Coward

        Re: Tolerate a nuclear attack my arse!

        I beg to differ

        If instead of a stupid user error the site had been hit with a nuclear device the rest of the internet would have been fine - in fact it may have been better off

        No one said the Internet could survive stupid users - stupid users has been a problem since prehistoric man dropped a really big rock on their foot and it has only gotten worse.

  10. Spiny_Norman
    Joke

    Wrath of the Gods #2

    Wow did someone get their tits out in an exchange?

  11. Hugh McIntyre

    Topical Washington Post story on total insecurity of BGP ...

    Specifically (and topically) on BGP issues like this one at Level-3 (but also the difficulty of moving people to BGPSEC):

    http://www.washingtonpost.com/sf/business/2015/05/31/net-of-insecurity-part-2/

    And more generally, earlier history on why the original designers did not expect so many attacks:

    http://www.washingtonpost.com/sf/business/2015/05/30/net-of-insecurity-part-1/

    ... but as the articles say, early hardware could not do encryption easily, NSA probably objected, and people at the time never imagined we would be doing on-line banking or there would even be a YouTube to censor :(

  12. Anonymous Coward
    Anonymous Coward

    AS3549 -> 3356 migration

    Level3 is in the process of migrating the old Global Crossing (AS3549) networks into the Level3 AS3356. I suspect some config generation handled within 3549 was migrated to the 3356 script, which put in a permit all on the Telecom Malaysia transit port, which had always been "broken", but it didn't matter before.

    Speaking as working for a network that peers with both 3356 and 4788, we solved the issue once we depeered 4788

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021