back to article Fat-fingered Level 3 techie reduces internet to level zero: Glitch knocks out connections

A fat-fingered configuration blunder at Level 3 temporarily sent a chunk of internet connectivity in America and the wider world to level zero on Monday. The global backbone provider admitted it was at fault for rolling outages that have plagued big patches of US this morning, West Coast time, particularly for Comcast …

  1. Yet Another Anonymous coward Silver badge

    The cloud

    Where the failure of a computer you didn't know existed ruins your day

    Where the failure of a supplier to a company that owns a computer you didn't know existed ruins your day

    1. Anonymous Coward
      Anonymous Coward

      'failure of a supplier.... that owns a computer'

      Bull! I've got my fat bonus from outsourcing and migrating our ops to the Cloud, and there's no bonus clawback... So long suckers!

    2. Mpeler
      Paris Hilton

      Re: The cloud

      They've looked at clouds from both sides now

      From up and down, and still somehow

      It's cloud illusions they recall

      They really don't know clouds at all...

      (apologies to Joni Mitchell)

      (Paris is still looking, too. CGNAT can be such a pain...).

      1. bobajob12

        Re: The cloud

        They paved paradise, and put up a data center.

      2. Jeffrey Nonken
        Thumb Up

        Re: The cloud

        I prefer the Judy Collins version, but I still voted you up. :)

    3. Fatman
      Joke

      Re: The cloud

      <quote>Where the failure of a supplier to a company that owns a computer you didn't know existed ruins your day puts you out of business.</quote>

      There!!!

      FTFY!

  2. Anonymous Coward
    Anonymous Coward

    Put it all in the cloud

    Yeah. Bollocks

  3. Mark 110

    WTF Cloud got to do with it?

    I can see that putting it in the cloud loses you access if te Internet falls over. But the internet doesn't have to fall over.

    Back in my Telewest days we, in response to failures by people ike Level 3, put in resilient links with different providers, through different geographic touchpoints. It wasn't perfect but we could keep soomething going if Level 3 or Telecity went down when previously we just had to look at the queue on the phones and shrug.

    Hopefully thhings are less reliant on the two of tem since the old days.

    1. Yet Another Anonymous coward Silver badge

      Re: WTF Cloud got to do with it?

      I use Microsoft's cloud service - I don't have any control over who their backbone is and what the SLAs are. If pre-cloud I had a contract with a specific online service I would likely know where my data was, and who to call when it went wrong.

      Instead I find out from el'reg how and why MSFT is fscked - while MSFT's own service page is saying "we are investigating reports of outages"

    2. K

      Re: WTF Cloud got to do with it?

      My old ISP / Datacenter on the South Coast did that.. IIRC correctly they once had a failure on the primary backbone provider, failed-over the secondary, a few minutes later this went TITSUPs.

      Turned out, the backup link got saturated, as they'd being progressively increasing their customer base and increasing the primary, but "forgot" to upgrade the secondary.

  4. Doctor Syntax Silver badge

    "Level 3 is one of the trunk roads of the internet"

    A sort of Information Superhighway?

    1. J. Cook Silver badge

      RE: Level 3 is one of the trunk roads...

      More like a series of tubes.

      I know that one of the companies that Level 3 borged some years ago boasted having it's own transit traffic infrastructure from Tokyo to London. I used to work for said company nearly twenty years ago, and we had a few 'whoopsie' moments like that. as an example:

      On of the router jockeys goofed one fine day and reconfigure an access router's default route to one of the customer's connection. Fortunately, the router crashed and rebooted before he could commit the configuration, but it was still a goof. A sign was put over his desk that said "We bring the Internet TO YOU!" as punishment. :D

      1. Blotto Silver badge

        Re: RE: Level 3 is one of the trunk roads...

        @j Cook,

        "I know that one of the companies that Level 3 borged some years ago boasted having it's own transit traffic infrastructure from Tokyo to London. I used to work for said company nearly twenty years ago, and we had a few 'whoopsie' moments like that. as an example:"

        would that company be Global Crossing?

      2. Anonymous Coward
        Anonymous Coward

        Re: RE: Level 3 is one of the trunk roads...

        leads to my favorite sequence of commands:

        # rel in 5

        (do $network_reconfiguration_thingy)

        (wait to see if I broke things)

        Especially useful if you're punchy from a late-night reconfiguration and are working on a remote router.

  5. Anonymous Coward
    Anonymous Coward

    Non-Redundant Redundancy

    Not backbone, but worked for a small business that paid for two circuits for redundancy. Found out both were on the same phone poles when a car took down a pole. Also found out they were going through the same hardware.

    I guess you pay for the peace of mind - and you have it, until your connection goes down.

    1. Blotto Silver badge
      Alert

      Re: Non-Redundant Redundancy

      @Etatdame

      you need to pay for diversity and then have your ISP prove the link is actually diverse, separate PoP's / Exchanges, separate non over lapping routes to your building, separate routes into your building preferably opposite ends etc.

      it gets interesting when you want 2 different ISP's to supply diverse routes to each other, as their planners both want to go the easy route.

      1. swschrad

        major pain in ass, and you have to stay on top of it

        it's not diversity if you are not reviewing the paths, offices, and facilities yourself.

        and every time a maintenance announcement hits the mailbox.

        costs a ton more.

        but if you must have uptime, it's worth it.

  6. a_yank_lurker Silver badge

    Wrong Villian

    Got home today and SWAMBO said Comcrap internet was having problems. She did not know it was someone else trying to go TITSUP.

  7. chivo243 Silver badge

    Minimum required

    I am glad we have only put the bare minimum of services in the cloud. My boss has just attended a meet and greet about cloud infrastructures, security and the ilk, and surprise, surprise, the prevailing mindset is a hybrid! Who'da thunk?

  8. DougW
    FAIL

    Scotty, beam me up

    The more they overthink the plumbing, the easier it is to stop up the drain.

  9. Version 1.0 Silver badge

    So this wasn't Kim?

    Think how much fun Kim Jong-un could have (or any other malactor) if they did this a little harder. We are a only an Level 3 outage away from empty shelves in Tesco, and hand-to-hand fighting over the last potato... (Thanks Della).

    1. Anonymous Coward
      Mushroom

      Re: So this wasn't Kim?

      Can't help feeling that should be a Meme, "John made a configuration error and rapidly things went last potato"

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021