back to article After config error takes down Rogers, it promises to spend billions on reliability

Canadian telecom giant Rogers will spend C$10 billion ($7.7 billion) to ensure that day-long outage earlier this month doesn't happen again, its CEO has said. We have also learned that, according to Rogers, the IT breakdown was caused by an early morning configuration update that jammed too much traffic through the ISP's …

  1. fidodogbreath


    Rogers will make its offerings more robust by "physically separating our wireless and internet services to create an 'always on' network." Staffieri said the move would prevent broadband internet customers from losing service in the event of a wireless outage. In other words, if the cellular network core goes down, it won't take out wired internet connectivity, and presumably vice-versa.

    So their C$10B solution will limit the blast radius of future config errors to only "all wired broadband customers" or "all cellular customers." I guess that a mitigated disaster is an incremental improvement over an unmitigated one.

    Seems like meaningful code review for router config changes would be cheaper and more effective.

    1. Yet Another Anonymous coward Silver badge

      Re: Oops

      Their internet failure, oops oh well no Facebook, also took down telephone for a lot of emergency services - less good

    2. gandalfcn Silver badge

      Re: Oops

      Rogers say something similar every time they go tsup so it seems the problem is with knowing what they're doing not how much they spend.

      1. Sixtiesplastictrektableware

        Re: Oops

        Pretty sure that with enough money, it doesn't matter what you know nor whom is put out.

  2. SilverCanary

    Hey hey hey, guess where the $10b will come from. Surely not by increasing prices in this duopoly* market for sure!

    Canada's internet prices are already among the highest in the world so what's an extra 10%.

    *depending on province

  3. Hawkeye Pierce

    Now going to do what they should have done all along?

    Is it just me or are they basically saying they are going to spend C$10B on what one might reasonably have expected them to have in place all along?

    1. gandalfcn Silver badge

      Re: Now going to do what they should have done all along?

      Given their track record they don't seem to know what they should have in place or why.

  4. Sparkus

    Rookie mistakes?

    A quick reading of the available apology letters seems to indicate that Rogers own-goals are universally rookie mistakes.

    From the setup of a 'unified' single network core to maintenance windows and equipment configurations that leave no room for error (or, gasp, business continuity or disaster recovery capabilities) there is every indication that the adults have long been gone from their engineering groups. Replaced with what/who, we can only guess.

    1. Yet Another Anonymous coward Silver badge

      Re: Rookie mistakes?

      Not clear they ever had any adults.

      They were a cheap cable TV company that added Internet, telephone and mobile as the rules were relaxed.

      They never had the telco "a downtime of seconds/year is simply not acceptable" mindset

  5. Johnny Canuck

    CTO also gone

    Jorge Fernandes will be succeeded by Ron McKenzie as the new CTO immediately.

  6. Anonymous Coward
    Anonymous Coward

    Standard procedure

    ""During that time, evergreen programming was aired from an MP3 player at the base of the transmitter until our engineering team was able to establish a connection"

    Standard arrangements for just about every radio broadcaster. When all else fails a silence detector will kick in a local source at the transmitter site. Nothing unusual there. Few listeners probably noticed the programme content was repeating every 25 minutes.......

    1. PRR Silver badge

      Re: Standard procedure

      > repeating every 25 minutes.......

      Even that is a step backward. In my day we jammed a CD player to auto-repeat, got 55 minutes of Neil Young over and over all night.....

      (Actually this was very early "internet radio". I set the CD in my office next to an early RealAudio server, drove home to listen.)

    2. John Brown (no body) Silver badge

      Re: Standard procedure

      ISTR an episode of WKRP in Cincinnati when they sent Johnny Fever (I think) out to the transmitter shack to keep them on the air. That show was from the 1970's :-)

  7. sarusa Silver badge


    Spoiler... it's going to happen again.

    1. gandalfcn Silver badge

      Re: Spoiler...

      Well they do have a history of fails, so a big yes.

    2. Richard 12 Silver badge

      Re: Spoiler...

      It'll repeat roughly once a year, unless the regulator turns it into an existential threat to the CxO level.

    3. Sixtiesplastictrektableware

      Re: Spoiler...

      The outages will continue until customer confidence improves!

  8. Anonymous Coward
    Anonymous Coward

    Yes, yes...

    I see it here in my ITIL book...

    The first step of proper change management...

    0. Don't implement proper change management, wait until something goes wrong first then spend loads (let's call it C$10bn, why not?) on implementing proper change management, but not properly...go to 0 until enough money has been pocketed / you can't take the piss anymore.

    Good on them.

    1. John Brown (no body) Silver badge

      Re: Yes, yes...

      C$10B sounds like an enormous amount of money to implement a change management system and duplicate some core routers. As the C$ devalued significantly recently and not as much as it seems, or does this sound like maybe this is the next five years planned investment with a bit added on to make out they are doing something big?

  9. Mr. V. Meldrew

    Would never happen to BT Openreach....

    Or could it?

    Answers on a postcard to the usual address.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like