After config error takes down Rogers, it promises to spend billions on reliability Canadian telecom giant Rogers will spend C$10 billion ($7.7 billion) to ensure that day-long outage earlier this month doesn't happen again, its CEO has said. We have also learned that, according to Rogers, the IT breakdown was caused by an early morning configuration update that jammed too much traffic through the ISP's …
Rogers will make its offerings more robust by "physically separating our wireless and internet services to create an 'always on' network." Staffieri said the move would prevent broadband internet customers from losing service in the event of a wireless outage. In other words, if the cellular network core goes down, it won't take out wired internet connectivity, and presumably vice-versa.
So their C$10B solution will limit the blast radius of future config errors to only "all wired broadband customers" or "all cellular customers." I guess that a mitigated disaster is an incremental improvement over an unmitigated one.
Seems like meaningful code review for router config changes would be cheaper and more effective.
A quick reading of the available apology letters seems to indicate that Rogers own-goals are universally rookie mistakes.
From the setup of a 'unified' single network core to maintenance windows and equipment configurations that leave no room for error (or, gasp, business continuity or disaster recovery capabilities) there is every indication that the adults have long been gone from their engineering groups. Replaced with what/who, we can only guess.
""During that time, evergreen programming was aired from an MP3 player at the base of the transmitter until our engineering team was able to establish a connection"
Standard arrangements for just about every radio broadcaster. When all else fails a silence detector will kick in a local source at the transmitter site. Nothing unusual there. Few listeners probably noticed the programme content was repeating every 25 minutes.......
> repeating every 25 minutes.......
Even that is a step backward. In my day we jammed a CD player to auto-repeat, got 55 minutes of Neil Young over and over all night.....
(Actually this was very early "internet radio". I set the CD in my office next to an early RealAudio server, drove home to listen.)
I see it here in my ITIL book...
The first step of proper change management...
0. Don't implement proper change management, wait until something goes wrong first then spend loads (let's call it C$10bn, why not?) on implementing proper change management, but not properly...go to 0 until enough money has been pocketed / you can't take the piss anymore.
Good on them.
C$10B sounds like an enormous amount of money to implement a change management system and duplicate some core routers. As the C$ devalued significantly recently and not as much as it seems, or does this sound like maybe this is the next five years planned investment with a bit added on to make out they are doing something big?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
