Canadian ISP Rogers falls over for hours, takes out broadband, cable, cellphones

Canadian ISP Rogers today suffered a major outage that has left subscribers unable to use cellular, broadband internet, and cable services. ATMs and debit card payments were also affected, leaving folks unable to take out cash and buy stuff in stores, and some were unable to make 911 emergency calls. Rogers has more than 10 …

[Anonymous Coward]

The problem is Roger's started as a cable company where "sorry, we can't fix it till Tuesday" was acceptable. They never have learned how to deliver the quality of service and reliability that a backbone provider should. This is not the first time they've screwed up this bad, either.

Make no mistake, though, they'll blame someone else for their gross incompetence - last time they blamed unnamed "vendors" for the problem, despite the fact that they don't use any vendors that aren't also used by other telcos in the country, none of whom had an issue.

[Anonymous Coward]

If emergency calls are not working... they are required to report accurately the reason to TPTB, why the fix took ages and why they hadn't planned for this outage. They probably will be fined as well. I (and Rogers officers!) hope that the failed emergency calls won't result in deaths.

Bogus claims about vendors and hand waving are for us, the regular Joe Sixpacks.

[UCAP]

It was not just 911 calls - apparently some Canadian ambulance services used the Rogers network for making dispatching calls, so even if you could get through on 911 they could not respond.

[NoneSuch]

They can only fix it between 1pm and 3 pm, next Wednesday.

On another note, given the midnight timing of the start the outage, I'm betting celebrations in FSB HQ are underway.

[DanceMan]

Re: FSB

Today NHK (Japan) had a report on the cyber war launched against Ukraine and the continuing war with white hat hackers and friends of Russia contributing back and forth. One does wonder if Canada as a major friend of Ukraine was targeted.

It also shows the danger of the vast majority of Canada's IT connectivity in the hands of only three giants and the unresolved issue of the Rogers-Shaw buyout which would make it worse.

[Anonymous Coward]

Re: FSB

I have no doubt that is high on Roger's "who shall we blame?" list.

[nintendoeats]

Re: FSB

While I completely agree with the second paragraph, lets start by assuming incompetence rather than maliciousness. Rogers has plenty of that to go around, they don't need any help from the second world.

[UCAP]

Re: FSB

It was incompetence - a maintenance update went wrong. Have I heard this story before ....

[Yet Another Anonymous coward]

Re: FSB

But can we be sure that a comfy monopoly meaning you don't need to hire good staff or have good processes wasn't a cunning Russian plan all along

[Kane]

Re: FSB

"Have I heard this story before ...."

Welcome to "Who, Me?" your weekly dose of IT cockups where it's never DNS...

[Anonymous Coward]

1pm to 3pm next Wednesday is rather an exact timeframe. I think between 8am and 5pm next Wednesday is more realistic.

[chivo243]

please wait for our text message to confirm your appointment with us!

[jake]

Insert something pithy about ...

.... eggs and baskets right about here.

[The Oncoming Scorn]

Annoying As It Is

I wangled a corporate rate with them a few years back, the last time they screwed up my next bill was half the normal amount.

At least they are being upfront about it.

Freedom are worse, I jumped ship after spending a late morning & afternoon ringing the support lines as all incoming calls were going to VM & all their "solutions" didn't work & neither did their call backs (Go figure). Finally after about 5 hours one support rep admitted there was a national outage. Phone services were finally restored 23 hours according to my best guess of when the "services" went down.

[FuzzyTheBear]

Time of writing

It's still down.

must have been something big. will be interresting to follow.

im not with rogers thank goodness but i can say yes .. interac transactions are also affected.

will be quite a lot of people abandoning them.

the outage is WAY too long for WAY too many people.

[d2]

Re: Time of writing...Rogers luvs loonies not public ire

c'mon, el Reg gals n' guys, you don't have to be a brainiac to see the forest for the trees...good ol' greedy rogers was given the carrot stick nod to suspend services to the 'fringe minority'[viz cdn freedom convoy supporters] by canuckistan's fearless leader...just to remind them how helpless&pathetic all great white snowflakes really are...(just a ret.mechanic's musings)

[Anonymous Coward]

Re: Time of writing...Rogers luvs loonies not public ire

Mind boggling. The depravity of what some people call "thought" and have the gall to post is absolutely stunning. There is definitely a shortage of mental health care professionals and services in the world...

[Anonymous Coward]

Started affecting us at ~0940 UK time, that's when we lost all communication with the mothership in Canada so email / network drives etc were unavailable. Made for a quiet day truth told.

[Anonymous Coward]

Long history of snafus

I recall when they first entered the "business market" with email and stuff. Called it Rogers for Business. For a period of a few weeks, email was being dropped on the floor, never delivered, lost forever, and customers not told.

[Anonymous Coward]

Re: Long history of snafus

"Introducing Rogers for Business. We've got everything you need. Connectivity? Get Rogered! Email? Get Rogered! 24/7 support? Get Rogered! Rogers for Business ... Has your business been Rogered lately?"

[Jamesit]

Still down 1545PT I hope it's up soon. way too long they should have a backup network for situations like this.

[No 3]

Should mention that this exact same sort of massive outage happened just a little over a year ago, with the same provider.

Rogers is trash.

[The Oncoming Scorn]

Its Back

Came on around 4.55pm when I noticed.

I saw a guy pulled over by unmarked car, my guess he noticed his phone back online (90 seconds before I did), decided to gave a Yahoo text or phone call to test & got a ticket.

[Jamesit]

Re: Its Back

Still down for me. I hope it's back soon, I just rebooted my phone and still no connection.

[Mike VandeVelde]

Interac

Interac is banking in Canada. It was a global pioneer. It is a beacon unto the world. It is truly unique, more important than universal healthcare. Today it is an albatross piling onto a global pandemic recovery and supply chain issues and rising interest rates and rampant corporate profiteering in energy and pharmaceuticals and etc. How did Interac end up with all of it's eggs in the Rogers corporate basket? Corporate basket case:

https://www.zeifmans.ca/blog/what-the-rogers-family-drama-can-teach-us-about-succession-planning/

How in the fuck did Canada's national inter-banking system end up in the incompetent hands of these clowns??? Write us a story on that please. How does one single telecoms company having "issues" bring down payment infrastructure across an entire nation? For dozens of hours???? Nobody could use a bank machine today, nobody could buy a transit fare to get to work, nobody could pay their bills, nobody could buy groceries. WTF? Please dig in deep and fill us all in on all the gory details. Vultures - sic balls!!

[Anonymous Coward]

Re: Interac

Banking did not end up in their hands; banking has redundant backbones, much like telcos.

The problem is Interac is not a bank; it is a consortium providing services to the banks, and not subject to the same financial regulations as a result.

[C R Mudgeon]

Re: Interac

"redundant backbones, much like telcos"

Some telcos. Rogers, apparently, not so much.

[Anonymous Coward]

Don't you love their way of communicating?

I mean, tweeting - who was going to be able to read it?

Looks like Canada got properly, er, Rodger-ed.

Bad jokes aside, it does illustrate clearly that Internet connectivity has become mission critical for a substantial chunk of any society now so I'm hoping that they (and others) will be put under pressure to work on their resilience.

[John Brown (no body)]

Re: Don't you love their way of communicating?

Came here to say exactly that! A comms company goes down and they keep their customers up to date on the recover progress (or lack thereof) by using the very comms they can't supply!

[ThatOne]

Re: Don't you love their way of communicating?

The point is to be able to claim they did keep their customers up to date, they won't at any point claim their customers were able to read those updates. Apples and oranges.

Besides, according to marketing any self-respecting person under 80 is tethered to Twitter 24/7, so Twitter is the best (cheapest, fastest) means to keep customers users suckers posted. Added benefit, the length restriction avoids having to disclose too much information, it's just the right size for a canned "Your whatever is very important to us, now please get out of our hair" type statement.

_{(Icon: Jolly Rogers)}

[Yet Another Anonymous coward]

Re: Don't you love their way of communicating?

What was the alternative ?

A message on the Jumbo-Tron at Roger's arena ?

Writing to all their customers and having Canada post deliver it ?

A giant flock of carrier pigeons ?

A Roger's executive drinking a lot of beer and writing a message in the snow ?

[John Brown (no body)]

Re: Don't you love their way of communicating?

"A giant flock of carrier pigeons ?"

That;s a good idea. A possible improvement might be to use Canada Geese instead.

[ThatOne]

Re: Don't you love their way of communicating?

> What was the alternative ?

Not using Twitter for starters! From there on everything is acceptable...

^{Personally, if I had to plan ahead for such an eventuality, I would probably set up a telephone voice server, distributing a canned status message: It's easier to find/borrow a working phone than a working internet connection (Note I'm not living in Canada, I don't know the local specificities, so I might be talking utter nonsense). Also, it isn't worth the money if it's just to broadcast a vague "A few customers are experiencing limited problems right now" for several hours on a row. It all starts with actually knowing yourself what's happening.}

[Yet Another Anonymous coward]

Re: Don't you love their way of communicating?

But Twitter will get reported on the news. Canada has a well funded public broadcaster who are quite capable, once the intern has shown them how, of regurgitating a tweet

[ThatOne]

Re: Don't you love their way of communicating?

> But Twitter will get reported on the news.

So what? The impacted people already know there is something amiss, the one and only thing they want/need to know is how long it will take to be fixed, and given the length of the news information path any information will probably only appear on the news long after the fact.

To rephrase that, as a user you need fresh, up-to-date information, not distant echos of something said by some PR mouthpiece half a day ago. Only up-to-date and precise information allows you to take any meaningful decisions. (As I said previously, that is assuming the company does indeed know what's going on, and doesn't just serve canned excuses.)

[_gh_]

Re: Don't you love their way of communicating?

Not sure about the rest of Canada but the age of POTS is dead here in North Vancouver.

We were to be moved to fibre based phone service so my last reason to maintain the landline went about a year ago.

Now I have to ensure that my ISP & cell provider use different backbones so that at least something works when the next outage occurs. Given that Rogers is trying to buy my ISP something is going to change unless the govt blocks it.

Basically it means that we'll be down to 2 providers for consumer ISP/Cable/Mobile Telus (that share a load of infrastructure with Bell) a telco that became an ISP & IPTV provider and Rogers a cable co that became a telco / ISP ...

Dig under the skin of the other providers and they use one of these 2 for infrastructure.

[Sudosu]

Re: Don't you love their way of communicating?

Ah yes, good ole RFC1149

A Standard for the Transmission of IP Datagrams on Avian Carriers

https://datatracker.ietf.org/doc/html/rfc1149

With addendum RFC2549 for important workloads

IP over Avian Carriers with Quality of Service

https://datatracker.ietf.org/doc/html/rfc2549

[TimMaher]

BGP

It’s always that. Or DNS... Or JavaScript... Or...

[RM Myers]

Re: BGP

Eh.

[Anonymous Coward]

Re: BGP

> It’s always that. Or DNS... Or JavaScript... Or...

Except when it's a beaver.

https://bc.ctvnews.ca/single-beaver-caused-mass-internet-cell-service-outages-in-northern-b-c-1.5944697

[Yet Another Anonymous coward]

Re: BGP

Even in Canada it's not normal to have Beavers update your BGP routes.

[Anonymous Coward]

Service as per normal ..... :)

I read the Headlines and thought ..... Service as per normal then, why the story ???!!!

(Experienced 'Rogers' when in Canada and thought that it must be an international 'BT Subsidiary' judging by the Service quality !!!)

:)

[Shuki26]

I'm speculating on some software third-party license expiry

Expired Jul 1 but had seven day extra warning. Something overlooked in some automated license check. Maybe some expired ssl certificates? Reg, please follow up on the story.

[Barry Rueger]

Also, greedy

Not noted above: mobile and cable contracts in Canada usually run $75-80 each per month. For a meagre amount if data.

[Anonymous Coward]

Re: Also, greedy

That's unfortunately the oligarchic system of N. America.

In Europe they have a true "free market", with real competition, and as a result contracts are around $30-40/month with all kind of freebies and bonuses. AFAIK data allowances for mobile phones are in the multi-GB range, and home connections are unlimited of course. And strangely enough, their companies make enough money to keep investing heavily (fiber all around, and so on).

Americans would do better to stop blathering about free market and start thinking about implementing it!

[Barry Rueger]

Re: Also, greedy

contracts are around $30-40/month with all kind of freebies and bonuses

France: 19 € a month. 210 gigs of data. Unlimited calling INCLUDING to North America.

France, like Canada, has 3 or 4 major players, so the old "free market" arguments are just nonsense.

I'm willing to guess that there's some level of government regulation involved.

Or, possibly, French cel providers are just not as insanely fucking greedy as Bell, Rogers, and Telus.

[RobThBay]

From an "expert"

One of the "expert" talking head guests on the news last night said..... they should have backup systems....like using paper....

She went on and on and actually said nothing.

[jake]

Re: From an "expert"

The "news" is just entertainment used to sell adverts for beer, cheese and tampons. If you want education, get thee to a library and learn how to properly use it.

[Jamesit]

"Rogers CEO apologizes, says 'maintenance upgrade' behind major outage"

"he president and CEO of Rogers Communications has apologized for a serious network outage that disrupted service for more than 10 million Canadians. The “network system failure” was caused by a maintenance upgrade, he says."

https://rock101.com/news/8978590/rogers-communications-network-outage-explanation-ceo/

[jake]

So ...

... they have upgraded their maintenance.

What does that even mean? Have they put a new schedule into place? Fired all personnel and hired a new lot who know what they are doing? Brought in consultants? Off-shored it to save money? Some combination of the above?

Does anyone, anywhere, think it'll actually help, whatever it is?

[vtcodger]

... And so it goes.

per the BBC https://www.bbc.com/news/world-us-canada-62110358

The company's CEO Tony Staffieri said the failure followed "a maintenance update in our core network".

...

Mr Staffieri said the maintenance work "caused some of our routers to malfunction early Friday morning".

Please explain to me again why this cloud thing is such a terrific idea.

[ThatOne]

Re: ... And so it goes.

Because it's hip. Clouds are known for their silver linings and all that.

_{You know the expression "living in cloud cuckoo land", don't you.}

[Anonymous Coward]

Roger the cabin boy

Captain, I think she’s #$&@*(

[Alistair]

interesting perspectives

1) Thank (deity/curseword/appropriate idom) I'm no longer there.

2) Interac was working in specific cases, although the bulk of the backbone does go through Rogers it was not utterly wiped out

3) Bank machines were fine, except for (PC/SuperStore/Shoppers/Rogers) I did hear from one person that their banks machines were not working but I believe they're a (virtual bank) type critter. (Yes, Loblaws has a bank, and Rogers has a bank, and Shoppers Drug mart too, and they are all the same thing)

Certainly at least TD and CIBC were just fine.

At this point, my neighbour has never lost the Rogers cable service, but their interwebs is down, as is the "home phone" service. Cell service for Rogers is off and on in this neck of the woods, and from checking a phone that was flipping on/off signal it seems to be an IP addressing issue as it went through about 5 different segments and subnets in about 15 minutes, some working some not.

For what its worth, I'm not on Rogers, and my interwebs has been just fine all day.

I love how some folks are talking Armageddon, but in reality 90% of things were happening just fine. Including the police raid around the corner. All 8 cruisers were just fine. Not sure about the crew they busted, but the cruisers were fine.

[T F]

What I don't understand is: Why did Rogers not create a contingency plan after last time? By them being that negligent says a lot about the company. I will not be using their service in the future.

[d2]

merely Coincidences?

strangesounds.org /2022/07/coincicence-wefs-cyber-polygon-2022-event-took-place-exactly-when-simultaneously-canada-experienced-extreme-internet-outages-on-july-8-2022.html

Coincidence? WEF's Cyber Polygon 2022 event took place exactly when Canada experienced extreme Internet outages on July 8, 2022 -

7/10/2022

So are we supposed to just think it’s just another coincidence? The WEF’s Cyber Polygon 2022 meeting took place on July 8, 2022… and simultaneously Canada was experiencing its worst Internet outage in decades…

On July 8, 2022, Canada’s phone services, 911, credit card/debit machines were down for several hours…

Coincidentally, the crash occurred on the same day than the 4th edition of Klaus Schwab’s Cyber Polygon…

Coincidentally, the participating companies had to undergo a targeted attack on their hybrid cloud infrastructure.

Coincidentally, this year’s primary theme of the conference was digital resilience in the cloud age.

And coincidentally, this Russian-based, WEF-backed annual cybersecurity exercise Cyber Polygon was first postponed to a later date…

Don’t forget that back is March 2022, the World Economic Forum removed Cyber Polygon from its webpage, preparing us for their CYBER WAR.

That’s just another coincidence?

As reported by the main partner bi.zone, Cyber Polygon 2022 took place on July 8. The fourth international cybersecurity training focussed on digital resilience in the cloud age.

The event consisted of three parallel tracks:...

[Flywheel]

Time for a name change

A rebrand always improves performance! How about Rogered ?

[Mr. V. Meldrew]

Should I worry?

Now not being familiar with Canadian internet structure of provision, resilience, support, infrastructure ... blah blah I ask this naïve question with the hope you will spare a poor disabled pensioner (EX BT for sins committed in previous live)

Would Rogers "outage" happen in the UK? Particularly with the current rollout of BT's FTTP?

This worries me. I live just outside Manchester and pay £15 a month to my ex employer for a poxy 7.6 D/L speed (U/L 810Kb). What hope do we have in a copperless society? (aluminium inserted in small lengths just to bugger it all up)

Should I be worried that the connected world may un ravel?