FCC boss orders probe into 'unacceptable' T-Mobile US outage after carrier plays dog-ate-my-homework card

T-Mobile US is attempting to pin the blame for a massive network outage on Monday on a third-party leased fiber network, though the head of America's communications watchdog has demanded a full investigation into the "unacceptable" blunder. The mobile telco, now one of just three giants in the US mobile market after its merger …

[Anonymous Coward]

Note to T-Mobile:

Your brown envelopes need to be larger.

Hugs and kisses

Ajit

[Anonymous Coward]

It was just a promotional stunt

They are, after all, the Un-Carrier.

[Kevin McMurtrie]

This redundancy failed us

Is that referring to Sprint's network?

[mutt13y]

Coincidence?

Backup failed at the same time indeed!

IME

Either the backup was active/standby and not tested at least not in the last 12 months.

or it was active/active and crept over 50% utilisation and no one noticed or cared.

[Xalran]

sh*t happens

I've seen a $TELCO network brought down ( for 4 hours ) by two unconfigured switches connected to the network backbone. ( yes yes The Dreaded Spanning Tree Loop )

I've heard of a virtual HLR that went AWOL with it's redundant peer going AWOL a few minutes later too and a $TELCO unable make phone calls happens for more than half a day ( except for the historical users that were still on old monolithic HLRs )...

I don't have insider information as T-Mobile is on another continent... But things can happen that brings down a modern telecom network for hours.

note :

- seen means that I know the culprits (or the unwilling/unknowing perpertrators), and know how it happened and why in detail

- heard means that I have some details, but not the full story

[My other car WAS an IAV Stryker]

Blackout of August 2003

'...the mobile carrier did have a redundancy system in place but it failed, causing an overload that reverberated across the whole network.

'“We’ve worked with our vendors to build redundancy and resiliency to make sure that these types of circuit failures don’t affect customers. This redundancy failed us and resulted in an overload situation that was then compounded by other factors,” he wrote.'

A circuit fails, the backup circuits fail, the whole thing falls over. Yep, sounds familiar.

(In the case of AC mains, it's more like "backup capacity" on all the live circuits; they don't have unused transmission/distribution lines just waiting to be switched on. Generation, maybe -- "spinning reserve" -- but not the wiring.)

Personal story: my missus was an EMT on-shift at the time, and stayed on 72 hours straight (with naps). One of the most critical things they worried about on the southwest side of Detroit was not crime, as one might expect, but the US Steel plant on Zug Island and the nearby Marathon oil refinery. Both were kept under constant surveillance by the cops and medics for issues. The steel plant was particularly risky: there's a vapor burn-off flame (like the refinery), and instructions were that if it ever went out DRIVE AWAY AS FAST AS YOU CAN because the whole area was about to become a smoking crater. (Yes, it's Detroit -- go ahead and joke that it would probably be an improvement...)

[RLWatkins]

The FCC boss is a former AT&T lobbyist, isn't he?

That would explain why he's hammering T-Mobile while AT&T gets away with whatever it likes.