Wonky whitelist update blamed for AT&T's nationwide 911 blackout

A wrongly updated whitelist was behind the five-hour nationwide outage of AT&T's emergency 911 service in March, a report by America's phone regulator, the FCC, has revealed. The whitelist of IP numbers plays a critical role in AT&T's backend systems, but was listed as containing only "customer" rather than "infrastructure" …

[David 132]

"This outage could have been prevented," said Pai. "It was the result of mistakes made by AT&T...."

Given that this is Ajit Pai you're quoting, are you quite sure you didn't mis-hear what he said?

I suggest that this would be more likely...

"This outage could have been prevented. It was the result of smothering government regulation passed by the Obama regime, which has for too long held back this noble and hard-working corporation from offering distinctive, value-added services to customers prepared to pay for a premium 911 service..."

[Anonymous Coward]

Are you kidding me?! Conservative controlled companies like every bank, AT&T, and our good pals at IBM NEVER EVER make any mistakes. Chairmon Pie has not 'kept to the official conservative script in this matter.' We all know this, or we are all told this every time a tory opens their flappy mouth to tell us how great they are and how we should all blindly support all business billionaire boys. To point out that these hypocritical toadies lack any morals or ethics and would gladly sell us their children, if they could up their stocks a few more points, is clearly the wrong thing to think. So, let's not think that, shall we? Thank you.

[Blotto]

@ac

You want to be posting that in the guardian comments section with that frothy nonsense. You'll be gladly received and upvoted galo re in that echo chamber.

[Anonymous Coward]

Troubleshooting the 911 core universal service platform backbone

'The whitelist of IP numbers plays a critical role in AT&T's backend systems, but was listed as containing only "customer" rather than "infrastructure" data and so did not go through vigorous checking before it was sent to the live network'

I smell retrospective ass-covering. Shouldn't they have run a diagnostic script after the upgrade, to verify that the 911 service was actually still working. That's what I would have done, but then what do I know.

'the initial outage was reported first to its 911 troubleshooting team [the] AT&T's voice-over-LTE (VoLTE) team. And then to the Universal Service Platform team. And then to the Core Backbone team'

Who then reported the issue to the Indian intern ..

[Fatman]

Re: Troubleshooting the 911 core universal service platform backbone

<quote>Who then reported the issue to the outsourced Indian intern tech support.</quote>

There, FTFY!!!! ..

[M7S]

"Americans in need must be able to reach Americans who can help"

I know its picky, but I find the wording disappointing. Leaving aside that not everyone in America needing help in an emergency is necessarily American, presumably other people are allowed to assist and form part of the employee/volunteer base of emergency services? I believe generally you have to be a US Citizen to be in the police but does the same hold for Fire and (where separate) Ambulance services?

I'm not American, but I have no problem helping, as a simple matter of human decency. Of course I appreciate that response times from the UK might be a little longer than desirable, but I assume that in areas of low population density (perhaps near either of the US's borders with Canada) there are some agreements to respond cross border if that will be more likely to save human life, possibly even to preserve property.

I have the feeling I'm going to be told this is a rather naïve view, but I'd be interested to learn whether or not cross border responses take place.

[Ankou]

So ISPs power 911

So remind me again how an ISP like AT&T doesn't fall under telco laws and title 2 restrictions again Ajit Pai? By definition if they power 911, Amber Alerts, Emergency Broadcast Systems, etc they are still under telco laws and restrictions. Literally on the day you vote to dismantle Net Neutrality: https://www.techdirt.com/articles/20170517/12241437395/fcc-ignores-will-public-votes-to-begin-dismantling-net-neutrality.shtml

[Alistair]

I first read this without

Considering *when* this happened.

911 issues being what they are (emergency service, critical) and handled in the wireless industry with some alacrity (If you can't get emergency services on the line it isn't a phone service etc), There is rather some import in handling things that affect 911.

VoLTE is *relatively* young tech in Norte Americana, and as such it still has some kinks (no, no, just weirdness not that stuff, we reserve that stuff for the C suite), thus I was reading this in the context of having happened, say, in 2014 or so.

This happened in MARCH of this year?

5 hours of outage?

No. Just no. No fines, no commercial penalties. *several* management types need to go to jail. Directly to jail, do not pass go, do not collect $2 million dollar golden parachutes. This is absolutely the result of a culture of putting the lowest cost body on the line to do the job.

[earl grey]

"whether or not cross border responses take place"

Unlikely now that the "i'm so scared" babies require passports to go whiz. Used to be no big deal going back and forth across the border (seriously, no big deal), but any more...ugh.