back to article Oh, Comcast. An Xfinity customer and working from home? Maybe not this morning

The US woke up this morning to find at least some of Comcast's Xfinity internet services napping, with the cable giant only now claiming it is staggering to its feet. At 0942 EDT (1442 UTC) on Tuesday, a staffer on its official Twitter support channel told angry subscribers, "We are currently having connection concerns around …

  1. chivo243 Silver badge
    Facepalm

    Where were you...

    ...during the last ComCast outrage! I think you missed a letter? High prices, capped speeds and now this. What's to be outraged about?

  2. Gene Cash Silver badge

    Wait...

    So the Sheriff's Dept tweets to people who have no internet connection to not call 911?

    1. FILE_ID.DIZ
      Devil

      Re: Wait...

      That metaverse man... It's life or death for reals!

    2. doublelayer Silver badge

      Re: Wait...

      Well, there's a logical part and an illogical part. They tweet to people so that people will see it on their mobile connection, which is probably still up when their home connection isn't. That is a perfectly acceptable way to send out a low-importance message to people when the cable connection is down.

      The illogical part is this: who calls emergency services when their connection isn't working. The ISP support/worst hold loop in the world, yes. Some other ISP because you're hoping they won't do this all the time, maybe. Emergency services should never be called. It's not illogical for them to tweet; it's illogical that they have to.

    3. Anonymous Coward
      Anonymous Coward

      Re: Wait...

      The big question (based on their graphic) is - what are they still using a Windows 95 computer for?

  3. Borg.King

    Xfinity drops me regularly

    5 minutes every 2 or 3 days.

    Just long enough to have all my remote desktops log me out.

  4. Number6

    I noticed it as I went to bed last night. A quick check showed that TV and phone had also gone down so I decided it was a good excuse to get some sleep. It was back up this morning when I got up. To be fair to them, outages are fairly rare here, we must have more modern kit in the local cabinets than some people.

    Looking at some log files, we lost service at 21:43 and it came back at 22:50 (PST).

  5. LordHighFixer

    Too funny.

    I realize that a lot of people actually use these entertainment networks for work. I even had a couple of people ask me about it. I have synchronous fibre to the premise and a backup satellite connection. I only go offline after 4 hours of power outage. (still looking at generators...)

  6. Youngone Silver badge

    Explanation

    Like McGuire, The Register has asked Comcast for an explanation for today's impressive outage...

    Well, you see, we're more or less a monopoly.

  7. Don Casey
    Black Helicopters

    Looks like something broader... or people are using Comcast pipes

    DownDetector shows a double spike for Comcast, 10pm and 6am.

    It also shows similar spikes for T-Mobile, Verizon, AT&T, and other single-spikes for other carriers that match one or the other of the two Comcast spikes (e.g. Frontier shows the 10pm spike).

    Feels like either Comcast infected large numbers of providers, and/or a major backbone/routing issue.

    1. whitepines

      Re: Looks like something broader... or people are using Comcast pipes

      Comcast provides some physical plant for other providers, so if they go down hard the little bit of Comcast in the chain breaks the connection for the other provider too.

  8. Chris Stephens

    I know exactly what happened..

    Great video Comcast has this issue. They were working on fixing it this week. https://media.ccc.de/v/32c3-7133-beyond_your_cable_modem

    Posted to FullDisclosure.

    > -------- Forwarded Message --------

    > Subject: Serious DOCSIS maintenance network issue

    > Date: Sat, 23 Oct 2021 21:09:16 -0700

    > From: Admin <admin@badmodems.com>

    > To: xxxx@cablelabs.com, xxxx@cablelabs.com, xxx@cablelabs.com, xxx@cablelabs.com, xxx@cablelabs.com

    >

    >

    >

    > Hi all..

    >

    > You guys hate me First it was Puma and the badmodems.com list and now this..

    >

    > I am sorry to directly email you. No need to respond. Its OK, I understand its a legal thing. I wont email again. Sorry for this hassle. Sorry for a long read.

    >

    > There appears to be a very serious gap in your security best practices and policies that could result in a very widespread serious incident that could effect all DOCSIS systems worldwide and result in a worldwide incident.

    >

    > This appears to be from MSOs deploying horrendously bad security on the maintenance network.

    >

    > The issues are being discussed publicly. This thread begins with discussion of firmware and then turns to the maintenance network which appears to have little if any security implemented possibly because there is no modern published best practices for the maintenance network beyond something from the 1990's. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-UDP-Network-Latency-Issue-Discussion~start=9780

    >

    > The maintenance network, which controls all the devices on a DOCSIS network, is susceptible to attack. In fact its nearly criminally negligent in its lack of security and appears to be based on 1990's security protocols of mostly security thru obscurity. .. A subscriber on the LAN side can determine his address on the maintenance network and can ping ANY CPE on the network as long as they are on the same ISP. The CPE are not walled off from each other in any way. This could result in a VAST compromise of the entire MSO network nationwide from a 0-day worm that self spreads via the wide open maintenance network connecting all devices. . . ALL susceptible devices on your network, 10's of millions, could be taken over in hours with a self spreading worm with a nearly impossible task of clean up and maybe a week of complete ISP downtime. This would also result in the largest loss of subscribers in history for cable as people flee to DSL and 5G that day trying to get internet. You would need new firmware for every device that addresses the issue, and getting new firmware will take weeks. All the susceptible CPE might be bricked with no hope of recovery once taken over. The current security practices are inadequacy. The news coverage would be devastating. Each modem/router could attack the subscriber side and scrape data and files. On the ISP side it would lock out all maintenance access and recovery of the devices, and the whole network, nearly impossible. It would setup a serious botnet - possibly the largest ever created when combined with the other top world wide ISPs. It might even result in a Ransom ware attack on a massive scale with all the CPE locked out from the ISP. A silent malware could spread stealthy and then sit on CPE and attack the subscribers quietly by doing fake DNS and even MiM attacks. This could already be the case. A botnet of CPE would be incredibly powerful

    >

    > This wide open gap appears to exist in most ISPs. So it is a CableLabs lack of proper security vision to keep up with modern threats by doing best practices for the maintenance network seems to be the main issue. 10G offers micronets and SDN containment of LAN devices,,, yet the ISP has nothing like it to protect its own network and its subscribers.

    >

    > Each ISP will need to do a 3rd party security audit and pentest of all the MSO's maintenance networks and secure them. The kinda emergency level, possible fairly easy temp fix is simple. Isolate each piece of CPE. Right now all CPE can see each other and spread worms. Simply doing a config change could wall off each device with NO downside. This might be able to be implemented maybe in a day. This alone would reduce the issue to nearly zero. BLocking access to the maintenance network from the subscriber is also key and most likely easy. MSOs REALLY need to do this and because these discussions are going on now, badguys could be reading, so RIGHT NOW is the time to secure MSO networks BEFORE a incident occurs.

    >

    > There may be simple quick solutions to avoid this doomsday scenario.. Make sure you read up to the current postings. https://www.dslreports.com/forum/r31122204-SB6190-Puma6-TCP-UDP-Network-Latency-Issue-Discussion~start=9780

    >

    > I will be following up to be sure you got this message.

    >

    > You can contact me for any further details or respond to this email.

    >

    > I am the guy who found the Puma issue. So you guys know I can be persistent and noisy. I would really like to hear that CableLabs is going to pursue a whole new approach to device security on the maintenance network including RAPID firmware deployment. EVERYBODY wins..

    >

    > Sorry for blasting the email. Sorry to start your monday kinda ruff. Think of it as a cool new feature.

    >

    > I have contacted all the top 10 MSOs and sent reports to the security teams. They are the guys who made this mess, but, they need a good best practice to follow and that does not seem to be there.

    >

    > Gone are the days of junk boxes with poor CPUs. MSOs are dropping POWERFUL devices with lots of RAM and Flash. They run Microsoft or Linux. They are connected to a massive bandwidth pipe. It looks possible to take over whole ISPs. These are prime targets no one has noticed yet apparently. Gone are the days of old.. These are high value targets and a bot net of incredible scale... Its time for a top down new approach to firmware and device security..

    >

    > Of course none of my doomsday scenarios most likely will ever happen.. And most likely everything is fine.. BUT MSO's can't just keep these maintenance networks so 1990s sloppy.

    >

    > IMHO..

    >

    > xxxxxx xxxxxxxxxx

    >

    >

    >

  9. Someone Else Silver badge

    Downdetector

    Interesting that Downdetector states "reported outages" The actual number will probably be higher, as all my attempts to report the outrages were met with a busy signal on the phone (a landline) using their nationwide 800 number.

  10. John Brown (no body) Silver badge
    Facepalm

    "Like McGuire, The Register has asked Comcast for an explanation for today's impressive outage..."

    "If you fucking people would stop fucking interrupting me for fucking status updates, the fucking job will get done fucking faster. Now FUCK OFF and let me work!"

  11. cjb

    Post a comment ? Why bother ? As has already been noted, we are at the mercy of a single provider ...

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like