back to article Xero, Slack suffer outages just as Let's Encrypt root cert expiry downs other websites, services

Websites and apps are suffering or have suffered outages around the world for at least some netizens today due to connectivity issues. Though the exact causes of the IT breakdowns are in many cases not fully known right now, there has been a sudden uptick in downtime right as Let's Encrypt, which provides free HTTPS …

  1. Anonymous Coward
    Anonymous Coward

    More than a prod needed

    I think one of our boxes that went in a huff with the cert change needs taken round the back and put out of its misery.

  2. Dwarf Silver badge
    Facepalm

    LetsHaveAnOutage

    LetsEncrypt should check out one of those services that allows you to generate and automatically renew your certificates, then they can avoid that sort of problem.

    Oh, hang on a moment ... How embarrassing. Good job nobody is paying for the service

    1. Anonymous Coward
      Anonymous Coward

      Re: LetsHaveAnOutage

      You do not understand. Go.

    2. Greybearded old scrote Silver badge

      Re: LetsHaveAnOutage

      Many failures to recognise sarcasm here. Maybe use the joke or troll icon next time.

  3. Smelly Socks
    Devil

    expiry

    "there has been a sudden uptick in downtime right as Let's Encrypt, which provides free HTTPS certificates to a ton of organizations, let one of its root and intermediate certs expire"

    they totally shouldn't have let this expire. That's so slack of them.

    -ss

  4. emfiliane

    My god, everyone had a whole year to fix this and LE has been on a media blitz the whole time to try to get sysadmins to give a shit. It's not like this is some overnight thing that was unannounced.

    1. Warm Braw Silver badge

      However, it has neatly highlighted yet another now-vital part of the infrastructure of the Internet that is both poorly understood by many of its users and entirely dependant on corporate sponsorship for its continued operation.

      It's all a bit precarious.

    2. DougMac

      Its not necessarily that sysadmins didn't fix things.

      As I saw in my iOS devices and my 3rd party email clients, the system level software decided to latch onto the old no-longer-in-use cert and associated it with many connections internally. When the old not-in-use intermediary cert expired, my devices decided that they should still use it and complain and refuse to connect.

      Even though my Let's Encrypt certs were all good with the new roots for quite some time.

      We have had root CA certs expire in the past with some fallout, but without them being as widespread as Let's Encrypt has been, they have not raised that much noise. We will have additional root CAs expire in the future, with the potential for more issues with system code.

      1. emfiliane

        Yeah, I get that -- you can do only what you can do at your level, and there's plenty of incompetence and shoddy shortcuts above and below it somewhere, too.

        The shaky house of cards is very real.

    3. PRR

      Meanwhile.... Amazon Kindle Wikipedia look-up is STILL borked.

      https://www.amazonforum.com/s/question/0D56Q000084k4bj/why-do-i-get-invalid-certificate-error-when-accessing-wikipedia

      Why do I get "Invalid Certificate" error when accessing Wikipedia?

      "I noticed on October 2nd that my {Kindle} gets an "Invalid Certificate" error whenever I try to access Wikipedia."

      Going on TEN WEEKS now. Confirmed on 7g and 10g devices. Restarting no help. No newer software.

      Amazon seems to be repeatedly unaware of the issue.

      Granted not everybody uses this feature. But the books I read, I need a lot of look-up. And we know Kindle logs EVERY finger-stroke to headquarters. (Yes, every stroke/swipe/tap.) You'd think the sudden spike in errors would stir some interest?

  5. tip pc Silver badge

    Lets Encrypt consultancy?

    they should do this every few years and cream some consultancy fees from the ensuing borkage

    It great they provide an easy to use service for free

  6. Anonymous Coward
    Anonymous Coward

    Ah, that explains my day

    Couldn’t complete my (late) accounts on Xero, and couldn’t for the life of me figure out why access to my local SVN-over-HTTPS server was complaining about an expired cert, despite it being valid for another six weeks. Now I know!

  7. druck Silver badge

    Slack slaking off

    Slack when down this afternoon, still borked at midnight.

    1. druck Silver badge

      Re: Slack slacking off

      And no change this morning.

      1. druck Silver badge

        Re: Slack slacking off

        Changed DNS to 8.8.8.8 and have got it back.

        1. DougMac

          Re: Slack slacking off

          Yeah, because your ISP didn't flush their DNS cache or install NTAs for slack.com after they borked themselves with bad DNSSec setup.

          DNS at the top domain level is cached for a day or two with a TTL of 2d.

          Google DNS (and other large providers) probably slapped some NTAs on slack.com to cut down on the complaint levels they were probably getting for slack.com's mismanagement of their DNS.

  8. Wzrd1

    Always fun watching SA's bungle SSL certs

    Some years back, I had to give an entire LAN/WAN shop a quick course on SSL, especially as regards to the resource human friendly hostname.

    At the time I was in information security, having moved up from that same shop. Implementing it isn't rocket science!

    SSL loses trust, check cert, root cert, awshit - got the new one, trust it, flush the proxy server cache, go to lunch early.

  9. This post has been deleted by its author

  10. Richard Cranium

    Why make changes on last /first day of the month?

    I used to work at Midland bank, payday was 20th of the month to steer clear of the end of month peak Bank activity. Likewise no system changes on a Friday night so there are people around to fix any unexpected problems

    1. Dave559 Silver badge

      Re: Why make changes on last /first day of the month?

      These are sound ideas, but here the problem is that LetsEncrypt was relying on another organisation's existing intermediate cert (created a long time ago, when people perhaps didn't think of that) and it happens to expire when it happens to expire, so there's not really a lot they can do about that.

  11. Chris 3

    Ah, this would explain…

    …. The alerts my iPhone calendar was giving me about invalid certificates- it is subscribed to the calendar at my daughter’s school and uses letsencrypt

  12. Richard Pennington 1
    Facepalm

    Finding dependencies in slow time ...

    Has the original Rosetta Stone expired yet?

  13. lostinspace

    I've never understood why certificates need expiry dates.

    Given they can be revoked, why do you need to guess when issued how long it will need to exist for?

    If the argument for expiring them is that hash and signing algorithms improve, them simply revoke the cert when it is considered sufficiently weakend by advances in cryotography.

    So many outages have been caused by certificates expiring.

    1. Wim Ton

      It is a business model for commercial CAs, they can sell you a new certificate every 2 or 3 years.

      Furthermore, it is NIST folklore that keys must be changed at least every 2 years (preferably more frequent)

      1. Androgynous Cupboard Silver badge

        Cynicism not without merit, but not really. Things change, companies change name or change hands, trust changes with it and certs have to reflect that.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like

Biting the hand that feeds IT © 1998–2022