Xero, Slack suffer outages just as Let's Encrypt root cert expiry downs other websites, services

Websites and apps are suffering or have suffered outages around the world for at least some netizens today due to connectivity issues. Though the exact causes of the IT breakdowns are in many cases not fully known right now, there has been a sudden uptick in downtime right as Let's Encrypt, which provides free HTTPS …

[Anonymous Coward]

More than a prod needed

I think one of our boxes that went in a huff with the cert change needs taken round the back and put out of its misery.

[Dwarf]

LetsHaveAnOutage

LetsEncrypt should check out one of those services that allows you to generate and automatically renew your certificates, then they can avoid that sort of problem.

Oh, hang on a moment ... How embarrassing. Good job nobody is paying for the service

[Anonymous Coward]

Re: LetsHaveAnOutage

You do not understand. Go.

[Greybearded old scrote]

Re: LetsHaveAnOutage

Many failures to recognise sarcasm here. Maybe use the joke or troll icon next time.

[Smelly Socks]

expiry

"there has been a sudden uptick in downtime right as Let's Encrypt, which provides free HTTPS certificates to a ton of organizations, let one of its root and intermediate certs expire"

they totally shouldn't have let this expire. That's so slack of them.

-ss

[emfiliane]

My god, everyone had a whole year to fix this and LE has been on a media blitz the whole time to try to get sysadmins to give a shit. It's not like this is some overnight thing that was unannounced.

[Warm Braw]

However, it has neatly highlighted yet another now-vital part of the infrastructure of the Internet that is both poorly understood by many of its users and entirely dependant on corporate sponsorship for its continued operation.

It's all a bit precarious.

[DougMac]

Its not necessarily that sysadmins didn't fix things.

As I saw in my iOS devices and my 3rd party email clients, the system level software decided to latch onto the old no-longer-in-use cert and associated it with many connections internally. When the old not-in-use intermediary cert expired, my devices decided that they should still use it and complain and refuse to connect.

Even though my Let's Encrypt certs were all good with the new roots for quite some time.

We have had root CA certs expire in the past with some fallout, but without them being as widespread as Let's Encrypt has been, they have not raised that much noise. We will have additional root CAs expire in the future, with the potential for more issues with system code.

[emfiliane]

Yeah, I get that -- you can do only what you can do at your level, and there's plenty of incompetence and shoddy shortcuts above and below it somewhere, too.

The shaky house of cards is very real.

[PRR]

Meanwhile.... Amazon Kindle Wikipedia look-up is STILL borked.

https://www.amazonforum.com/s/question/0D56Q000084k4bj/why-do-i-get-invalid-certificate-error-when-accessing-wikipedia

Why do I get "Invalid Certificate" error when accessing Wikipedia?

"I noticed on October 2nd that my {Kindle} gets an "Invalid Certificate" error whenever I try to access Wikipedia."

Going on TEN WEEKS now. Confirmed on 7g and 10g devices. Restarting no help. No newer software.

Amazon seems to be repeatedly unaware of the issue.

Granted not everybody uses this feature. But the books I read, I need a lot of look-up. And we know Kindle logs EVERY finger-stroke to headquarters. (Yes, every stroke/swipe/tap.) You'd think the sudden spike in errors would stir some interest?

[tip pc]

Lets Encrypt consultancy?

they should do this every few years and cream some consultancy fees from the ensuing borkage

It great they provide an easy to use service for free

[Anonymous Coward]

Ah, that explains my day

Couldn’t complete my (late) accounts on Xero, and couldn’t for the life of me figure out why access to my local SVN-over-HTTPS server was complaining about an expired cert, despite it being valid for another six weeks. Now I know!

[druck]

Slack slaking off

Slack when down this afternoon, still borked at midnight.

[druck]

Re: Slack slacking off

And no change this morning.

[druck]

Re: Slack slacking off

Changed DNS to 8.8.8.8 and have got it back.

[DougMac]

Re: Slack slacking off

Yeah, because your ISP didn't flush their DNS cache or install NTAs for slack.com after they borked themselves with bad DNSSec setup.

DNS at the top domain level is cached for a day or two with a TTL of 2d.

Google DNS (and other large providers) probably slapped some NTAs on slack.com to cut down on the complaint levels they were probably getting for slack.com's mismanagement of their DNS.

[Wzrd1]

Always fun watching SA's bungle SSL certs

Some years back, I had to give an entire LAN/WAN shop a quick course on SSL, especially as regards to the resource human friendly hostname.

At the time I was in information security, having moved up from that same shop. Implementing it isn't rocket science!

SSL loses trust, check cert, root cert, awshit - got the new one, trust it, flush the proxy server cache, go to lunch early.

[Richard Cranium]

Why make changes on last /first day of the month?

I used to work at Midland bank, payday was 20th of the month to steer clear of the end of month peak Bank activity. Likewise no system changes on a Friday night so there are people around to fix any unexpected problems

[Dave559]

Re: Why make changes on last /first day of the month?

These are sound ideas, but here the problem is that LetsEncrypt was relying on another organisation's existing intermediate cert (created a long time ago, when people perhaps didn't think of that) and it happens to expire when it happens to expire, so there's not really a lot they can do about that.

[Chris 3]

Ah, this would explain…

…. The alerts my iPhone calendar was giving me about invalid certificates- it is subscribed to the calendar at my daughter’s school and uses letsencrypt

[Richard Pennington 1]

Finding dependencies in slow time ...

Has the original Rosetta Stone expired yet?

[lostinspace]

I've never understood why certificates need expiry dates.

Given they can be revoked, why do you need to guess when issued how long it will need to exist for?

If the argument for expiring them is that hash and signing algorithms improve, them simply revoke the cert when it is considered sufficiently weakend by advances in cryotography.

So many outages have been caused by certificates expiring.

[Wim Ton]

It is a business model for commercial CAs, they can sell you a new certificate every 2 or 3 years.

Furthermore, it is NIST folklore that keys must be changed at least every 2 years (preferably more frequent)

[Androgynous Cupboard]

Cynicism not without merit, but not really. Things change, companies change name or change hands, trust changes with it and certs have to reflect that.