As Chromecast outage drags on, fix could be days to weeks away

Older models of Google’s Chromecast media-streaming sticks remain broken, and independent research suggests a fix could take potentially weeks to materialize. Second-generation Chromecast and Chromecast Audio devices stopped working on March 9. Google hasn’t said what went wrong, but an expired device authentication …

[Richard 12]

How does that work?

Surely anything signed while the intermediary was valid should still be valid now - unless actively revoked.

Seems like they (deliberately?) built a hidden timebomb into the product, thus making it faulty at purchase and they're liable to refund or replace every single one of them, no matter the age - at least in Europe.

[diodesign]

Re: How does that work?

AIUI, if the intermediate CA expires, the chain of trust is broken. Software -- standard cryptography libraries used by today's apps -- looks up the chain and sees the dead signing CA and rejects the signed device cert.

Per-device key-pair certificate --> signed by intermediate CA --> signed by a trusted root CA.

If the intermediate CA expires, the chain's toast, the device cert is invalid, and the device isn't trusted by the client app. It's not that the cryptography suddenly magically stops working mathematically, it's a decision by the client apps (specifically the libraries they use) to reject certificates that have an invalid chain of trust.

Hence why one of the workarounds is to push out apps patched to overlook the certificate expiry or hard code an acceptance of the last-known good certs.

C.

[sev.monster]

Re: How does that work?

This, and while it's possible to ignore expired/invalid/failing certs, that opens the hardware up to exploit. Which is why I imagine they are taking so long to """fix""" this. The last thing they want is third party vendors and open source software to shoulder in to their ecosystem, as they have shown us many times now.

Remember unauthenticated guest mode? Remember connecting to unsecured devices not using a Google-supplied cert? Remember how the early protocol versions got mapped out by hardware hackers, just for Google to push unskippable forced updates to all Chromecasts that requires the latest protocol, which also require a Google account now? Pepperidge Farm remembers.

[Richard 12]

Re: How does that work?

That's still insane.

What date does the root cert expire? It'll take down the entire world!

It literally does not matter if today is after the expiry date of the intermediary. If the signing itself was done when it was valid, and none of the chain has been revoked, then it's still valid - nothing has changed since last week.

Windows installers work this way.

When an intermediary expires, the chain cannot be used to sign a new installer - but all already-signed installers remain accepted - unless actively revoked.

[collinsl]

Re: How does that work?

No it isn't, that's how TLS certs work. You are supposed to have replaced the certificate with a new one with a new intermediate CA with a new lifetime before the old one expires.

I agree that Google should have stuck with their original position of not checking for date-based validity of the CAs however it doesn't change how TLS certs work or that by default you should respect the validity dates of CAs.

[CrazyOldCatMan]

Re: How does that work?

You are supposed to have replaced the certificate with a new one with a new intermediate CA with a new lifetime before the old one expires.

Google *should* have:

Generated a new intermediate certificate with a nice long validity

Worked out how to distribute said cert out to the end-user devices (if they were hard-coded in the devices, tough luck. WEEE disposal time..)

Communicated to end users to say "apply the pending update if you don't want your device bricked"

But, in true Google style [1] they sat on their hands and did nothing. Probably because the employee who knew about it was kicked out in one of the many "rightsizing" exercises and they, despite all their computing power and technology, can't seem to manage to track certificates [2]. Calling google chaotic is an insult to chaos.

[1] "Alleged" style.

[2] Does your new product contain or use a Google certificate? If yes, tick the box and put in the expiry data and product ID. Then some poor muppet gets the weekly "these products are going to be affected by a certificate expiry in 6/3/1 month(s)" report and is required to do something to inform someone that action is needed.

[Anonymous Coward]

Re: How does that work?

By your logic, a leaked private key would be valid forever as any miscreant would simply set the clock on their computer before signing their malware.

(Yes, CRLs are supposed to invalidate any leaked private key but they're not used anywhere near as much as you'd expect)

[Richard 12]

Re: How does that work?

Not in the slightest, because the signing timestamp is not your local clock. It comes from a trusted third party (or the certificate issuer).

I'm describing how they actually work.

[Anonymous Coward]

Re: How does that work?

Yes, we're talking intermediate certificates - which are used to sign certificates.

So if the private key to the intermediate is known, you can use it to sign a certificate claiming to be at any time you like. You basically become that "trusted third party"

[entfe001]

Re: How does that work?

IIRC (I've been no Windows user for many years now), Windows signed executables have an additional timestamping signature to attest when the file was signed. With it, the certificate chain is validated against the certified timestamp, provided it is properly signed itself. Executables without a signed timestamp, which I've seen sometimes, have its signature appear as invalid as soon as any of the certificates in its chain expire. Otherwise, the signature date is as trusted as the system used to make the signature.

Not sure what happens when it's the timestamping CA the one which expires.

[Justin Pasher]

Re: How does that work?

Root CA certificates have a much longer expiration date. They are also kept much more secure (i.e. they only directly sign intermediate certificates and (should) be kept offline or in secure hardware so they are much harder to compromise).

The idea is that the TLS libraries used by software (e.g. web browser, curl, wget, etc) to interact with a TLS endpoint includes long lived Root CA certificates in its trust store. Because it is much harder to update the trust store for all of the installations everywhere if there is a compromise, an Intermediate CA certificate is used to actually sign the final certificate. That Intermediate CA certificate has a shorter expiration (ten years is on the higher side, and twenty years is actually insane). If the Intermediate CA certificate gets compromised, it can be revoked much easier, as it's not typically part of a TLS library's trust store. The TLS library only trusts the Intermediate CA certificate because it's signed by a Root CA certificate, which it directly trusts.

This is where the concept of a "certificate chain" is used. If we take a web server as an example, it will typically provide the site certificate and any intermediate certificates used to trace it back to the Root CA certificate. The browser will look at each certificate in the chain, and if the final one is signed by a certificate it trusts, it considers the site certificate trusted. If an intermediate certificate is compromised (which is very rare), it can be revoked. The device is supposed to use the CRL to verify whether a certificate is revoked. How well this works in practice, I'm not sure.

The big takeaway is that certificates must have some sort of expiration. The regular certificates are short lived (anywhere from three months to one year nowadays) because they're easy to replace. The Intermediate CA certificates are trusted longer (but not so long that a compromise would let a vulnerability linger for a long time). The Root CA certificate is trusted even longer (because it would involve updating millions or even billions of devices). Because of its importance, the Root CA certificate is very heavily protected. If that is ever compromised, you will probably not be a CA anymore.

[Richard 12]

Re: How does that work?

These were originally 20 years, then reduced to 10.

So, how long is "much longer"?

Anything that needs replacing at very long intervals is absolutely certain to cause a disaster. By the time it needs doing nobody remembers it, and everyone involved last time has left the company.

That's basically the entire reason for the very short-lived leaf certificates.

[Kevin McMurtrie]

They forgot

They forgot the final software update that makes the device slow and progressively more unreliable so people buy a new one.

[sev.monster]

Re: They forgot

If you put expiration dates on your certificates, you don't have to even bother!

- John Google

[Empire of the Pussycat]

Re: They forgot

Apple holds multiple patents on that.

[WolfFan]

Re: They forgot

Please provide a few examples of Apple devices borking due to expired certs. Two or three would do.

[Anonymous Coward]

Re: They forgot

He was replying to "They forgot the final software update that makes the device slow and progressively more unreliable so people buy a new one."

Nothing to do with certs.

[WolfFan]

Re: They forgot

Cool. Please provide a few examples of that. Two or three should do.

[Richard 12]

Re: They forgot

There was an actual court case. Google it!

[WolfFan]

Re: They forgot

That’s _one_. M’man said ‘multiple’. I see three downvotes at this time, but just one example…

Typical of the Apple hate around here.

[WolfFan]

Re: They forgot

Ah. Six downvotes, just one example. Gee. I wonder why that might be…

Apple hate. Just plain Apple hate, that’s all.

[Pascal Monett]

"earlier Chromecasts have a longer certificate validity, of 20 years rather than 10"

Okay, so why the deliberate decision to shorten the cert validity time ?

This thingamajig appears to be useful to many people. Did Google estimate that the newer version wouldn't last more than ten years, or what ?

Somebody made this decision conciously. I'd really like to know the reason.

[abend0c4]

Re: "earlier Chromecasts have a longer certificate validity, of 20 years rather than 10"

According to the linked Reddit post, Google software did not originally check the expiration date, so the duration was irrelevant. Google then, at some point, is supposed to have swapped the crypto library for one that checked the expiry of intermediate certificates by default, but noone appears to have realised the consequences.

Arguably this is a mis-application of certificates - you don't really want device credentials expiring at all. However, I expect it was a case of needing some sort of DRM solution, having a certificate library to hand, using it with the validation turned off without fully documenting why and then passing the code over to a maintenance team who decided if would make sense to use a standard crypto library instead...

Again according to the Reddit post, whereas this can be fixed (ultimately) in Google Play services for Android, each iOS app that supports casting will need to be rebuilt with an updated client SDK, which will be particularly inconvenient.

You have to remember the "I" in "PKI". If you're using certificates, there needs to be an infrastructure to manage them - they're not a standalone solution.

[A Non e-mouse]

Google are pushing the certificate industry to reduce certificate lifetimes saying we should all automate certificate replacement rather than doing them by hand.

It seems Google need to dogfood their own PR.

[breakfast]

Googleprecation

Good to see that Google maintaining their grand tradition of randomly and unexpectedly deprecating things that previously worked.

[Andy The Hat]

Re: Googleprecation

I accept that Google is wonderful and would never disadvantage ordinary people in any way.

But I do wonder whether the action of selling products with time constrained certificates or baked in reliance on an external web sever which can be killed at will, without telling the customer that the product they have purchased in good faith has a manufacturer controlled end of life date is a legal practise.

Deprecation and not providing support after n years is fair enough but including a time bomb ...?

[breakfast]

Re: Googleprecation

It's a natural next-step from their decision to shutter every useful service after a couple of years - why not build the shuttering in on a low level, that way they don't have to worry about forgetting and leaving some useful piece of technology accessible to users.

[Irongut]

I'm glad I replaced my Chromecast Audio with a Bluetooth RX/TX a few months ago. It's has more features than the Chromecast, doesn't tell Google what I'm listening to and it still works! ;)

[terry 15]

That's all well and good but it doesn't replace the functionality offered by the Chromecast audio - fire and forget audio streaming without relying upon the phone connectivity and no notifications interfering.

I'm not aware of any alternative to it that's as easily used.

(I've not received said email from Google mentioned in the article about the issue. Not as if they don't know I personally have one register to my account...)

[jeffdyer]

I just tested my ChromeCast Audio and it still works, no need to dig out the CD collection quite yet

[Alan_Peery]

Bluetooth + Chromecast a bigger win

I really prefer streaming my music to my speakers with Chromecast because that's a separate channel to Bluetooth. I can leave the Chromecast device quietly streaming music when I join a Teams call from my Android tablet using a Bluetooth headset.

Windows now has a native ability to send send separate audio streams different places, BT+Chromecast is a useful half-step...

[stewrogers]

NOOOOOOOOO!!!

Yea this sucks. For all the Bluetooth replacements out there, they don't match the functionality of the Chromecast audio. I have a couple of them, and they are keeping some old (but still great) hifi equipment relevant and its really rather depressing that that has gone away. It's bad enough Google took the decision to stop producing them without killing the ones out there already.

[jeffdyer]

Re: NOOOOOOOOO!!!

Have you tried yours?

[Alex 72]

Maybe have some Pi

If you can get one the Raspberry Pi zero W has WiFi Bluetooth and enough bandwidth for good audio. I don't see an easy out of the box one click installer but given Debian minimal and the Pi OS will run I am sure it can be done and when its done updates will likely outlive me if not you.

[Anonymous Coward]

Re: NOOOOOOOOO!!!

aside from one new nest mini and one new chromecast, it's killed two google homes and two chromecast audio. At least the old home minis can be a reasonable Bluetooth speaker though.

[Alex 72]

I know they have no market share and they are £10 more or something but I have not seen anyone have these issues with roku. Maybe google when developing this product finally became Yet Another Mega-corp and finally forgot the first motto was don't be evil.

Edit: I know they have done alot of evil already but often by omission or in ways they could justify, at least to themselves. This feels like the builtin obsolescence of the tech titans google replaced and there doesn't seem to be any attempt to disguise this.

[Miriam57]

Newer ChromeCast

Will a newer model Chromecast work?

If so, which one? And is it still sold?

[Alan_Peery]

Re: Newer ChromeCast

The newer Chromecast devices are HDMI rather than audio, so you'd need to split the audio channel out or have a speaker device that accepts HDMI.

[tomthumb47]

Thanks for the updates. I have 5 of these I use with vintage hardware, and they perform almost perfectly. We LOVE having them throughout the home. Really missing having music in the house.

[Anonymous Coward]

Working now.

Mine has apparently been fixed. It was working fine on Saturday afternoon. I just left it alone as Google recommended and waited for them to sort it out.

[Zippy´s Sausage Factory]

A cynical person might say that it feels like the delays will continue until the backlog of product gets sold.

Of course, that would be absolutely cynical and I couldn't condone such thinking.