23,000 HTTPS certs will be axed in next 24 hours after private keys leak

Customers of HTTPS certificate reseller Trustico are reeling after being told their website security certs – as many as 23,000 – will be rendered useless within the next 24 hours. This is allegedly due to a security blunder in which the private keys for said certificates ended up in an email sent by Trustico. Those keys are …

[robsonde]

Private keys are private

With any PKI system if you need to move or copy the private key from the box it was created on, then you are likely doing something stupid.

It is fine to ask why they are being revoked and why the keys got emailed, but why did they have the private key!?!?!?

This company should never be trusted to be involved with any part of a PKI chain, they clearly do not understand the basics of PKI.

[Lee D]

Re: Private keys are private

Probably they had some kind of VPS or "easy to set up SSL" thing they were pushing, so it was actually them generating the certificates, and they got compromised.

Because, as you say, NOBODY but me ever has the private key or access to the private key to any of my certificates.

You SIGN the CSR with the private key. That does not reveal the private key but, proves that you are in possession of a key that is capable of decrypting a message that was encrypted with the certificate you're putting out there.

Then your visitors get a copy of your public key (in a roundabout way), which they use to encrypt their traffic to you. They know it's you because only you can read messages encrypted using that. And the CA has confirmed that you have both that key, and some form of ownership of the domain in question.

And because only you have the private key, only you can decrypt the visitor's messages.

The only places the private key need be plugged in are the endpoint that decrypts the messages (i.e. your webserver), and that key should be passphrased to prevent it being used if stolen. And in most cases you can't even run the webserver without the permissions on that key being ridiculously tight (generally 600 and owned by root on a Unix-based machine) - generally the software will refuse to start or service that site if the key permission is anything else.

So the only way to get my key is utter root compromise of the webserver it runs on (which, given that's what's holding the secure information anyway is game-over before you start). If you're using a VPS that could be done by compromise of the hypervisor hosting it.

But it shouldn't be in any email, it shouldn't be anywhere but the machine hosting it, it should be passcoded so that even this kind of "email it out" thing can't compromise your actual key (though that does mean entering a passphrase every boot or tucking it inside something like TPM or something on the webserver), and if you have half a brain you generate it on an entirely different machine to the one that's going to use it - hopefully offline, with a good RNG, and secure.

[Natalie Gritpants]

Re: Private keys are private

> But it shouldn't be in any email, it shouldn't be anywhere but the machine hosting it,

It should be backed up somewhere, possibly a disk/tape/usb-stick in a fire-proof safe. If the key is pass-phrase protected the pass-phrase should be backed up too (post-it note stuck to the back of the picture of the queen you have in your office).

[Lee D]

Re: Private keys are private

No more than normal processes.

You can always regenerate a certificate. At worst you might run into trouble with HSTS or pinning, in which case you probably have a backup procedure in place.

And your backups should be encrypted and are reading data to backup AS ROOT anyway. Thus it's not accessible to anyone who doesn't already have full access to the entire machine anyway, and encrypted anyway, and should still be passphrased anyway.

And the passphrase should be handled the same as the root password, the domain administrator password, etc. Which means DON'T write it down on anything you can't revoke or encrypt.

[Anonymous Coward]

Re: Private keys are private

"With any PKI system if you need to move or copy the private key from the box it was created on, then you are likely doing something stupid."

Not really, exporting a certificate to another machine is quite common if you are using a load balancer or active/passive setup without a proxy or you have wildcard certs that you are using on many machines. The private key is encrypted within the export bundle.

[bombastic bob]

Re: Private keys are private

" why did they have the private key!?!?!?"

that bugs me too. I've done self-signed certs and the cert doesn't need the private key (specifically the web server's private key) in order to be created. As I recall, you (effectively) generate a public key from your own private key, and include THAT as part of the 'request' submission to the certificate authority for your SSL server cert.

On the other hand, the CA has its own private key for generating the cert. If THAT got out, it would be just as bad.

[Mookster]

Re: Private keys are private

"With any PKI system if you need to move or copy the private key from the box it was created on, then you are likely doing something stupid."

(there, I fixed it for you)

In any PKI system it shouldn't be possible to move or copy the private key from the box it was created in. If you can, then you are likely doing something stupid.

So use an HSM.

[Jonski]

It affected me

And seriously, I rank it along with the likes of Trump tweeting the nuclear codes for gross negligence, malfeasance and delinquency. I'll throw in words like espionage, sabotage and malpractice too.

I'm in a time zone where we have about 4 hours from open of business to revocation of cert (although it's now been extended, a bit). Fsck these guys with a pitchfork, sideways.

Their turf war, my neck on the line. I got notified by an email that went to me (not my team) and landed in the Other folder for casual perusal when I got round to it. Luckily, I got around to it only an hour after I got in and caffeinated.

I'll now have to explain a risk mitigation strategy to our compliance team on Monday. I've done due diligence on dos and malware attacks and almost everything else under the sun, but deliberate betrayal by bad actors or rogue employees at the root is beyond my ken.

I'm in the process of replacing the EV certs with a bunch of 30-day ones from different vendors, and I'll not darken their doors again. At least my boss will shout the beer when it's over.

[robsonde]

Re: It affected me

Do you know if you generated the original CSR locally or did you use the web based tool on the site?

As far as i know they are only revoking the cert for which they have private keys.

the only way they can have private keys is if they generated the CSR for you via the website tools.

[Anonymous Coward]

Re: It affected me

"the only way they can have private keys is if they generated the CSR for you via the website tools."

Thank you, this is the bit I was missing and confused over. Where they got these private keys as the normal process, the CA never gets them, only the CSR. The web UI explains it, although not why they're keeping them after the process is complete.

[rmullen0]

Re: It affected me

Website tools to create a private key??? That makes no sense. Sounds like an idiotic idea to begin with. It wouldn't surprise me if they are in cahoots with the NSA or the like.

[Crypto Monad]

Re: It affected me

I'm in a time zone where we have about 4 hours from open of business to revocation of cert (although it's now been extended, a bit). Fsck these guys with a pitchfork, sideways.

Although I feel for your problem, I'm afraid the fault is ultimately yours: you gave away your *private* key.

Your private key should never leave any system that you control. If you give it to someone else, then they are likely to do something stupid with it (like send it via E-mail, as we just saw), and you have no control over this.

If you allowed someone else to generate it for you on your behalf, then the same applies - especially if they are as stupid as Trustico and actually *kept* the private key, which they absolutely NEVER need.

Certificate generation should work like this:

1. You create a private key on YOUR server

2. You create a Certificate Signing Request, which contains the corresponding *public* key only

3. You send the CSR to your CA

4. The CA sends you back a certificate (which contains your public key, your domain name, an expiry date and serial number, and the CA's signature)

5. You install the certificate on your server, which hands it out to anyone who connects.

Your server will prove, to anyone who connects, that it owns the private key which corresponds to the public key in the certificate. The connecting client checks that the CA's signature is valid, that the expiry date hasn't passed, and that the serial number is not in any published revocation list - this is the bit which allows your certificate to be terminated before the expiry date, as is happening right now.

The critical point is that the private key is born, lives and dies on your server alone. If there's any backup copy it's one which you took yourself, and hold securely yourself.

My advice: take the free replacement certificates now, and then migrate to LetsEncrypt. It takes a bit of getting to grips with initially, but you get software which automates the whole key generation / CSR / signing process, and you get fresh certificates every 3 months, at no cost. The main reason they limit their certificate lifetime to 3 months is to make it essentially mandatory that you automate the whole process.

[Danny 14]

Re: It affected me

changing certifcates sitewide every 3 months does not sound like fun. the paperwork would not be good (we have to check the siteseals ahavent changed etc for the web guys documentation. then we need to test with mobiles and other browsers for compatibility etc. a pain but thats testing for you)

[Nick Ryan]

Re: It affected me

Precisely. I work on the principle that the lower risk to the organisation is that the fewer people who have access to the private key the better. While it has a certain risk, that boils down to me, with instructions on how to access it if I leave or are otherwise unavailable long-term or permanently. This means that none of our developers, IT support agencies, finance staff or anyone else have access to the private key. If required I may install a copy on a server but that's done by myself and nobody else but does require trust in the server itself which is a weak point in the grand scheme even if relatively low likelihood of risk.

It's not kingdom building or protectionism, it's just a suitable level of paranioa. This way if (and I always work on *when*) the key escapes I have just myself or what I do, how I do it or where I do it, to investigate.

[akoepke]

Wow, glad I had already ditched those certs a while back

I received the revocation emails for a couple of certs that were almost due to expire. Thankfully I had stopped using them around a year ago as we are now using a free AWS certificate.

The fact they had retained a copy of the private key is an absolute betrayal of trust.

[mikegre]

Re: Wow, glad I had already ditched those certs a while back

Same for us. Our almost 3 years old RapidSSL is (was? since it is now revoked...) due to expire in April, when I received the renewal reminder from Dontrustico I set up a Let's Encrypt certificate procedure instead, which was actually much easier a task than going through their process... So easy that I added at the same time a SSL certificate to a handful of non critical domains hosted on the same server and that were still using only HTTP.

I did that 3 days before I received the revocation notice from DigiCert... Good timing.

[Nick Ryan]

Re: Wow, glad I had already ditched those certs a while back

I got fed up of the process and scripted the entire damn process using PowerShell and Let's Encrypt ACME interface. It's a pain, largely due to the totally inadequate, usually no, documentation but very easy once in place and I now have a system that can verify the deployment status of every certificate in use.

Now if only Microsoft actually implemented adequate certificate management in PowerShell without having to piss around calling external IIS utilities and arcane dead-chicken-waving context spaces...

[Anonymous South African Coward]

Nice way to start off your day - gandcrab on the one hand gobblin' wobblin' at your data, and now this...

Being sysadmin is fun sometimes.

[Anonymous Coward]

as the article says

"And, of course, don't forget you can grab free HTTPS certificates from Let's Encrypt that all major browsers trust."

[Joe Harrison]

Some of us value our time too much to use certificates from Let's See If It Feels Like Working

[Anonymous Coward]

Some of us value our time too much to use certificates from Let's See If It Feels Like Working

The thing is, if you can't automate the Let's Encrypt certificates then the 90 day hassle isn't worth it unless you work somewhere with lot's of spare time and your productivity isn't important. How much time can you dedicate to build your automation, updating the scripts and such? What if it's an appliance that can't be automated to change the cert?

A single 2 year certificate costs very little (<$30). I do outsourced IT stuff to clients and for $30 you'd get something like 15 minutes of my time. If you have plenty of devices and services requiring certificates then a wildcard - while more expensive - may be the solution. And LE doesn't offer those (yet).

LE is perfectly fine, but the 90 days is just way too short interval for my taste.

[Nick Stallman]

Of course you automate it. You'd have to be crazy not to!

Every certificate I deal with (thousands) is fully automated these days except for specialty types like wildcard and I have them partially automated.

Anyone manually mucking around with certificates in this day and age either doesn't have many, has some very pedantic requirements or doesn't know any better.

[~chrisw]

Pedantic is slightly unfair for some people, sometimes they're forced to be specific and have to handtool everything.

Some of the systems I've recently obtained certificates for have variable implementations of CSR generation - passably OK at best, deteriorating to crude or archaic at worst. They all have annoyingly long-winded and confusing routines to obtain the CSRs. This can even be from different products in the same family from a different vendor (and not cheap, either).

If only they could be automated! Once you step away from mainstream systems or devices, it quickly becomes pot luck. Sadly not every appliance will support offline generation or key replacement to facilitate the totally automated method you espouse.

I'm with you on automation but it seems a lot of other vendors still consider certificates an afterthought.

Heck, several of the appliances can ONLY generate CSRs which will always flag as invalid due to them not even having the CN as a SAN. And this is latest firmwares etc. Not much hope for the dream of total automation just yet...

[Anonymous Coward]

PoS

What a veritable PoS concerning an allegedly unveritable company.

Holding and then emailing private key WTF! They should be closed down and prosecuted.

[Snow Wombat]

Woooo not me!!!

Yaaay!!! I picked the provider that doesn't lick windows in it's spare time! ^_^

[Killfalcon]

Re: Woooo not me!!!

Well, not these specific windows, anyway. ;)

[brucedenney]

Tout, flog, peddle.

These emotive words say something about the perspective of the writer.

[Frank Bitterlich]

Tout, flog, peddle

Welcome to The Register, new reader!

[Destroy All Monsters]

Certificate Authorities are link banks: Print money and fuck you over

> Sends private keys in e-mail

> "We didn't say these are compromised"

What the actual fuck.

> drive to Comodo HTTPS certificates

Not even once.

https://nakedsecurity.sophos.com/2011/03/24/fraudulent-certificates-issued-by-comodo-is-it-time-to-rethink-who-we-trust/

[~chrisw]

Re: Certificate Authorities are link banks: Print money and fuck you over

I remember the social engineering experiment a chap undertook last year to see how gullible/lax CAs were in respect to issuing confusingly named EV certs that could be used for sophisticated onward phishing/scamming. I think all bar one CA was guilty of slipping up at issuance...?

I'm happier with my Comodo certs than I would be had I stayed with my RapidSSLs at renewal. 2011 is a long time ago in computerland.

[DerekCurrie]

Would that we could shut down Symantec entirely

Symantec has been a bane of the computing community, IMHO. I lost respect for them decades ago and they continually reinforce their poor reputation. End it.

[chivo243]

Re: Would that we could shut down Symantec entirely

Norton Utilities was the last good thing they did...

[Steve the Cynic]

Re: Would that we could shut down Symantec entirely

Norton Utilities was the last good thing they did...

Norton Utilities for DOS.

[Craig Foster]

Re: Would that we could shut down Symantec entirely

Norton Commander

[WallMeerkat]

Re: Would that we could shut down Symantec entirely

They became Malware very quickly, especially when bundled on new machines with no straightforward clean uninstall procedure - surely that is the very definition of malware?

[Anonymous Coward]

Re: Would that we could shut down Symantec entirely

"They became Malware very quickly, especially when bundled on new machines with no straightforward clean uninstall procedure - surely that is the very definition of malware?"

I worked at PC World when this started and I hated Norton with a passion even as a spotty teenager. It left the customer with three choices:

1. Subscribe to Norton

2. Pay PCW to remove as much of it as we could and install something else (that you also have to pay for)

3. Try remove it yourself, but know that if you bring it back to PCW, we'll charge you to run the recovery tool and restore Norton, leaving you with options 1 & 2, plus the cost of 3

To be fair, a lot of customers went with option 4: Slip me some money to re-install Windows from an OEM disc free from all the crud (using their key obv.)

[Dan Atkinson]

Trustico Statement

Trustico released a statement offering some information. Basically they stored private keys in cold storage.

https://www.trustico.com/news/2018/symantec-revocation/certificate-replacement.php

[S4qFBxkFFg]

Re: Trustico Statement

I chortled at the extract from Symantec's agreement - who else read it in Ali G's voice?

"IF YOU IS A CUSTOMER OF A RESELLER (AS DEFINED HEREIN), SUBSCRIBER REPRESENTS AND WARRANTS THAT IT AUTHORIZES SUCH RESELLER TO APPLY FOR, ACCEPT, INSTALL, MAINTAIN, RENEW, AND, IF NECESSARY, REVOKE THE CERTIFICATE ON SUBSCRIBER’S BEHALF."

[Anonymous Coward]

If only there were a way...

...to like add a SHA thumbprint along with my DNS A record. Just, you know, to double check that the certificate that you when you visit my website get was the one I intended.

I guess that'll be the dane.

[Anonymous Coward]

It would appear that we have two very different stories from DigiCert and Trustico, so at least one of them is lying.....

I think compensation is in order (not just a replacement cert), and legal action is sure to start shortly.

[Fred M]

"and the reassuring green padlock is displayed in visitors' browsers"

Padlock? Are you sure it's not a little green handbag? Like one of those being swung around today.

[Is It Me]

Now you have pointed it out I will always see it as a handbag, you are a cruel person.

[cantankerous swineherd]

the internet is unsuitable for anything serious.

[x 7]

"In future, Trustico will flog Comodo HTTPS certificates rather than peddle Symantec-branded certs"

Frying pan, fire.............

who the hell trusts Comodo?

[~chrisw]

A lot more people than before! Is there even a decent CA left who's proven to not have made mistakes recently and whose certificates also don't cost hundreds of pounds a year?

[wyatt]

With certificates meant to help improve trust on the internet, why are these idiots behaving like this? How are the general public who won't know too much about this meant to understand what is happening, I'm not sure I do fully.

[Alistair]

Gentlemen

You will both be expected on the green at dawn, with your seconds and surgeons. The weapon of choice in this matter will be either flame throwers, handgrenades or twitter. Your choices. The victor will be awarded the sweepings of the bus to stupidity from the previous day.

/sarc

There is absolutely no way I would have dealings with *either* of these entities in the future. I forsee legal and financial destitution in their futures.

[fobobob]

Pow!

Right in the trusticles...

[Ben Norris]

Lets Encrypt is not fit for purpose, not sure why you are recommending that.

[Anonymous Coward]

And why might that be?

[Ben Norris]

Cos if you look closely at the technical details they don't have the platform to guarantee reliability, nor the physical security to be trustworthy, and the unnecessarily short expiry that enforces automation means that any interruption to that will render you without a working site. There's no pragmatic fallback to make it robust, they've chosen to design it in a fragile manner because their ideals are more important than your uptime.

It's a bunch of idealist amateurs. If you are playing around, learning that is fine but for any real production site it just isn't credible.

[Anonymous Coward]

What a load of tosh.

[Ben Norris]

Thumbs from the cowboys on a bandwagon amongst us mean nothing to me.

[plams]

Trustico non trust

Their site is now down completely

Even when it was up we couldn't get through - chat, email, phone all blocked off

The coupon codes they sent use are invalid

There was no guidance on how to renew so you have to guess the product and try your coupon code.

(has anyone done this successfully yet ? If yes please state how you did it)

Is Trustico one guy in his garden shed?

Looks like we'll have to go with symantec's offer of free certs now even though they are going to be blocked in 12 days time or so by google.

[Anonymous Coward]

Re: Trustico non trust

I wonder if it has been DDOSed, possibly by someone looking for the Private Key stash or other weaknesses on their systems.

[TonyHoyle]

https://twitter.com/Manawyrm/status/969230542578348033

Trustico execute commands typed into a URL as root.

The incompetence knows no bounds.

With any luck their currently down site will stay down permanently.

[teknopaul]

I dont trust any of these bastards

One day im going to build a browser with Zero root certs, no cert chain support at all. The three ssl certs I care about pinned and all the rest show a red icon, insecure!, since that is the truth of the matter.

[Mike 125]

I don't want to kick a man when he's down but..

..oh go on then.

Inside a shed out back of Trustico.

[hellwig]

Google Apologists?

Cynics have suggested the Brit reseller ordered the revocation of its Symantec-umbrella certs so it could drive its customers onto Comodo certificates, and thus avoid the looming Google Chrome HTTPS certificate apocalypse without losing many, if any, punters. In effect, website owners have been caught up in a turf war between Trustico and DigiCert.

So Google and Symantec got into a fight and Google revoked their certs. But you blame Trustico and DigiCert?

Seems to me, Trustico wouldn't need to replace DigiCert's certs if Google hadn't blanket-banned Symantic.

Lets put blame where blame is due. Google and Symantec.

[Ilsa Loving]

Wait, Geotrust will be blocked too?

The way the article is written, it sounds like Chrome is going to block all certificates under the Symantec umbrella, including RapidSSL and Geotrust.

Am I reading this right? Cause that's going to be devastating.

[JCitizen]

Symantec..

can go to hell or high water as far as I'm concerned - They have stumbled around like a fool, and yet acted in the most conceited manner. It is almost like they really believe they are too big to fail.