Only '3% of web servers in top corps' fully fixed after Heartbleed snafu A study of the public-facing web servers run by some of the world's largest firms has suggested only three per cent of the machines have been fully protected against the OpenSSL vulnerability known as Heartbleed. The research, carried out by security specialists at Venafi Labs, examined 550,000 servers belonging to 1,639 …
"Mopping up after an incident isn't as simple as it used to be," ...bla... "You can't just stick a patch on and call it done."
Well, it depends on the issue and the patch, doesn't it? A current patch for a buffer overflow would be very simple to apply and forget, as usual. Otherwise I'd like that guy to tell me what exactly changed in the environment that would cause his comment to be true...
Does this guy happen to sell custom vulnerability mitigation stuff/consultancy services or something?
In this case the issue is a vulnerability that was around for 2 years and could have led to certificates being stolen. If those certificates aren't revoked then patching the hole that leaked them doesn't make you secure.
Did you read the article?
-----BEGIN CORRECTION-----
It *is* the certificates that need to be revoked, but it's the corresponding private keys that present a problem if they've been stolen, because the certificates are already public.
-----END CORRECTION-----
This post has been deleted by its author
The issue isn't with how to roll out changes, its that when the change has been rolled out the compromise has already allowed certificates to be forged, it is the same as a jewellery store putting in better security after the jewel thief has already struck, good practice for the future but isn't going to get the jewels back. Here you had to apply patch, create a new signing key, create a new CSR and submit to a certificate authority for new SSL certs, apply those and then ask for the old certs to be revoked. Try doing all that with a group policy.
You don't need group policy for that - simply because Windows was never vulnerable to Heartbleed in the first place.
Anyway GPOs are just a layer in Windows system management capabilities, something that comes built in with any domain controller. Then there's System Center....
Anyways Windows has built-in features for PKI management - and Windows PKI aware applications can take advantage of it, check for example http://blogs.technet.com/b/pki/ about how you can manage certificates in Windows.... and automate a lot of tasks.
You may not need Puppet for that, but why would you use a monoculture management toolset when you can use a toolset that works with everything? Why restrict yourself? What benefit does that give you as a business owner? Keeping geeks with biases happy isn't a viable rationale.
"Group Policy"? What are you, from the aughties? Puppet, ya old crank. Puppet. That's how proper sysadmins handle systems management today. "Group Policy". Next you'll tell me you still develop software for legacy Wintel systems! [1]
Group Policy is limited to low value-for-dollar high-licensing-requirement Microsoft OSes. That's it's flaw. It would be great as a management infrastructure if it could support Linux, BSD and so forth, but it can't. So why both investing in it? Puppet can handle what needs to be handled, is cross platform, and allows you to get all the benefits you would have had from GPOs and GPPs.
Using Puppet you can do what normal people do: Buy RHEL support for Dev and Test, then run CentOS for production. Manage the whole lot with the same Puppet scripts. Drive your licensing costs into the floor, keep your support costs virtually nonexistant.
[1] Okay, that's disingenuous. I know nobody with cognitive capacity is still developing new Wintel software. But it's still hilariously fitting given the whole "group policy" thing.
.... something alike apt-get update apt-get upgrade is always all they need to keep their server secure. Many also wrote it here (check the threads about Heartbleed) when pointed at the risks Heartbleed exposed and the need of a full remediation including changing keys - but they really didn't understand what the vulnerability was about - many thought patching OpenSSL was enough.
LOL! I work for a company with over 70.000 employee... how large is yours? And working exactly in IT SEC, I see a lot of complacency among sysadmins. Sure, they got Puppet (just, last time I checked, it doesn't buy new certificates on your behalf) and many other tools. Just, those tools alone does nothing. Unless you understand what you need to do, and how to do it, you can use the tool to do it. Just, many, don't.
Wake up and understand how the world really works...
You could work for a company with millions of employees, that doesn't mean a damned thing if you can't do your job.
You're absolutely right that you need to worry about things like "buying certificates." Except that you can script that by having Puppet call the cert site's API, request a cert renewal, etc...or even just buy the cert and push it out using Puppet manually. It's like two lines of code to ensure that the old certs are removed and the new ones installed.
Complacent sysadmins are a problem. But the biggest complacency issue are sysadmins who refuse to learn new technologies that can help them be better at their jobs. Puppet and similar tools are the future. GPOs and other monoculture tools are relics of a best forgotten era.
Actually, my role and my pay says I can do my job. The very fact that you would put CentOS in production because you're too mean or greed to pay for RedHat on production machines (but use it on dev/test ones!) says *you* can't do your job - I would never deploy something that way just to save some money. Everybody does it? Only foolish, greedy companies. Good ones deploy on exactly what they have tested for (and obtain true support if something doesn't work as it should).
Sure Puppet can do a lot of things - yet it doesn't think by itself. If you don't understand cert needs to be revoked and changed, Puppet won't do itself on its own. If someone didn't take the time to understand what Heartbleed really meant, well, he will never configure Puppet to update certificates, and anyway, obtaining new certificates is not usually just tech stuff, the easy part, but there are usually, at least in sount IT environments - some rules to follow because certificates are a "sensitive" resource, someone has to validate requests and issue them (or pay for them).
Windows AD capabilities and its system management tools are not something I would call a "relic" - you look like someone who discovered Puppet yesterday and think he found the "definitive TOOOOOOL!", LOL! Are you going to start a religion about it? Puppetology?
If you're the kind who can't see beyond his single tool of choice, well, it just shows how limited your horizon is - and probably you never learnt what Windows AD can really do, especially in latest releases, because your hate blinds you. Don't refuse to learn it just because it's Windows... it just makes you look like you can't do your job.
"The very fact that you would put CentOS in production because you're too mean or greed to pay for RedHat on production machines (but use it on dev/test ones!) says *you* can't do your job - I would never deploy something that way just to save some money"
So, you're actually, factually an idiot that is overpaid and doesn't know how to do their job. Congratulations, you are the living embodiment of the Peter principle.
Why do you need RHEL in production for VMs you are never going to change or manage directly? And why the metric monkey fuck are you changing or managing your production systems directly? They should all be automated and orchestrated through vCAC and config-managed with Puppet. There is nothing about a production system that should ever require you to log into it. Logs should be collected centrally, configs pushed centrally, and everything about the system automated and disposable.
There's no business case for spending an RHEL license on that. You spend the RHEL licences on Dev and Test, which is where you actually to the work of building out new configs, testing your dependencies and checking for errors.
Also, I never said I don't know hot to use Microsoft's toolchain, you fucking numpty. I get paid to know how that all works. I have all-Microsoft production environments (well, for the moment they are) and I am willing to be I spend more time learning the ins and outs of that technology in my lab than you do working on it in production. Knowing that shit inside out and backwards is my job.
And yes, it's a relic. What AD and System Center can do, Puppet can do better. I don't tout Puppet because I like it, I tout it because it's the best. As a matter of fact, I hate how Puppet is implemented. I'm a GUI baby and I dislike this "lines of code" fuckery. But you can't argue with results, and Puppet is emphatically superior to Microsoft's monoculture management tools.
You are a Microsoft fanboy. You always have been. You can't see past your own emotional investment in the company and it's tools.
I was a Microsoft fanboy, once. I still deploy their stuff widely. But it has been a long time since I was narrow-minded enough to think them the solution for all ills. What matters is getting the best results in the shortest time with the lowest expenditure. If that isn't your goal as a systems administrator than you are doing your employer a massive disservice and you should quit now if you hope to retain a shred of personal honour and dignity.
Learn a bit about how IT has evolved in the past 14 years since Microsoft Monoculture was ascendant. You might be surprised at how amazing it has all become.
Since Heartbleed we have found that a significant number of servers were never vulnerable, either through configuration or because they were running older versions of the library. In fact, we found that *NONE* of our Windows-based web servers had the flaw, and only slightly more than half of the linux-based ones.
Yes, the servers needed to be re-keyed, and it was a pain (especially on the multi-server wildcard certificates), but trying to claim and 97% remaining exposure rate when the number was never that high even without patching is nothing more than spreading Fear, Uncertainty and Doubt - and he is undermining his credibility and that of Venafi.
TLS/SSL is a protocol, OpenSSL is an implementation of that protocol... and not the Windows native one.
IIS is not the only wbeserver, but it's the native one - and usally the reason you use Windows as an (expensive) web server. And because most Windows web servers are IIS ones, they coudn't and can't be affected by any OpenSSL vulnerability.
Did that post implied the machines were not running IIS? Sure, they could have been running old versions of Apache not affected by Heartbleed (but with other bugs...) or couldn't they simply have been running nothing of OpenSSL?
So only 15k of the 550k servers have changed their private keys, and on that basis it's assumed the remaining servers are vulnerable!?!
As others have mentioned, many of those companies are likely running older Linux OS versions, which will be using pre-v1.0.0 OpenSSL which wasn't vulnerable. On top of that, of the top 1000 FTSE companies something like 35% to 45% of them are running IIS on their web servers, so no OpenSSL and again no vulnerability. So loads of those 97% of companies won't have changed their keys because they didn't need to in the first place!
This thread has been somewhat of an eye opener. People still use GPOs?
I thought they were a relic. Ive been writing custom scripts for years for pushing out updates and so forth.
Ive always hated GPOs because of the lack of feedback you get and the requirement for a reboot / logoff / gpupdate / wait for ages.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
