After Debian's epic SSL blunder, a world of hurt for security pros It's been more than a week since Debian patched a massive security hole in the library the operating system uses to create cryptographic keys for securing email, websites and administrative servers. Now the hard work begins, as legions of admins are saddled with the odious task of regenerating keys too numerous for anyone to …
How apt! Working with Ubuntu Gutsy 7.10 I am currently dealing with another after-effect of the patch which can lock administrators out of remote OpenVPN servers if not caught *before* the security updates are applied.
When operating OpenVPN in the server and multiple-client configuration, if the clients (or the server) are allowed to apply the security updates ad-hoc then the existing tunnels will be broken and be unusable since as part of the (Ubuntu package) patch the openssl-vulnkey utility tests existing keys and refuses to use them if they are potentially compromised.
The problem with that is when the OpenVPN tunnel is the only secure access method to remote systems (SSH 'public' access not enabled) then the only way to remotely manage the remote system is immediately lost.
If you don't have some out-of-band fall-back method to access the remote system you've gone from bad to worse!
If you have some form of access to an updated system in this situation the work-around is to temporarily replace openssl-vulnkey with a sym-link that tells openvpn the existing key is okay:
$ sudo mv /usr/sbin/openssl-vulnkey /usr/sbin/openssl-vulnkey.bak
$ sudo mv /usr/sbin/openvpn-vulnkey /usr/sbin/openvpn-vulnkey.bak
$ sudo ln -s /bin/true /usr/sbin/openssl-vulnkey
Now OpenVPN will start with the iffy keys and you can at least update the keys, which as well as the client certificates may include dh1024.pem, ca.{key,crt}, server.{key,crt,csr}, ta.key
Useful reminder:
http://openvpn.net/index.php/documentation/howto.html#pki
Tip: Generate keys in a virtual machine image which is itself encrypted rather than on a regular system.
Is anyone keeping track of which SSL issuing authorities are providing instructions and free (zero-cost) certificate re-issues for affected certificates which need new CSRs?
The two year exposure is the big kicker really.
In a way the many eyes argument has been severely weakened.
Still such is the nature of opensource not too much repercussion for the person who made the patch, and anyone using the Debian packages shares if not a legal responsibility then an ethical responsibility for the vulnerability, I assume the patch in all its source code goodness was publicly available throughout.
And whilst a system would probably have been vulnerable for a cuple of years, it doesn't look like exploits were active - but of course the cleanup is not simple, and quite fraught with locking yourself out of the box.
It is actually prudent to keep and openssh connection to a remote system, whilst you do this.
Whilst in no way do I guarantee your connection will not be dropped, I have noticed that sshd options can be changed and the daemon restarted with the ssh client still connected (to test you ssh in at another shell).
Though, I do tend to have other options set to keepalive so that may be a factor.
There are some people on this planet who typed the code that did this, there is a group of people directly responsible.
Who are there, I want to see them publically lynched, I want them kicked out of the debian project and humiliated on slashdot, total destruction of their online status and shown as an example for all to see.
Who the f**k does something like "remove" a key component of the RNG in the first place, a dumbass, thats who, a total waste of space, do they even acknowledge the damage they have done?
Just a good job that most of the people I know wouldnt touch debian with a 10 year, out of date bargepole that debian is.
I know some people who are very angry about this and some people who are apologetic (freetards!!)
This is a disaster.
Our extreme, but effective solution, was to rebuild all our Debian servers with CentOS instead. My predecessors had decided to use Debian on a whim, as they were GNU fanatics (a number of poor quality GPL libraries and tools were chosen over those with BSD and ASF licenses for "political" reasons as well). We've had no end of trouble with the machines, and the OpenSSL mess was the final straw. In all, we had to rebuild 14 machines and decommission 2 more,
The problem with the Debian project, is that a number of its package maintainers believe they know better than the the original authors of the software. They muck about not only with the configuration of the software, but the source code itself. OpenSSL is a classic example, and the dubious Firefox patches that resulted in them having to call their version "Iceweasel" at the insistence of the Mozilla Foundation is another.
Only Debian and Debian based distributions, chiefly Ubuntu.
Redhat, Fedora, Slackware, Gentoo, Mandriva etc aren't affected.
Public humiliation would be too good for the Debian packager responsible, but all the same he/she should be stripped naked and placed in stocks in the centre of a local city where locals will be encouraged to shout abuse and throw objects.
I'm sick and tired of packagers, whatever the distribution, who mess with code. Complaints and support requests just end up going to the original project who have enough to deal with without broken hacks added by idiots. The number of times I've seen packagers attempt to fix bugs without reporting them upstream where they might be fixed properly and for everyone. Equally there are a large number of bugs which can be traced directly to mistakes or bad decisions made when packaging occurs e.g. Wrong compile options, or non-standard install locations.
Good packagers are appreciated for the work they do, bad packagers make life hard for users and developers alike.
Go to http://metasploit.com/users/hdm/tools/debian-openssl/ then take the second link from the top saying 'removed' which brings up the diff, then look at the new version:
#ifndef PURIFY
/*
* Don't add uninitialised data.
MD_Update(&m,buf,j); /* purify complains */
*/
#endif
then you notice that this would not have compiled at all because of imbricated block comment just created. Do they always compile with -DPURIFY, thus killing the lats MD_Update anyway?
My original comment was #1 here. Although annoyed about the extra workload the problem has required I'm amazed and somewhat disappointed at the small-minded responses attacking the Debian packager(s).
If anyone cared to do even a cursory search they'd find out that the original intention of the Debian patch was to "...remove valgrind warnings related to the use of uninitialised memory within the openssl libraries. This was done to try to make it easier to debug C applications that use the openssl libraries which is a good thing to do."
Also: "...that this was discussed with the !OpenSSL team and that no objections were raised at the time."
It apparently comes down to "...the main culprit seems to be a misunderstanding about which is the right [mailing] list to ask this question on, followed by misleading answers from the list."
For an overview see http://wiki.debian.org/SSLkeys#head-76415d1c2f25f12ebbcc99bb93959a97f18e3b88
For the specifics of how and when the bug was introduced see
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=363516
Quite some time ago, I stopped reading things like the Secunia mailing lists or Bugtraq, as I am only the TL and not the admins responsible for this sort of thing. I have too much to do in my day without needing to worry about this sort of thing. Perhaps I need to go back to reading these lists - or at least the weekly summaries - so that when my admins don't find out about such a critical bug, I can "let them know".
I'm about to walk out of my office, and have a word with the "security" admin, and find out his plans for patching the 42 servers with multiple vulnerable certificates on each. Fortunately most are generated in house, but there are a few that will require new certs from the authorities.
It's good to know that The Register is still an excellent site for pertinent info in this rapidly moving IT society.
Paris, because well I think even she would know about this sort of vuln if it was her very well paying job to know about it.
Oh Michael, can you please come in here, NOW?!
...........................................
This post has been deleted by its author
> then you notice that this would not have compiled at all because of imbricated block comment just created. Do they always compile with -DPURIFY, thus killing the lats MD_Update anyway?
The submitter is "kroeckx" (Kurt Roeckx). However, this is not necessarily the person who modified the code, only the person who reviewed and committed it. Given the poor job of modifying the code (it does not compile unless PURIFY is defined, as mentioned above), the person who did write it should be excluded from submitting patches in the future.
A bigger problem is Debian allowing idiots to modify one of the most linked and most security-sensitive libraries.
"Ha ha! So called "secure" Linux is still open to serious problems."
"The wonder of open source, you can at least find these problems."
"MS/Apple/SUN/IBM/etc wouldn't have admitted to this."
"A devout MS/Apple/etc zealot, I would never lower myself to use Linux."
"Well Ubuntu has always been problematic, that's why I prefer to spend 16 weeks compiling my kernel in Gentoo, so I can be all smug."
"At least MS/Apple/etc would have made the patching/correction/fixing this sort of thing easier to install."
"Shareholders would want blood, if this had been MS/Apple/etc."
"No wonder everyone thinks Linux is a joke."
Feel free to add more....
only 2 maybe 3 comments down there are calls or public floggings.
Its a security issue, with a computer system, it conceivably could have and in the past has been any computer system.
If such barbaric responses were made all and any indiscretion then no one would be programming computers and most the likely the person who suggested this course of action would be the most flogged.
"This is a bit of a nightmare for anybody who used Debian"
past tense. USED debian... I doubt they'll be using it any more! This distribution (and ubuntu) will likely be black listed from all fortune 500ers simply because they have no legal recourse.
This event has highlighted an important computer science mantra. Don't mess with other peoples cryptographic routines, you'll most likely break them.
This is an interesting bug as it affects anyone who has trusted anyone who used the affected OpenSSL library.
So *everyone* needs to check *all* keys and certificates on *all* of their systems.
It's a real bugger, fortunately we're small, and all our certs are signed by a self signed CA.
It only took one late night, and that was because there was no warning on the OpenSSH package update that it would prevent affected keys from being used, so I locked myself out of the servers.
>It vastly reduces the amount of entropy used when programs like the Apache >webserver, Sendmail, Exim and some implementations of Kerberos use OpenSSL >to perform basic cryptographic functions.
Exim uses GNUTLS in Debian[1],.. maybe before blowing your horn off you should do your research? Anyone that actually has "Sendmail" and not a wrapper around a real MTA installed should be shot anyhow.
[1]
libgnutls.so.26 => /usr/lib/libgnutls.so.26 (0x00002aaf7d373000)
As far as I see it, the biggest problem with debian is that they can't use software like everyone else, they "import" the official tree into debians tree, modify it so it runs better with debian and then thats their tree they use for the distribution.
Anyone remember what retarded noise came from them when Mozilla's icon came up in the press a while back.
paraphrasing "We are going to fork firefox to call it iceweasel because the icon is not free (as in speech)"
They are the most retarded linux devs on the planet, and I am glad that they do very little in terms of developing linux compared to the bigger players like redhat and suse and novell (even if they are tainted by MS, they still do more work than debian)
If you have a patch to stop valgrind making noises, why not just put the patch into the official tree, let it be audited by everyone and then pushed out through a normal release, why patch it in the debian tree and push it out from there, hardly anyone I know uses debian anyway, so the more eyes make bugs shallow argument doesnt necessarily fly (because with debian, there are not so many eyes)
Can anyone remember the last time debian made something revolutionary for the linux ecosystem? Graphical installers? dbus? gnome? kde? firefox? x.org?
Yeah, I am sure they "contributed" but by comparison, my take on this is that they are the poor family of the linux world who nobody really likes.
I am glad they are spiralling around the cesspit of death, got no money? awwww, poor debian, maybe you should pay more money on developers and less on "committees". Ubuntu stealing the wheat from debians chaff and I'm glad of it.
This bug affects ALL Linux distributions and indeed all systems that use SSL keys, including MS-Windows and MacOS.
Any user of your non-Debian system may have created their SSL keys on a Debian system.
For example, if you're running SSH on a Red Hat or MS-Windows server, you need to check whether any users have uploaded weak Debian keys to their .ssh user directories.
This brings the closed vs. open source argument sharply into focus. It seems to me that the situation is disturbingly like Brittanica vs Wikipedia. Brittanica is a lumbering, bloated dinosaur produced by a byzantine system of professionals that is full of information that was once correct, wherease Wikipedia is a dynamic collaborative project that is up-to-the-minute, but where any part of it can be wrecked by anyone, even if they have followed etiquette and asked on the Talk page about their changes first.
No, any other system that uses Open SSH are also affected as a compromised key generated on a Debian host that uses trusted key logon to access a remote server makes that server vulnerable to a brute force attack - I think someone said the key combinations can be iterated in about 20 min. There is a python script doing the rounds that checks for compromised keys - it's linked from the Debian wiki article IIRC.
(disclaimer: I am a Debian developer)
I find it highly ironical that the people asking that only developers "who know what they are doing" be allowed to contribute are precisely the people who do not appear to know much about the issue.
For instance, there is constant ignorance (or deliberate omission) that the Debian OpenSSL maintainer discussed the patch that was submitted to him with the original submitter, then discussed his proposal with the OpenSSL upstream (stating "I have no idea what effect this really has on the RNG") and getting what really looks like an OK from an OpenSSL developer ("If it helps with debugging, I'm in favor of removing them"). Yet I don't see anyone asking for that developer to be kicked from the OpenSSL project.
To be honest, as a software developer I really wish distribution maintainers would tell me more about patches they apply. But this is certainly not specific to Debian: I can name patches for my software that I had to find myself in at least FreeBSD, Mandriva, Fedora, Ubuntu and ArchLinux and that the maintainers never bothered to tell me about. But in that regard, the Debian OpenSSL maintainer has been absolutely faultless and I wouldn't trade him for ten of the people who want his head.
It's nothing to do with open vs closed source or even Debian
The Debian people brought their tool error problem to attention of OpenSSH, who ignored it. The Debian guy then did an incorrect fix.
1) Because the code was C
2) Because the lack of a comment in the source
3) Because the design by OpenSSH of using allegedly uninitailised RAM to make it more random was flawed.
4) The Debian guy didn't understand what OpenSSH had done (which creates a tool error when you compile.
5) Because OpenShh ignored his requests for help resolving the issue.
6) because a HW RNG is *STILL* not standard on all platforms. (Some old Intel chipsets, some VIA and some ARM have a HW RNG).
There is quite a flame war between some OpenSSH, Debian and random other folk on the blog of one of the OpenSHH team.
This was an unusual mix of issues not helped by the general crapness of C / C++ as a secure accurate programming language.
Almost all buffer overflow vulnerabilities are also as much due to the nature of C / C++ stupid array bound checking (by default none) and stupid string handling as inept programming. Java and C# only slightly alleviate these issues.
Mines the one with Modula-2, Ada, Occam and Oberon on the back
So basically, it affects everyone because people like to use debian to generate the keys, but if they generated the keys on fedora for example, or centos, there would be no problem.
So yes, it does affect fedora, centos, suse, everyone, but ONLY if you generated the keys on debian.
So there you have it, everyone stop using debian and use a better more up to date distro with it's head not stuck in the sands of time.
Simple really, these guys are a joke, this just proves it, no wonder they are getting killed by ubuntu.
>Can anyone remember the last time debian made something revolutionary
>linux ecosystem? Graphical installers? dbus? gnome? kde? firefox? x.org?
apt? dpkg? debian-installer has had a graphical option since Etch...
Also your argument about Debian "importing trees" is total crap. The Debian packaging tools advise "against" creating native packages that depend on the Debian build tools, and actually create a complete trail of changes that have happened to the original source....
It takes one slip up to bring out all the whiners and idiots... Debian isn't only vendor that patches software; I'm pretty sure Fedora patch their packages to put config files in really stupid places, maybe it's just magic?
El Reg is populated by super-humans that never make mistakes apparently.
>Simple really, these guys are a joke, this just proves it, no wonder they are getting >killed by ubuntu.
Ubuntu imported the exact same duff patch into their archive, which sort of proves how much auditing is happening on their packages. Which is ironic, we're being told "people that don't know what they are doing shouldn't get involved" but it's fine for Ubuntu to mass import packages with little or no knowledge of how those packages work?
Flawed logic.
This post has been deleted by its author
They don't generate random numbers. Any number a computer generates is a function of it's programming and it's inputs. If you are smart enough you could recreate those inputs and get the same number. People say the real world is like this too but experiments only work under the simplist of controled conditions.
No, it's not flawed logic, I'll explain myself better.
Right now, almost all the debian devs are leaving, or pi**ed at the project, ubuntu is creaming the crop from what I read and everyone who installs linux, either installs fedora, ubuntu or suse, only mega hardcore geeks who are more interested in playing with config files than use their computer use debian.
The fact that a dumbass debian developer and NOT a ubuntu dev introduced this flaw wasnt particularly hard to foresee, when was the last time ubuntu devs screwed up in this way?
Yes, they import debians tree wholesale, but thats an issue of trust, not stupidity, you think now that trust is intact? I wouldnt be surprised if you start to see the gulf between debian and ubuntu widen as a result, most of the interesting developments in ubuntu didnt originate in debian, they just base off it because it's free work and you can't blame them for NOT catching this problem, since the retarded debian devs didnt catch it for two years either.
Then again, it would have been better if they didnt introduce it at the same time.
The argument that the guy was "ignored" and basically went off to write his own fix I can't see how it has anything to do with it, the guy was ignored because at the end of the day, the "problem" he described was a valgrind warning, it wasnt a big red balloon with a siren saying "VULNERABILITY!"
So, this guy fixed valgrinds warning! CONGRATULATIONS, you just fu*ked everyone in the ass! well done!! So lets all group hug and thank debian for thinking that valgrind warnings are worth solving, if it causes a major security hole and millions of SSL certs are now hackable.
Phew!! I can't wait until the next valgrind warning! we're bound to win the desktop this way!!
Expect to see ubuntu doing more work in the future, I can't see them relying on debian like they used to, if I was them, I'd start to leech devs who can be trusted and leave debian as a rusty old hulk.
Ok, so I've been called "small minded", "not worth ten of the packagers" and that "I don't understand the issue". None of these are true, I was the first one to correct the original story a few days ago when it appeared on The Register. I'm an open source developer and know first hand the problems caused by bad packagers.
Packagers are put in a position of trust, users trust them to make their life easier by providing applications in an easy to install format and developers trust them to distribute their applications. Those who change code before distribution are a menace and rules need to change to stop the practice.
This packager and all those like him that mess with software are at fault, not because he never asked about it, but because by his own admission he didn't know what he was doing*. His emails to the openssl list never gave context for the line in question and the packager went on to comment out an additional line which was NOT harmless. It isn't enough, especially in such a sensitive application to get permission, you need to understand the code before you touch it.
My main point however, is that really doesn't matter who they spoke to or what they thought they knew. They still applied the patch locally instead of forwarding it upstream where the mistake would have been spotted. It's simply not good enough for packagers to change code without sending those changes upstream. It's bad practice and dangerous, if not to users of their own distro then to users of distros which never benefit from the good fixes.
I know a lot of good packagers, but there are also many who took on the role because it gives them power to mess with code and applications used by thousands of people but without having to justify themselves to other developers. It's a loophole in the quality control system of open source. Developers on a project have to write code of a standard high enough to pass inspection by the other developers but packagers don't have anyone in an oversight position reviewing their changes. They don't have the original code authors to deal with and I've seen numerous patches rejected by developers make their way into packages downstream because the packager thinks they know better.
* Packager said "I have no idea what effect this really has on the RNG".
Right, first off, anyone who has never made a mistake can stand down - and now you can p**s off because you obviously have never done anything constructive in your life !
As to the clueless f***wits slagging off Debian for making any change whatsoever to 'their' software - get a life. I use Debian and I have a good idea where to look for the config files for any package. Sometime I have to use a non-packaged app that hasn't made it into the repositories yet, and guess what, more often than not the various files are randomly scattered about the system according to the whim of the developer. Assuming I can find all the relevant files, I then have to sort out my own dependencies.
Surely every mainstream distro makes changes to the packages to fit in with that distro's view of what the world should look like - Debian is not alone in that.
Yes, it was a serious bug (and yes I've been busy making new keys), but having read some of the material available on the topic, it seems the openssl team must take a big chunk of the blame for not taking any interest in the query - and apparently not even being truthful about what the correct mailing list is !
...because the world is really analog. Using the least significant bits of a voltage measurement taken at the thermal noise level (using an inexpensive 24 bit ADC measuring a voltage drop over a 5K resistor) would be one way to generate a non-repeatable and non-predictable starting value for a standard RNG. Don't know if that's been tried.
No, no way are you serious, the debian packager who doesnt know his RNG from his TNG is 100% to blame, he is the one writing the code, he is the one admitting he doesnt know what the code in question does, he is the one who should know that this is a critical security software and shouldnt be touched unless you're an old hand. Then he ships crap sandwiches to everyone and openssl are to blame????
So, it's very simple Simon (pun?) did openssl team have anything to do with this patch and is the patch part of the official tree, if so, then yeah, they are partly to blame, if no, then debian is completely to blame.
And you are correct in saying that distributions "repackage" software for their distribution, but the difference here is that they ONLY REPACKAGE, they don't ALTER THE SOURCE CODE AND COMPILE A NEW VERSION just for them, packaging up and cleaning up the file heirarchy is nothing similar to changing the source and redistributing that.
It is simply not openssl team's job to respond to every mail on the mailing list, it's their job to look after their project and their code, if debian packagers decide to write an email and get ignored, it's not the fault of openssl that this guy then wandered off, dumped a whole world of hurt onto everyone and walked away, you're insane if you think that makes sense.
And the advantage of open source, peer reviewed software over proprietary efforts is.....?
This could have happened to Mac OS X, Microsoft Windows, even something like Juniper's JunOS - but the bottom line is that the OSS zealots have been spouting that proprietary software must be more insecure as there are less people reviewing it.
Guess what.....
<Karma is a beautiful thing don't you think...?>
P.S. Not getting into the security debate - everything has holes. My point is that this issue has blown a massive argument for OSS out of the window.
> Then he ships crap sandwiches to everyone and openssl are to blame????
No one said that the person that did the change *isn't* to blame. The problem here is that your and other like minded peoples responses are totally irrational.
People make mistakes.
>And you are correct in saying that distributions "repackage" software for their >distribution, but the difference here is that they ONLY REPACKAGE
All distributions patch the original source files to some degree. That could be something as simple as changing the location of the config files,.. and that could still introduce security problems that weren't present in the vanilla source.
Are you saying we should change everything to DJB-style licenses?
>It is simply not openssl team's job to respond to every mail on the mailing list, it's >their job to look after their project and their code,
They did respond. There seems to have been some confusion over what their response meant; Yes comment it out, or maybe comment it out only for debugging. The fact remains the damage is done. No amount of being a tit on either side is going to fix that. Debian announced the problem publicly, and have/are putting in place measures to limit the damage. (i.e. openssh now depends on a package that blocks known weak keys)
> it's not the fault of openssl
Instead of telling people off maybe you could help? Debian is a community project and I'm sure they could do with more people like yourself that understand everything and never make cockups.
Daniel,
Irrational? ok, being in the software industry, if I caused this kind of damage, you think I'd be collecting a phat paycheck at the end of the month? or my P45?
Seriously, this guy needs ejecting and find his fun somewhere else and yes, in this case, he isnt paid, but should he continue to work on the project?
We are constantly living in a society where failure is tolerated at any level and while I agree that people make mistakes, they should also be shown that severe mistakes, are taken and fixed severely. There are some bugs you shrug off and others which you cannot and the only recourse is your head, sorry, but this guy was editing and compiling code which he isnt even remotely qualified to do.
and as for helping, I am helping, by using fedora and telling everyone else to not use debian and the second fedora makes this mistake, I'll go hunting somewhere else too.
Sorry, but for a decade I have had a hatred of debian because of it's idiotic mindset and looking upwards at some of the comments, I read I am not the only one.
Note to all those clamouring for all distros to stop patching anything: if we actually did that, you'd all be running systems that were something between LFS and Slackware.
not that I have anything against those projects. They fill a need. But come on, that is not what most Linux users want out of their desktops.
Distro patching is, for most distros, an unfortunate necessity.
I'm sure all the developers posting in this thread about how terrible it is that distros ever patch anything will swear that they've never broken a tarball release, stuck the fix in SVN or on a mailing list, and told people to 'just use that'. I'm sure they've never screwed up their autotools scripts so that their app won't install properly on a system which uses /usr/lib64 rather than /usr/lib . I'm sure they've never ignored bug reports from downstream distributors for months at a time. No, they're all fine, upstanding citizens, I'm sure.
Unfortunately, such lowlife cads of software developers do exist. Upstream developers *do* make mistakes, and then they *do* ignore - or flat out reject, for their own reasons - fixes developed by distributors. In that situation, guess what? We have to patch it. And in umpteen other situations too, but you get the picture.
Distro patching is going to happen. Pushing for all distros to stop patching stuff is just tilting at windmills. It isn't going to happen. It would be better to focus on a calmer and more rational discussion over exactly when it needs to happen and when it doesn't, and it'd be even more productive to go to your mailing list and/or Bugzilla and make sure you don't have any requests from downstream sitting around unattended to.
In the current case, it's obvious that the Debian maintainer made a massive booboo. And it's certainly a good rule of thumb to say 'don't patch security-critical code'. But it's not a good reason to call for distros to stop patching everything, ever.
>Irrational? ok, being in the software industry, if I caused this kind of damage, you think >I'd be collecting a phat paycheck at the end of the month? or my P45?
Your employer should have liability insurance and contracts et al. worded in such a way that you aren't directly liable.
Read the first couple lines of *any* opensource license and you'll see the developers have nicely decoupled themselves from any responsibility. Many proprietary licenses shed as much responsibility as legally possible. Your's should too.
Aside from that the employment laws in the UK would make it almost impossible to just sack you and withhold your salary.
>Seriously, this guy needs ejecting and find his fun somewhere else and yes, in this >case, he isnt paid, but should he continue to work on the project?
Not being part of the Debian community is that really for you to have say so on?
It would be very easy to revoke this DD's rights within the project but that is for the project itself to decide not Random Internet Commentators (TM).
>We are constantly living in a society where failure is tolerated at any level and while I >agree that people make mistakes, they should also be shown that severe mistakes, >are taken and fixed severely.
Again, you're saying that instead of trying to limit the damage this has caused we should be telling people off. If someone gifts you a TV and it doesn't work do you go around their house and give them a mouth full of crap about it? You didn't pay for it, what right to you have to start a witch hunt against anyone? The licenses this stuff is distributed under actually forbids you any right to complain.
>There are some bugs you shrug off and others which you cannot and the only >recourse is your head, sorry, but this guy was editing and compiling code which he >isnt even remotely qualified to do.
More assumptions. Do you know anything of this guy's background? He could Schneier's genetic clone.
>and as for helping, I am helping, by using fedora and telling everyone else to not use
Well, even with Fedora you could have weak keys in your system; Keys created by debian users etch and beyond. Actually, with a Fedora system you are more vulnerable ssh-wise because you don't have the blacklist for weak keys. Maybe you should spend your time telling your sysadmin friends to watch out for keys generated on Debian boxes and to install blacklists where possible?
>Sorry, but for a decade I have had a hatred of debian because of it's idiotic mindset
So what you're basically saying is - I don't really care about this, I just don't like Debian and because of that I should decide the course of action taken in resolving the problem at hand. Is it possible you have been scorned by the Debian community for *your* lack of ability hence have this negative outlook towards them?
I spent today fixing a bunch of servers running Etch. It sucked, I don't like ssl, it's too much of a black box.
But, really, if Debian were not a good distro nobody would use it. The package maintainer f**ked up, sure, but how many of you people suggesting terrible punishments have never made a serious programming or sysadmin mistake? How many of you admitted it publicly to the world, and went to great lengths to show people how to fix the mistake?
Glass houses, I say. Stop being so bloodthirsty.
ok another post full of mistakes, here we go:
1) if I was at a company and personally created a disaster that caused millions of pounds of damage, it doesnt matter ONE LITTLE BIT that they have insurance you dumbass, it has NO RELEVENCE WHATEVER on the fact that at the end of the day, I am collecting my P45, they are covered, thats true, but I AM OUT OF THERE.
What world do you live in where that is not true? Are you seriously trying to call yourself an informed debater in this little conversation here? coming out with crap like that?
2) it depends on whether you want to be taken seriously or not, if I had an employee who did this and basically if you work for "me" you're my employee, if you caused that much damage, I'd eject you like bad wood, you'd be on the scrapheap and don't call me for a reference, I have my companies image to look after and having you around, doesnt pull in clients, it pushes them away because they imagine you'll do the same with them too. This is pretty standard and simple stuff.
3) If the TV was straight out of the box and then at night set fire to my house killing my wife and children, yeah I'd be around your house to find out what happened with that TV that made it do that and if you tell me it was standing in a puddle of water!! and THEN you give it to me, I'm gonna do something very nasty to you. So yeah, sometimes a gift CAN be complained about. Don't mistake free of money to be free of responsibility, they are not the same thing.
4) but he obviously isnt, because if he was, he wouldnt make such a f**king n00b mistake would he.
5) Since I learned that debian is for idiots, I pretty much stayed away from it and I have nothing to do with it, and I enforce that with everything I do, I dislike their entire band of brothers do much, I never run into this problem. However, some of my friends, have.
6) Thankfully, I've never had to directly deal with idiots from debian, so I've been mostly free from having to interact with them and be "infected" with cool 1337 ideas like removing parts of RNG code.
Seriously man, get a grip, millions of pounds of damage has been done and thousands of man hours wasted over a f**king valgrind fix, this shit does not happen to good developers. Stop protecting the weak, their death is SUPPOSED to happen. It's called nature.
This post has been deleted by its author
>1) if I was at a company and personally created a disaster that caused millions of >pounds of damage, it doesnt matter
Millions of pounds of damage? Evidence? Even if some bigwigs lost "millions of pounds" because of this patch they have no legal grounds for complaint. They used the code under the license it was offered.
RSA have sold console makers like Nintendo encryption and it hasn't worked,.. Nintendo haven't sued. Figure that out.
>ONE LITTLE BIT that they have insurance you dumbass, it has NO RELEVENCE >WHATEVER on the fact that at the end of the day, I am collecting my P45, they are >covered, thats true, but I AM OUT OF THERE.
You need something like 3 warnings of increasing severity to be fired unless your mistake can be considered "gross misconduct". I hope silly human mistakes aren't considered gross misconduct, I think that assumes intent to commit wrong doing.
Companies actually give courses on how to fire people these days....
>2) it depends on whether you want to be taken seriously or not, if I had an employee >who did this and basically if you work for "me" you're my employee
Far be it for me to tell you how to run your business, but you should have code review, a peer should be checking for an obvious blunder. It's very easy to miss things. The DD that patched OpenSSL tried to get peer review from the OpenSSL developers.
>3) If the TV was straight out of the box and then at night set fire to my house killing >my wife and children, yeah I'd be around your house to find out what happened with >that TV that made it do that and if you tell me it was standing in a puddle of water!!
You didn't check it for signs of damage before plugging it in? I offered you no warranty of the fitness of the goods and you should have expected as much. If you don't have the common sense to protect what is important to you these things will happen. What if it was stolen? You'd be liable for handling stolen goods.
>4) but he obviously isnt, because if he was, he wouldnt make such a f**king n00b >mistake would he.
So your informed critique comes down to the DD being a "n00b". Pat on the back.
>5) Since I learned that debian is for idiots, I pretty much stayed away from it and I >have nothing to do with it, and I enforce that with everything I do, I dislike their entire >band of brothers do much, I never run into this problem. However, some of my friends, >have.
You will never run into this problem? Chances are you have communicated with a server running Debian that has been using weak keys. Everyone is affected.
There must be billions and billions of pounds of damages outstanding from dodgy webservers, broken MTA's,... You'd think having to brute force thousands of keys would be a minor issue in comparison.
>6) Thankfully, I've never had to directly deal with idiots from debian, so I've been >mostly free from having to interact with them and be "infected" with cool 1337 ideas >like removing parts of RNG code.
So you don't actually understand what happened? The intent was to disable an almost unless part of the entropy generation process (uninitialised memory isn't a good source of entropy), but by mistake the DD managed to knock out a fundamental part of OpenSSL's entropy generating process.
>Seriously man, get a grip, millions of pounds of damage has been done and >thousands of man hours wasted over a f**king valgrind fix, this shit does not happen >to good developers. Stop protecting the weak, their death is SUPPOSED to happen. >It's called nature.
Earth quakes cause millions of pounds of damage, I think the damage this has caused is subjective at best. Maybe someone is replaying old SSL encrypted credit card transactions against the known weak keys in a hope of getting some usable data? Otherwise I'm totally lost as to where these $18.500.000 Million dollars (Eighteen Million Five Hundred Thousand
us dollars Only) have been lost. You would have thought all those big companies that rely on SSL to protect their loot would have to use encryption accelerators anyhow.
First of all, don't be an ass.
Second of all, if you're so outraged by this why didn't you fix it the second it happened? Oh, what's that? You didn't know? Seems to me this bit of code that everyone is screaming "noob mistake" about is seriously weird, probably not a good idea in the first place, and wasn't commented properly as to what it did.
Yes, this was a serious problem. Yes it's caused open source a black eye. Yes, something like this was bound to happen sometime--and probably will again.
Sounds to me like there's plenty of blame to go round. Fix the problem, fix the process, and move on. And don't spout when you're completely clueless.
I Like debian, I think it is fantastic, I thank all the developers working on it.
As per the problem, Linux boxes are being hacked by the thousands every day spreading malware to windows boxes every day anyway, and most of those are redhat/centos/suse/ubuntu/debian, no one seemed to find a good explanation. Could this be related?
I do remember things like blaster/code red happening to windows 2000, I do not remember MS paying my company for the damages those caused to about a 1000 desktop boxes.
No one asked to lock MS's coders on a box and throwing them to the Thames...
So guys chill out please.
This post has been deleted by its author
>They're very often done in order to make software fit Debian's "policies". The >problem is that the "policies" are arbitrary - they're Debian-only standards which >the original developer of the software may not be aware of, and there may be very >good reasons not to do things the way the "policy" says.
The developer of the software is aware that the license they distribute allows Debian, Redhat et al. to do these sorts of things with their code. If you don't like it, don't distribute your code with licenses that allow modification and redistribution.
DJB did exactly that.
>In fact as I mentioned on another one of these comment pages, the first time I >found a bug in a Debian package which didn't exist in the original source codes >(i.e. a bug which could be removed by uninstalling the Debian version of the >software and then reinstalling from the original source code) was back in 1996
I guess you never reported said bug to the package maintainer so that it could be fixed? When you "reinstalled from source", were you linking against locally built libraries or the shipped ones? There are numerous things that could have caused a bug on your system that had nothing to do with the package you experienced problems with. Like out mutual friend Chris Thomas you seem to have had one bad experience with Debian and in your eyes that seems to give you the right to bad mouth the thousands and thousands of hours of time people have volunteered into the project.
I don't even see why you brought Windows into the argument.
Ideally as a consequence of this there will be a major paradigm shift in the process of changing the code. It's less to do with creating more responsible patch policies, restricting who can make them etc. And more to do with encouraging developer responsibility.
Really should never be the case where a developer looks at a block of code, commented, borked, free or closed, says to himself "Hmmm I have no idea what effect changing this might have" and changes it. It is less to do with "human error" or "n00b mistakes" and an awful lot more to do with downright irresponsible recklessness. If he tried to find a peer review, and found no help forthcoming the obvious next step is to just not touch it. Unless he has at least a vague idea as to the purpose of the code.
Defend him however you want, but he showed a reckless disregard for good sense which would almost certainly have cost him his job in an organization. (As you said earlier, it takes two or three small mistakes - but this was pure arrogance)
As a non expert who gets trough life by GUI only I have come to understand that there is no such thing as computer security. Those who think that free software can not be used because somebody makes a mistake could still benefit from its efficiency and low cost by adding a proprietary encrypted tunnel to the SSL SHTI STD's etc. and perhaps another one inside that one that you roll yourself. Keep adding tunnels until you feel comfortable. If your business is storing atomic bomb detonation codes for the government I would recommend pen and paper and an old fashioned safe.
Since the worlds business increasingly relies on shifting secret information through a global net that is used by all it is very important that the encrypting tools are made up by free software. What would you rather have: Microsoft saying we have made these tools and your are going to have to trust us when we say your secrets are safe or Debian where you can see for yourself exactly what is going on.
The remedy is not to argue for the death of Debian but to join Debian and help work on computer "security". It takes more effort than calling for peoples heads but you would be adding your weight to masses of people who have pulled up a gem that is untainted by political and commercial interests.
Art R Big
> Is anyone keeping track of which SSL issuing authorities are providing instructions and free (zero-cost) certificate re-issues for affected certificates which need new CSRs?
I got a nice email from GlobalSign this morning:
"One of the advantages of being a GlobalSign customer and having the GlobalSign Certificate Center (Formally known as the Global Agent System) is the ability to re-issue a certificate in the event of private key loss, or in this case a possible weakness in the keys.
"If you do wish to re-issue your certificate, then please follow these simple instructions:-
"(1) Address the vulnerability by patching the affected operating system to the latest level.
"(2) Create a new CSR remembering to use exactly the same information that you used last time. (The information can be seen within your account area or within the certificate itself.)
"(3) Log into your GlobalSign Certificate Center account and click on the 'Certificate Application History' menu option and select the appropriate certificate that you wish to replace.
"(4) Click on the 'Re-Issue' button and enter your new CSR.
"(5) The new certificate, based on your new CSR and new private keys, will be issued and available for you to download in moments.
"Many thanks for choosing GlobalSign. We hope you see the benefit today and every day."
Surely in this day and age, when crims have massive botnets at their disposal and even smartcards and WEP are found to be flawed, you wouldn't base SSL or any other encryption on a random seed that wasn't KNOWN to be truly physiclly random? e.g. LSBs from a recording of the microphone jack, or even a dedicated RNG PCI card?
Seriously, it's down to common bloody sense and covering your arse to the limits of your technical ability these days. No excuses.
Flame cos I know the excuses will flow thick and fast.
Well of course you want the source of randomness to be random - and it is not that hard.
I hazily recall having to bash away at the keyboard; in some quasi finger dance to different melodies mixed by a DJ with no concept of the fade buttons.
And having to quaff a few pints (actual number of pints and music listened to not revealed, so as to reduce chance of replay attack), in the spirit of entropy creation.
Oh, those were the days.
I've written a Windows tool that allows scanning of remote web servers (or anything that answers a TCP connection with an SSL exchange) to determine whether they have the weak keys identified by Ubuntu.
Read about it, then download it, from my blog at http://msmvps.com/blogs/alunj/archive/2008/05/22/1626252.aspx
Written in C#, with source code provided.
Suburban Inmate is right - there are several known good sources of entropy.
Unfortunately, the change to Debian meant that even if you _had_ good sources of entropy, they were ignored.
The line that was changed isn't the line that asked ssleay_rand_add to add 'entropy' from an uninitialised buffer.
The line that was changes is the line _in_ ssleay_rand_add that added entropy from a program-supplied buffer.
So, every time any caller (including other parts of OpenSSL) asked to add new entropy, the function threw it away.
Saying "if it helps with debugging" is not the same as saying, "go ahead ship it out to all of your users and downstream it into a bunch of child-distros"
If something like that is intended to help debug a certain issue then that's fine, because the random number generator becomes a lot less random and therefore it makes it possibly to debug certain aspects of the code easier than before.
If you want repeatable tests you need to make random numbers predictable for the purposes of the tests, "debugging" this is not simply a golden ticket to ignore the issue. An inexperienced developer made a mistake, string 'em up!
** downward thumb because well, isn't that what the romans did? **
As for the person who said using an unitialised spot of memory somewhere fairly randomly placed in the system is a bad idea should think again, its probably one of the best sources of entropy on a computer system. If for instance hard disk interrupts, they can be manipulated, network interrupts can also be manipulated, also keyboard interrupts etc are all prone to injection.
However, I still agree we should all have some kind of frequency hopping galactic background radiation based hardware RNG system, or something equally as impossible to control.
AC wrote "My point is that this issue has blown a massive argument for OSS out of the window."
Sorry man, you've missed the point - this cock up with Debian does in no way invalidate OSS as a method. Like others, I'd suggest that if this was a closed-source project, then it's likely that damage control would be invoked and the fix would be slipped out quietly. At least team Debian have 'fessed up to their mistake, and issued a fix.
However, having disagreed with your generalisations, I will agree with you on a specific point - to whit, that this has clearly demonstrated (to me at least) that Debian's code control process is in dire need of review. As a software "professional" myself I find it alarming that a _critical_ piece of software like SSL/SSH is allowed to be committed with only _one_ peer reviewer!
I would also agree with others that, if Debian's packagers are making modifications to code that are not purely to do with different locations etc, then these need to be subject to more scrutiny than would otherwise be the case. And they definitely should be feeding these "improvements" (?!) back to the original development teams!
I'll continue to use Debian after this, (mainly because I find it a deal more stable than Ubuntu), but I hope that they learn from this gaffe, and put in a process "fix" as quickly as possible!
The real problem now, is has any debian openssl encrypted traffic been sniffed over the last couple of years.
We know, that BT and Phorm have been doing it, if a webserver was running debian patched openssl, and a customer gave their credit card details, then potentially those details could now be compromised.
And people who used an ssh encrypted connection with debian patched openssl key, to setup user accounts, may also find that those details are compromised.
This is why illegal wiretapping should remain illegal. The man in the middle attack does not have to be done in real time, they can siphon off the data and wait until a vulnerability like this occurs, or a time when the problem of the primes is solved.
Interception of data, should not be allowed, for reasons such like this. Phorm cannot hide behind the fact that data may be encrypted so they have no access, right now they probably have a load of encrypted data that is now vulnerable.
Most of the changes I've personally seen package maintainers apply to their packages were not made by the package maintainer; the package maintainer simply reviewed the code, verified it fixed the issue in question, and either applied it, or in the case of source code distros, properly marked it for being applied at a specific time relative to other patches for that package.
It's quite rare for a package maintainer to apply a patch to move files around, because most packages contain provisions to specify alternate locations for files via a configure script. Of course, this does not stop some people; I've seen Gentoo packages which patched the upstream package to do what could have been just as easily done by giving the proper command-line arguments to configure, and I've seen other Gentoo packages which patched the upstream package to move related files into unrelated directories - the configure script only allowed for selecting where each related set of files went. (The latter move, by the way, was very annoying for those of us who understood why the upstream package maintainer felt those files were related...)
"It's ok if it helps debugging" means that a certain change is suitable for a debugging version of the package; it doesn't mean it's suitable for everyone to use.
Note that the OpenSSL maintainers aren't responsible for the Debian developer deciding to noose all of his users. They advised - and their advise was not understood, because the Debian developer assumed that their mindset was the same as his, rather than seeking additional clarification.
Also, the problem has not had a huge impact on *everyone*, although the impact is certainly beyond Debian-based systems. At work, we ran all our certs through the blacklist tool, and found we had not been hit - except, of course, for the two Ubuntu systems themselves. The impact would've been much bigger if they had been considered the stable systems to work from, rather than being new and experimental.
As Germany, has now banned pen testing software, how would they have insured that the OpenSSL keys generated were good? And one assumes that if they were to use a tool to identify the problem today, they would be using an illegal tool in Germany.
Sure, someone could have done this by hands and eyeballs. But, any German writing software to check would have fallen foul of their laws.
And with the UK moving swiftly in the same misguided direction, if any UK citizen distributes software to aid in the identification of such flaws, they too will fall foul of the law soon.
I have just noticed Alun Jones above, is offering such a tool, now that tool may be useful, but of course it could be used for nefarious purpose as well. And of course a lot of the pen suites have already placed code in to detect servers that are using weak keys.
Most admins I would bet will be using such tools, and seeing the benefit, perhaps even checking prior to launch, but soon in the UK that will be illegal as the law stands.
I don't know how anyone can comment on open-source in a bad way. There is nothing negative that can come from open-source other than of course those greedy people cannot take money from the people who could use it in better ways, but hey that's another subject altogether.
I think we can should all take a big step back for a second. Unless the developer who made this apparent self-made mistake pushed his way into his current postion by nothing but complete lies then a lot of people here need to start questioning their own ability, especially within the development world.
How can one possibly leave such a large task to one man? I have no doubt on my mind that this person asked for help and that this person will even take the wrap for what has happended even after asking for support.
What we have hear is a lack of communication and nothing more, we might of seen development skills from both sides of the table but I know which side of the table needs blaming in my opinion and I'll give you a hint - it's not Debian.
Seriously, what the hell is going on these days.
This vulnerability was introduced in Sep 06, meaning it was in the patch for the RSA exponent 3 vulnerability. This vulnerability was a great example of how complicated cryptography is, and how many things have to be right to get a secure system. The default exponent 3 was used by most CA's for the ease of calculation it provided and this was considered safe until a mathematical vulnerability was found and an exploit published.
I think it needs to be highlighted that this defect was almost definately injected in a patch to fix a major security vulnerability, which should have highlighted the difficulty in getting these things right.
Any tinkering to a patch fixing a major security vulnerability is probably not a good thing unless you really know what you are doing.
Complexity is the enemy of security. Personally I like and use Debian for a cheap hosted virtual machine server and Kubuntu on my desktop. Ubuntu can't afford to do all the work done by Debian volunteers, though pays for a few of those working on Debian full time. The more complex the system the more likely there exist things about it we don't understand and would not like if we did. Sometimes a developer or distributor has to release or package a product without knowing as much about it as they would like. I've been there and done that as a programmer far too often to count, and almost any bug is a potential vulnerability in some circumstances. And those who imagine that achieving software quality would be helped by the introduction of blame culture into programming don't deserve the benefit of being able to use computers programmed by other people.
So what are the alternatives ? Not to use computers ? Put up with a proprietary OS which is 10 times as complex due to the commercial need to maintain binary compatibility with everything everyone has ever done on it in the past ? Hope that the fact that only criminals, spies and employees of the company that sells the proprietary OS have source access means that the employees and spies know enough consistently to defeat the crooks ?
The value added by distributions makes running free software OSs feasible for most users. Having consistency in how things are installed and removed and upgraded such that this process can be automated makes a massive difference if you want to use an OS which includes several thousand programs developed independently by tens of thousands of programmers - where you only have time to read the source code of less than a dozen of these programs or none at all.
Debian followed by Ubuntu are the only OSs I have used where I have consistently been able to upgrade through several major releases without ending up with a cruft-ridden or unstable system. Using Red Hat, Slackware or Mandriva I didn't have quite the same level of confidence in system performance and stability without reinstalling from scratch every 18 - 24 months - though maybe these other distributions have got better since I stopped using them.
So thanks to all Debian and Ubuntu developers out there, as well as all original free software programmers who work on making this possible. So long as you stay open and accountable, I don't expect you to be perfect.
What I would like to see come of this, is full explanations of patching.
OpenSource is a funny one, sure the code may be open source, but not many bother to publish the design docs. I have tested the waters on this myself, with an application where both the design and the source code is open for viewing, and I have noticed most people just go straight to the download. But, I personally have found it quite useful to refer to the design docs of my own applications, when extending or changing, so I have a space to rationalize about change.
With that said, myself personally, and I think quite a lot of other people, are interested in what a distro package maintainer does as far as patching is concerned. I would like to see a command that lists all the patches along with a succinct reason as to the patch.
See, opensource is great, but we are giving it lip service in many areas, and I personally would like to see some commercial benefit for developers of opensource code, ie. I think that monetary recompense to the developer for code, would actually create better opensource applications, and that's what we should be working on.
We need to get off the 'freetard' merry go round, keep the advantages of open source and put in place some form of monetary model, instead of just throwing our hands up in the air and saying we cannot weave monetary renumeration into this.
"See, opensource is great, but we are giving it lip service in many areas, and I personally would like to see some commercial benefit for developers of opensource code, ie. I think that monetary recompense to the developer for code, would actually create better opensource applications, and that's what we should be working on."
The overwhelming majority of work on open source software (OSS) is paid work, as demonstrated by extensive automated analysis of contributions to Linux based on source code line-count and copyright signoff, see
http://lwn.net/Articles/222773/
The motivations for creating software are various. In my case this research supports teaching I am paid for. Some software creation is motivated by the sale of packages which might be harmed by opensourcing them, but package sales are a minority interest amongst programmers.
When looking at the benefits of OSS to its creators the greatest is the reduction in transaction costs in a world where a useful system has to be integrated from contributions by hundreds of thousands of programmers working for tens of thousands of independent organisations, for which sale of the software could never be our main line of business because the transaction costs would be greater than the sales value.
Creators of this code do want your lip service to support our viral method of distribution, but we value your custom when you buy one of our courses, consultancy services, software publications or hardware in which OSS is embedded or which is sold on the back of OSS even more. If you can help identify and fix a bug or contribute a useful patch that works for you and which benefits you by having it included in the upstream version even better.
I do think someone above, should perhaps revisit their maths' books.
Take a dice roll, roll that dice once and we would say that the dice had a 1 in 3 chance of being a a 3 or 4.
But roll that dice a million times, and you will find that the average of the rolls tends to 3.5.
Now, every roll of the dice is random and you cannot predict what will be shown, but to test if something is random, you can run a basic test like this, and on average you will also see that the numbers selected are all fairly evenly spread, as long as the sample set is large.
If the numbers are not evenly spread, then your random system is not that random.
Sure there is a chance that a random system will return always the same number, but I personally would not use such a system, and in fact a sequence of +1 would be more secure than a random system that returned the same number each time.
And using uninitialized memory is not secure from being gamed either.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
