nothing is sacred these days....
and here i was in the belief that linux is so secure ... now they have rootkits too...
guess it's time to pull the network plug and go back to my osbourne-i using cp.m ...
Attacks in the wild are under way against Linux systems with compromised SSH keys, the US Computer Emergency Readiness Team is warning. The attacks appear to use stolen SSH keys to take hold of a targeted machine and then gain root access by exploiting weaknesses in the kernel. The attacks then install a rootkit known as …
This is not some BS from CERT.
Phalanx used to harvest SSL/SSH keys is out there, it is pretty sophisticated, it appears to archive the keys for collection by the operator of the rootkit, and it is not exactly rare. We've found several infestations of of this rootkit in the past few weeks where I work - we have a lot of Linux boxes.
If you run Linux machines then it would be a very good idea to install rkhunter (http://www.rootkit.nl/) and chkrootkit (http://www.chkrootkit.org/) and run them periodically. These are detection tools not cleanup tools.
The cleanup is, depending on your paranoia level, (a) boot off an install DVD to repair mode and delete the rootkit, or (b) format your disks and reinstall Linux.
If your keys have been lifted then at the least you must regenerate your system keys and user keys. Look out especially for users that have set up host-based authentication (ie ~/.ssh/authorized_keys) between machines they use.
This is potentially a very bad one, people.
"I'm still absolutely adamant this is a problem system administrators should have handled a long time ago,"
I suppose the problem in question would be keeping the weak SSH keys around, right?
Of course, the problem instead might be people from userland trying to ring up the BOFH or the Boss giving out clue-depleted orders etc., all of which can be handled.
(And why "still absolutely adamant"? Might there be reasons to make him less adamant or un-adamant later on, given a re-assessment of the situation?)
"The cleanup is, depending on your paranoia level, (a) boot off an install DVD to repair mode and delete the rootkit, or (b) format your disks and reinstall Linux."
This sounds like a whole heap of overtime watching screens of scrolling sanity checks.
Motto suggestion: El Reg, keeping readers busy whether they wanted to be or not.
"The CERT advisory makes no mention of the flaw in the Debian random number generator,"
Because it's been fixed,... and the debian ssh packages depend on blacklists that block weak keys from being used for ingoing and outgoing connections, ship with tools to find weak keys etc blah blah blah. You realise a good starting point for finding keys would be to bruteforce boxes than allow password authentication don't you?
It's an issue that everyone should have fixed months ago (everyone running effected boxes that is). It was advised on the Debian security mailing list and a few others. Plus a simple apt-get update should have raised it to attention too.
For other distros, the same rules apply, different mailing lists and update commands.
"The cleanup is, depending on your paranoia level, (a) boot off an install DVD to repair mode and delete the rootkit, or (b) format your disks and reinstall Linux." ... By E Posted Wednesday 27th August 2008 01:43 GMT
Sounds like a scorched earth retreat cleanup but with no viable policy shared for future protection, E.
"The cleanup is, depending on your paranoia level, (a) boot off an install DVD to repair mode and delete the rootkit, or (b) format your disks and reinstall Linux."
Not much different from what is required for a Windows rootkit then, although option (b) is far more preferable for 'doze as it's not easy cleaning a system from a boot disc due to problems with registry dependencies / hooks etc.
The main problem I have with this is that it still looks like an amateur programmed it. So they've managed kernel-level access, brute-forcing of low-entropy keys, they collect keys and spread between trusted machines - fantastic. But nobody could be bothered to make a primitive attempt to hide it's presence in the filesystem, thus making it trivially detectable (especially if you have anything that monitors filesystems) and (less important) trivially removeable.
Why does it need more than a prescence in something like /tmp in order to initially infect and then it's got kernel-level access, so why does it show up at all? There is all sorts of code available in projects both legitimate and not that will let you hide data inside files that are not visible, will let you pre-pend code to kernel images, will let you modify bootloaders, etc. - if it weren't visible in the "real" filesystem, it could quite easily avoid manual and automatic detection by rootkit hunters etc. and be much more prevelant and dangerous.
Again, viruses built by pluggable modules that just show that the actual author of it (should they be caught) aren't the problem - someone, somewhere wrote a set of modules to do the difficult stuff and *they* are the problem, here, not some 12-year-old that joined a few of them together in "Build-a-virus '98".
Having said that, it was only a matter of time before someone did this properly and the Debian flaw is the perfect opportunity to kick-start such an attack.
This is why I build any SSH keys on a totally seperate machine to the one they are going to be used on (physically and technologically), give them long passphrases and then keep them in as few places as absolutely necessary. If I'm the only one going to use them, then it's onto a USB key with a backup on the machine that created the key should the worse happen. Lose the USB key - you can't get in (except if it's a real emergency, when you can do it from a manually-created copy from the key-generating machine), but you absolutely **know** that someone else may have them and that you need to regenerate and change the keys immediately.
Oh, and install something that will absolutely go mad if someone gets in and does anything - you want to log every login, even if they immediately try to delete the logs, you want to log every significant file change, even if they try to immediately fix the checksums, disable the monitor program etc.
It's interesting, yes, but it probably won't hit critical mass. If it does, it'll only be among home and hobby machines and possibly the odd cheap hosting outfit. The problem is that we now have a warning about what its successor may do - and that could be a lot more nasty...
Once you can get access to a system, the whole security requirements changes.
There is no system in existence that will prevent apparantly authorised users from doing some damage on a system, but the degree of damage is what is important. Where Linux benefits is from a strong divide between normal and privileged access. Sure, if your private key AND IT'S PASSPHRASE are compromised, then someone can get access as you to your system. But this is just your non-privileged account, isn't it.
Of course, if lazy admin's directly access root using SSH, or have passphraseless SSH keys, or have sudo rules that allow them to cross the security divide without further confirmation, or store both private and public keys on their boundry systems, or use the same private key throughout their whole environment, then these fools deserve to get their systems compromised.
So here are the rules.
- Use a non-privileged account for initial access any system
- Su or sudo to obtain root access, but use additional authentication steps
- Don't use passphraseless SSH keys, unless you tie down what can be run (see SSH documentation).
- If possible, use hardware based authentication to secure private keys
- Guard your private keys like your life depens on them
- Don't store private keys on systems that do not need them
- Make sure that the permissions on your .ssh directory only allow your ID to see the private keys (I know ssh does some checks, but 0600 is best on files, and 0700 is best on directories)
- Use different keys in different parts of your organisation
- Consider using passwords with SSH (with strong change and strength rules) rather than SSH keys for very critical systems (really)
- Be careful about storing your private keys on shared Windows systems, or systems that have remote users with administrator access (consider portaputty and store the keys on an encrypted USB key).
- If you are really paranoid, regularly change your private and public keys on all your access boxes (please note it is NOT enough just to change the passphrase!)
If you follow these, then even if one of your private keys is stolen, then the amount of damage can be limited. As always, you can run a potentially secure system in a non-secure way. Security is only as strong as the weakest link, and this is often the sysadmins!
Oh, and by the way, if you want to see a file that cannot be seen using a hacked ls, try "echo *", or "find /etc -print". Or maybe use filename-completion in the shell. This is UNIX (or close to it) so there is nearly always more than one way to do something
@E
i checked .. at least on latest available versions for Gentoo rkhunter chkrootkit at 5 am EST do not detect that particular rootkit .. at least it aint in the list that they check for.
bit amazing but that's what the program spits out in it's out on terminal.
bunch of rootkits .. but no phalanx.
still the solution is the same : dont need ssh on the public interface running ? shut it down.
btw .. that's an issue known for months and a non issue on a lot of
distros .also .. that aint the first rootkit .. that aint the last...
a lot of noise for much of nothing. :)
FuzzyTheBear
FuzzyTheBear ..
I do occasionally noticed my network monitor blipping away because of attacks but I've noticed them that little bit more in the past few days. My OpenSSL library is up to date and I'm not running anything Debian-based so I'm probably okay but I shut down the SSH server just to get rid of them because it's like having the big bad wolf continually knocking on your door.
"There's a viral aspect to this attack. As new SSH keys are stolen, new machines are potentially vulnerable to attack."
Bunk! That's the mark of a worm, not a virus. The biological behavioral analog of this attack is a bacterial infection, which is what defines a worm, not a virus.
Kindly remember that a computer virus is a malware code fragment that emulates a biological virus by embedding itself in an executable to activate it's payload and/or replicate itself, often using some of that executable's code as a means of getting control of machine functions.
The attack here is clearly a worm, because it spreads itself using the classic bacterial model - break in by compromising a well known authentication mechanism, copy in a tool kit, and consume the host system's resources primarily to replicate itself and install backdoors for later use.
Another clear sign that this is a worm, is that the rootkit uses hidden directories to hide in plain sight, rather than burrowing into the bodies of executable files, which is a much more difficult way to hide. File size & checksum scanners have been around for years, and will find that sort of thing if not properly done. Also increasing the difficulty level, most OS binaries on *nix are writable by root, so you have to compromise root before you can hide there.
I don't believe I'm explaining this stuff to Reg writers :-(
/gds
@Adam Williamson
Did so on the Debian boxes, all machines are updated regularly. We found the rootkit on a few up to date Redhat 5 boxes. I do not think the two are related.
@Richard Hebert
You probably want to download rkhunter and chkrootkit from the noted sites. There is no guarantee that any given distro is up to date or has not modified the package. I used the two programs built from source downloaded from the noted web sites. I did detect Phalanx successfully.
@Colin Wilson
Yes and no. Linux doesn't suffer from a registry, but there's no reason why a rootkit cannot be used to patch standard binaries. So maybe one uses tripwire or some such and can detect this. Problem is that no Linux distro that I've ever seen installs and runs tripwire by default.
@Peter Gathercole
Good rules and I in fact use them all. Linux being multi-user it's hard to enforce them across the board. Couple that with known exploits and something like a rootkit is going to succeed occaionally anyway.
"Lazy Admins"
1) I don't know that this is entirely fair, or at least not all the time. The RH5 boxes I found the rootkit on were fully patched, their admin had yum running update as a daily cron job. There was no "unpatched openssl or weak Debian keys" and the kernel was also current.
2) So, what should we do: regularly delete all the users' keys, force them to recreate the keys, and change the system's keys? Basically that would break so much functionality that they might as well use Windows. IMHO, that's not a vigilant admin that's a fascist admin.
3) Accounts will get compromised whether they are on a Windows, Mac or Linux box, offering an entry for the back hats. The means of subverting an account are numerous and not all involve software weaknesses - you'll have heard of social engineering and the ubiquitous sticky note?
Maybe this RK was written by someone who is a lot smarter than some here suggest. Maybe, just maybe they wrote and released this version of the RK as a precursor to the real RK that does everything just right so that lazy admins and clueless users have one last chance to get it right. Or then again, maybe, just maybe it was released by Bill Gates and his crew to finally show all of the Linux zealots that they are not immune to the RK BS. Maybe, just maybe.................................................................................................
I was called to look into this because the fellow managing those machines was out of town - I consider it a very serious security compromise, and took care of cleaning it up even though they are not 'my' machines.
Believe me, sir, I did not consider it a regular part of the job in either sense!
If my tone in my original post was less than grave, I suppose it is because the event happened a week ago, it's been dealt with.
One thing that needs re-enforcing is that unless you have a security hole that allows a non-privileged user across the security divide, it is just NOT POSSIBLE to install a rootkit on a properly run Linux system. Rootkits, by their very nature, need to alter/add code to the kernel, libraries or modules that are used to run the system. And this needs root permissions. This is why it is very important to make sure that you do AS LITTLE AS POSSIBLE as root on a UNIX-like OS.
There are a number of well known ways to try to subvert a user currently logged on as root, but a reasonably savy sysadmin should be able to avoid these (you know, don't browse the internet as root, check that your path does not allow commands from the current directory, make sure that there are no executable script files with world-write, don't read mail as root, keep firm control of the permissions on directories in root's path, all the simple stuff).
If a rootkit cannot be installed, it cannot compromise your system, nor can it get access to SSH keys other than the user it is running as.
Please note that I am not saying Linux is totally secure, there have been, and will be in the future, code and design defects which could allow a system to be compromised. I firmly believe, however, that the open source model allows such things to be identified and fixed much more rapidly than a closed source model. Couple this with an effective notification and patch delivery system, and Linux just is more secure.
Contrast this to Windows, where many people by default use an account with Admin. privileges, or with the security notifications turned off? Asking for trouble as far as I can tell.
But the amazing thing is that the UNIX/Linux security and source model is decades old (I've been using UNIX for 30 years, and the Bell Labs. UNIX V6 and V7 code, and all the BSD code used to be available for inspection and modification to the academic community and others for at least that long). And Microsoft (who, in fact, have been UNIX licensees for at least 24 years - they did the original Xenix port to various architectures, before spinning off the original SCO) just don't seem to be able to learn.
I am sick of this crap. There is a real security problem here that is potentially very very large. And most of what I have read is finger pointing or people suggesting that either "now Linux is as bad as Windows", or it is the fault of bad, lazy people.
Fact of the matter is that this rootkit *is* international, it attacks one of the basic things we've all trusted for the past decade - SSL/SSH - and we all may as well face up to the fact.
So, I'll point a finger at the fules who claim to know the score and who tell us about who's to blame.
I know a *lot* of computer sci people, I know a *lot* of sysadmins, and I know about two or three people who really understand public key crypto when I ask them hard questions. For all you armchair security experts: public key crypto == underpinnings of ssh/ssl.
And, well, if you don't know basic XML syntax, you can go back to school, and good riddance to you.
re: Peter Gathercole, but in re-reading this screed before posting, in fairness only partly:
<rant and="fuck off and die if you don't like it" also="there are some fair points below">
Yes, Windows user as admin is a common thing and a problem. We know that XP makes the first user - if created as part of the install process - an administrator and that is a problem insofar as that user typically is, well, just a user. We also know there is a massive ecosystem (gahhh) devoted to keeping exactly that Windows user safe including liberal use of system modal dialog boxes and idiot-light style pop ups from the system tray. You've gotta be pretty stupid not to notice when your Windows A/V software thinks there is a problem.
But! on Linux there is no such infrastructure. Rootkit hunter apps are not even installed by default AFAICT, and I've used a lot of distros: RH, Fedora, SuSE, Debian, Slackware, Yellowdog.
So - fine. I'm an admin and every box I've built in the past two years was built from a recipe and the recipe included rkhunter and chkrootkit running as cron jobs. I know this because I wrote the recipe. Might I f*ck up and miss a step sometimes? Perhaps, but I go back and check.
You know, I'm careful but I am not perfect and neither is anyone else who raises the "lazy sysadmin" flag. And neither is anyone at all.
Way back in the day, before a lot of you people had outgrown your milk teeth, before kernel modules, it was known and commented on that the monolithic kernel was safe from device driver (read: kernel module) attacks. Don't bother replying with remarks about /dev/kmem: I have not said the monolithic kernel was safe from that. Modules happened anyway, and they had to because else we need distro kernels with support for every possible device - how would you like a 100MB kernel (+/-)? (or, by now in 2008, hundreds of install kernels with limited device support - Slack had maybe two dozen kernels at one point circa the early 90's before modules - I *knew* the hardware in my boxes and still I sometimes picked the wrong install kernel). But I suppose that just makes me a lazy sysadmin, eh?
So, here is a suggestion, and it is not meant as a finger of blame by any means. How about the makers of distros just add things like rkhunter and chkrootkit as standard software, installed by default and updated by default and run periodically via cron, with notifications sent to the "main user".
I'll leave defining the main user to the lazy sysadmin finger pointers - they ought to be able to figure that out, having already defined who's to blame elsewhere.
I do not think that the bulk of Linux users will be any better than the bulk of Windows users at actually ameliorating a detected rootkit.
Windows A/V software has however raised awareness to the point that I, as a sysadmin, do get asked by Windows users if such-and-such a Windows notice or dialog is a problem. I have never been asked by any Linux user about their suspected virus/rootkit/trojan/substitute-your-fave-noun.
People who say it's all up to the admin are either living in a world where the admin *is* actually God or they are dishonest. In academia anybody who can get grant money can build her/his own Linux box, and many do so, and most of them do not ask for support or inform anyone who might have a clue. Outside academia anybody who buys a computer can install Linux and he/she does so if so inclined. And, you know, WTF, why not, it's a free country, right?
How many of you arm chair security experts have, or have considered purchasing, a "Got Root?" t-shirt? How many of you have thought about exactly how very easy "got root" has become? Don't even bother replying - I'll just tell you to download an Ubuntu install image - remember that problem where the first XP user is an administrator? Grunt, grunt, uhhuhuuh, grunt, drool. *Don't* waste my time.
Is it just possible that, at some 5% or 10% market penetration outside the corporate controlled datacentre, Linux is a big enough and so very useful target to the black hats? Or that insofar as it is a multiuser-from-the-network-OS, very easy to install but not so easy to understand, it is far more attractive for black hats than something like Windows? Wake up, a**holes: Linux has grown up. It's time for the users and distros to do so too.
</rant>