Linux webserver botnet pushes malware A security researcher has discovered a cluster of infected Linux servers that have been corralled into a special ops botnet of sorts and used to distribute malware to unwitting people browsing the web. Each of the infected machines examined so far is a dedicated or virtual dedicated server running a legitimate website, Denis …
Servers need to have their security patches applied !
Most the time when something happens like this it's some lousy admin
too lazy to install the patches for the servers.
Patch your boxes , run regular updates and pay attention to GLSA's or
their equivalents for other distros.Stay out of trouble and have a pint.
Brute force password attacks are possibly the biggest issue facing Linux (any Unix-like) machines. My own servers show many many attempts at password guessing by bots (the usernames they attempt are mostly generic account names or common people's names).
FTP servers are a typical method for this brute force attack, but any password authentication system could be used.
Passwords are a poor way of securing servers, but they're so convenient!
Why? it is much easier to secure a Linux box with Apache than Windows with IIS.
There are server distributions which are hardened and there's plenty of advice on the net on hardening installations. A hardened install is not substitute for hardware protection, this should be the first line of defence.
With Windows you're stuck with the default state Microsoft ship Windows with.
If you read the article, it appears that the Apache webserver isn't the problem, it's a second webserver that's been planted on the affected machines. The Apache webserver is serving up the legitimate traffic, while the malware serves up traffic on port 8080. Not clear whether the servers were unpatched and exploited this way, or if the root pwds were gleaned some other way.
"I am soo glad that I haven't taken the plunge into running Apache on a Linux box.
I assume this is a Linux ftp privilege that was abused."
No, it was "careless administrators who allowed their root passwords to be sniffed". What do *you* run Apache on? MacOS?
Also, "Servers need to have their security patches applied !" - applying patches won't keep out someone with the root password. Sure, when your system is vulnerable, you should patch it, but patching for its own sake does more harm than good.
I guess I'm missing the bit abouit what was done with the root password.
Any non-root user on anything other than a server locked down with high levels of paranioa can start a server process. As long as it listens on some port # > 1024, typical Unix (not just Linux) system configurations will allow this. All that is required is that someone gets hold of any user account, logs in, sets up a second server instance and starts it on a high numbered port. One doesn't need to lose the system root password or hack the Apache installation.
Its possible to block all ports from being opened by non-root users on a system. But this can cripple authorized applications. A better approach is for a sys admin to run a security scan from time to time, looking for open ports, identify the attached applications and throw users off the system for running unauthorized services.
What kind of an idiot uses FTP? They send their root password over the public internet in plain text!
Anyone with an ounce of sense does everything over ssh. I find that rsync over ssh actually transfers files (at least if you have more than a few to transfer) much faster than ftp does.
Some of these idiots are going to be running their own sites. Hosts should just shut down the sites, as they have done. If any of them are being paid to run a site they should be fired.
Systems should only serve on ports appropriate for what they serve, eg. port 80 for a http webserver.
The fact that there is traffic to port 8080 and it hasn't been spotted, let alone the fact that the firewall isn't blocking this gives some indication of the problem.
Even if the server instance admin is not competent, why has the virtual hosting company not spotted the traffic anomoly ? Surely the users of a virtual host should have registered their intended services so the common host can be locked down properly ?
If the root password of a server has been compromised, then anything can be done. Moral ? Keep it secure ! If you have a separate firewall, the password to that should be different, or local access only.
I suspect poor practice in multiple areas here. FTP/telnet or poor root password by the admin, and poor hosting company security policy to start with.
It is a shame that the use of the compromised servers is not identified in the article, home, or business ? The number of servers identified is relatively low (100 ?) so perhaps this suggests home users running their own servers, and/or one or two virtual hosting companies who need to look at their security policies.
Every IT admin I've had the pleasure to work with has known what they were doing, so I find it hard to imagine that this is the result of professional mistakes. If LAMP was the source of the problem, much higher numbers of servers would be compromised.
I doubt FTP passwords got from sniffing - probably from the following script which if run on systems with plesk then you get all the FTP passwords in clear text for the shared accounts,
mysql -u admin -p`cat /etc/psa/.psa.shadow` psa -e "select s.login,s.home,a.password from sys_users s,accounts a where a.id=s.account_id"
So all it needs is someone to be able to read /etc/psa/.psa.shadow and there goes a whole shared server of vhosted accounts because once the passwords have been got then all the .php and .html files on all shared accounts will get appropriate iframes added by a bot net that usually has 1 file edited per bot using a process of a) ftp get b) edit file c) ftp put. This makes a joke of the webhoster strategy of limiting FTP connections per IP - 50 different IPs hit the site and all edit 1 file each !.
How does someone get to read /etc/psa/.psa.shadow ? Well that is obviously root and psaadm only that can read that but my feeling is that root doesn't get exploited but something that runs like as root or psaadm on the server. So no need to brute force FTP passwords nor reset them (which alerts the owner that something is wrong) - once in through some broken application that has escalated then plesk gives you the cleartext passwods for the rest of the FTP accounts.
judging by the myriad attempts to connect to accounts on servers that i look after, i would think that weak passwords have a part to play here.
Use fail2ban to block connections after a certain number of failed attempts. Most of the ssh attempts come from china and i don't expect much assistance from the administrator of their networks, so they will continue to bash away on other people's ssh ports
There is nothing in this article or the linked blog to indicate that the linux web servers are actively communicating with anything or modifying any payload by themselves. There might be some central control somewhere else uploading content to the webservers to be served up to the botnet nodes running on the compromised windows machines, but the linux webservers are doing nothing at all besides serving up data, as webservers do. This is not news and does not indicate anything new in the world of botnets or linux exploits. Webservers have been broken into for years by brute force attacks or stolen passwords. This is not an exploit. Only shoddy administration. The mention of nginx is also nothing to be alarmed about. It is just another webserver as opposed to apache running on a different port.
Nothing to see here. Not news.
Everyone who has commented about the root account being hacked, or the root password being guessed, should go back to UNIX school and learn a few things.
For ports > 1024 any user can open a listening socket. Unless you do something to stop it.
If you didn't know that you shouldn't be offering any opinions or advice or comments about how things are hacked, because you really don't know anything.
For the updatedb monkey, that it not guaranteed to find anything. Check the prune path, because something might be installed there.
A lot of Linux people who think they are gods just because they run Linux make me annoyed. They comment on things which they demonstrably know very little about.
The malware being put on linux servers (which are usually 'hacked' by poor password / vulnerability in apache/php / bad script, etc) still only targets Windows users.
i.e : Browsing to a malware infected page on a Linux machines does not effect you.
whats with the Linux and GPL software hysteria slant on this?
I guess the website admin have been lazy and havnt done basic server admin such as updating their version of Apache - or some other component of the server...they then got 0wned because of that..
and 'nginx' ? well of course any good cracker would use that...its very small, serves pages faster than Apache has a stupidly small memory footprint. if they didnt use that, they'd use Hiawatha instead - only a fool runs Apache if you dont need too!
so...the real angle here? Linux servers are just as vulnerable if that are not maintained. GPL web servers can serve malware just as good as IIS if they are allowed to. ?
"Everyone who has commented about the root account being hacked, or the root password being guessed, should go back to UNIX school and learn a few things. For ports > 1024 any user can open a listening socket. Unless you do something to stop it. If you didn't know that you shouldn't be offering any opinions or advice or comments about how things are hacked, because you really don't know anything."
How do you suggest the bad guys *installed* a bloody new webserver on the boxen without root privilege, Smarty McSmartypants?
"How do you suggest the bad guys *installed* a bloody new webserver on the boxen without root privilege, Smarty McSmartypants?"
By making the bloody new webserver listen on port 8080 as it says in the article, Mr RTFA
Although Mr Michael Fremlins wants to take a chill pill really...
@ElReg!comments!Pierre:
"""How do you suggest the bad guys *installed* a bloody new webserver on the boxen without root privilege, Smarty McSmartypants?"""
Very well? You can install whatever the hell you'd like in your home directory, or any other that you've got write privs on. Then you can run it.
What you can't do as non-root is modify a firewall. People really need to take host firewalls seriously and not accept incoming traffic to ports that aren't expected to be open.
"""No, it was "careless administrators who allowed their root passwords to be sniffed". What do *you* run Apache on? MacOS?"""
OpenBSD FTW. Nah just kidding.
Re: how to detect?
Assuming you haven't been /seriously/ compromised with a rootkit or patched binaries, ps or netstat should show you odd things. Even lsof will show listening tcp ports open.
Of course something along the lines of Tripwire, checking for file changes to configs, init/rc files, etc wouldn't have hurt either.
These are all, of course, a little above and beyond for the average unconcerned admin. As are most of the brute force blocking methods out there (None of which really work so great against a botnet in any case, since the safe ones typically block the attack source IP for a while, which dosn't work against thousands of coordinated attackers.)
@ElReg!comments!Pierre
Like that - find remote how in CGI script (Wordpress, Joomla! - whatever. Check such projects bugtrackers. Some people aren't making updates at all...), run system()-like function to invoke shell, get server app (you name it) and just run it! Doing it at high port (as 8080) doesn't require root rights Smarty*Smarty^McSmartypants
Anyway article is lame - author should consult with experts before posting it: modifying some poor guy site's content, and using his credentials to run another app, isn't making zombie from the box! One exception - it was root. I was working in webhosting company - most of such cases was about weak password ("123test"?), worm on users local machine, or some LAN sniffer.
"How do you suggest the bad guys *installed* a bloody new webserver on the boxen without root privilege, Smarty McSmartypants?"
He's suggesting it runs with user priveleges only. The compromised servers are running nginx on port 8080, so satisfy the port condition. The installation would presumably be in the user home directory only, which would normally be where the user's web page directory structure would be.
Is this the correct terminology to describe this situation? Infection tends to imply a virus, rather than a cracked server. The bots may have been compromised using a virus but this does not seem reasonable for the servers, from the story context, no matter what OS. If it *was* a virus infecting Linux servers to achieve this, this story would be very newsworthy indeed.
@Giles Jones: You state "With Windows you're stuck with the default state Microsoft ship Windows with."
If that were true then you wouldn't be able to run a web server at all on Windows as IIS is not installed by default. You have to explicitly install it.
Your ignorance is astounding and it is that sort of attitude that creates many of the security issues in the first place. You simply have a set of blinkered assumptions about how systems work.
Just so as you know: it is perfectly possible to simply not install IIS and instead put Apache on Windows and run that.
"it is much easier to secure a Linux box with Apache than Windows with IIS"
er 'scuse me, if you are already a Linux guru !
Who did you practice on ?
and @Richard Herbert.
"and pay attention to GLSA's or their equivalents for other distros"
See what I mean !
If you want to, it is possible to harden Windows by turning off stuff like ftp and all the other applications that Microsoft has switched on with the default install. I know this is a pain, but as far as I have seen so far, any self respecting Linux admin has to do the same. Otherwise you have exactly the same situation. Ubuntu comes with so many apps already installed and active, you need to be a fully paid up Linux guru badge wearer already,
or, you are accused of being
"some lousy admin too lazy to install the patches for the servers"
Thanks Richard, this is just the sort of helpfullness one encounters on the Linux forums.
My question was not an uneducated one.
There is often a semantics problem when talking about "servers".
That is, are we talking about the hardware box that supports the OS and the web server, or are we talking the OS (LAMP) server versions of Linux, or indeed are they talking about the actual web server application ?
That is, Hardware Layer, OS Layer, or Application Layer ?
Please.
Otherwise any would be web host just might as well not bother if they don't have penguins blood running through their brains, er sorry veins.
ALF
The sites in question are Chinese-hosted sites aimed at online gambling, porn and World of Warcraft gold-selling.
The choice of nginx is interesting, too. There is a relative scarcity of documentation for this server in anything than the creator, Igor Sysoev's, native Russian. It is widely used by Wordpress and Rambler, but outside of the Russian-speaking world, it is still quite unknown - desspite being an excellent piece oif software.
So, we're probably looking at an organised gang of Russian mafia, targeting poorly secured web servers, offering semi-legitimate content, from China. My guess is that the servers weren't even compromised, in the conventional sense of the word. I'd say there was a good chance they were simply sold. There's a global recession on: you can buy anything, these days, if you find the right market.
Sounds like there's a market for a distro where you can *only* run locally-compiled software -- anything that was not compiled on that machine falls over.
All you need to do then is have your development toolchain on a USB external drive, which is kept unplugged while it's not in use. Then, anything running on your box either was compiled by you, or is running through an interpreter (so at least you must have the Source Code, and therefore a fighting chance of understanding what it's doing).
"With Windows you're stuck with the default state Microsoft ship Windows with."
Maybe what he was getting at here was how closed the Windoze OS is as compared to *nix where you only run what you need and don't have to worry about Windoze having open channels to phone home and tattle on you.
This sounds suspiciously similar to an exploit I and my colleagues have been tracking for the better part of 4 years now.
A spam affiliate group previously known as "Bulker.biz" have been using this exact type of exploit to acquire hosting on hijacked web servers. I documented this in 2006:
http://spamtrackers.eu/wiki/index.php/My_Canadian_Pharmacy
The servers *always* have very easy / lazy root passwords, and it's usually one out of a list of only 15 easy passwords. All attempts to report these hijacks fall on deaf ears because these servers are mostly abandoned.
In *most* cases, the main function of the unix servers which are hijacked isn't even to serve out websites at all.
Whoever is doing this, they don't care what damage they do to these servers. They delete a large number of fundamental unix commands from the servers which mean that the root user can't do basic things like restart the server or change the password.
I've been attempting for years to gain the attention of any kind of law enforcement regarding these rampant hijacks, and not one of them has taken this issue seriously.
Someone should seriously follow the money from Bulker.biz (which have since changed their name again, to what is unknown.)
There's far more than malware going on here. This is completely illegal activity, and it's happening on probably hundreds of these servers every single day.
SiL / IKS / concerned citizen
We had a very similar problem on one of our servers a few weeks ago. One of our clients' websites had code like this inserted into every index page. We discovered that their web designer's laptop had got infected with malware, and he then did a website update. They captured the FTP password from the keyboard. No need for any access to privileged account, would have made no difference if the server had been Windows. Nice thing about Linux is being able to grep through every web page on the server for hidden iframes.
The one thing that I was most aware of when I was setting up dedicated servers was... don't let anyone else in.
Personally, I don't use FTP, all ports are blocked except 80 and a non-typical port for ssh. I use a ssh fingerprint (whatever it's called, I no "Guru") that means that only my computers are allowed in on ssh. Also I took time to limit/disable anything that I didn't need and chose a good strong password for when I want to su to root.
For me, that's all rather obvious. there's basically only me going onto the systems and *no-one* can get on with a simple username/password or chose their own stupid passwords.
Sadly, the world is not all that simple and as Michael Fremlins rightly points out, users can and will choose stupid passwords letting the bad guys into logins (ie. ordinary users) where they can run stuff like PHP, Ruby etc which are perfectly capable of running a web server.
How is this stopped on large multi-user systems? I'm not sure, but I suspect that it is possible... then again, as a "one man band" non-guru that's why I only run a small amount of my own stuff.
Moral. Read, learn, take advice and maintain your web server and know your limits, oh, and write down your good strong passwords :-)
No need to chill old bean ! Good point made.
I neglected to make exactly the same point wrt unpriveleged ports and user applications being able to bind to them, so enabling them to serve data. The thought later occurred to me that maybe the compromised servers haven't actually been rooted, and it is just an issue of insecure user passwords or sniffing a user FTP session. Perhaps more plausible ?
As I posted before I am puzzled by why the traffic anomolies were not identified by the hosting companies, etc.
Paris cos she would have made a more sensible post than I originally did :-) !
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
