Hmmmm..
http://forum.lxlabs.com/index.php?t=msg&th=12365&start=0&
Seems like this has been known about for a bit but they don't seem to be doing anything about it!
A large internet service provider said data for as many as 100,000 websites was destroyed by attackers who targeted a zero-day vulnerability in a widely-used virtualization application. Technicians at UK-based Vaserv.com were still scrambling to recover data on Monday evening UK time, more than 24 hours after unknown hackers …
That's not nice.
Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there.
I run a vps (cpanel) for some smaller clients & testing think I'm going to check my backups (2nd raid disk) and pull a set down to my local servers "just in case"
There really are some evil people with no fathers out there who probably also have carnal relationships on the maternal side
I have (had!) a backup server hosted by these folks... I used to have some more important stuff there, but pulled it out a few months ago because HyperVM was making me nervous. They pulled the entire control panel down several times since recently due to suspected vulnerabilities in the software.
Basically, HyperVM looks like a house of cards so I think it was only a matter of time before it got hacked. The control panel appears to run as root on each VPS host, of course any outward-facing thing can get hacked but there ought to have been some level of abstraction between the control panel and the VPSes to slow down the hackers. Doesn't seem like there was though.
Pretty glad I moved my stuff when I did.
A server which was hosting a very large campaign website was one of the servers hosted and attacked. I am still waiting on VASERV to issue an update regarding my node... although so far they have dealt with the matter well.
What has happened is a serious criminal act and those involved should be brought to justice. I am surprised this has not been mentioned in the mainstream news.
This is a major wake up call.
I have an unmanaged VPS that has disappeared. I feel sorry for VAServe, other than choosing poor management software, they couldnt have predicted this! The VAServe status page is a bit sketchy, and i dont know which physical server my node was hosted on, so i have no idea if all my data has been lost, or when it might be restored :-(
If find the use of the term "zero-day exploit" a bit rich as(according to http://www.milw0rm.com/exploits/8880) the vendors were notified on the 21st of may and the exploits has been 'in the wild' for a few days now. Strangely enough I have a VPS from these guys and it has not been affected, perhaps to do with my server being xen based....
"Low-end" customers get no backup service? Since it's a VM-based structure, surely one day a week, the "low-end" accounts could be taken offline, and the directory holding each VM's files be ZIPped or RARed?
I used to run some shell boxes in VM for people, and would, upon request, backup the contents for them - even though they paid me NOTHING for the service, nor for the backup procedure. Backups were password-protected and placed in a private FTP directory for the user to "collect" within a set period of time. if the user chose not to download their backups, they were copied to an external HD, as well as burned to DVDrom for safekeeping. I once actually snailmailed a user's backups to him, as he only had a dialup connection and as such wasn't up for the large download from FTP.
Mind you, my service was a hobby, and was only accessible to a select few "trusted" friends. It hosted no "sensitive" data. It was not "important", nor was it ever advertised.
Tux, because... WAAARK! (or whatever noise Penguins make).
This post has been deleted by its author
More info here:
http://www.webhostingtalk.com/showthread.php?t=867100
and here:
http://66.71.245.2/~vaservc/
I host a number of VPS's with vaserve, their communication has been pretty good through this. They are a no frills host and it's pretty clear that if you want backups, you need to make sure you get backups.
@ Christopher Ahrens
HyperVM is not the hypervisor, it is a web based management tool for Xen and Virtuozzo which are both "real" virtualisation technologies.
The real culprit here is LX Labs for not delivering a secure product, despite their claims on their website:
http://lxlabs.com/software/kloxo/security/
"We take security as the most serious of the concerns and have worked hard to create a secure environment where you can be confident about the server's state."
This is clearly rubbish.
Of my two VPSs one has been unreachable for over 24hrs. :(
Over the last year I've had a good experience with Vaserv and consider them quite low on bullshit. Even today I've emailed them twice and had replies in 2 minutes and 15 minutes. That is VERY fast given the scale of the problem they are facing. Obviously this whole situation shouldn't have happened and I'm facing data loss. However I could have bought the add on back up from them, used the web based control panel to do it or used rsync. So I can't really blame them for the lack of a more recent backup.
Initially they did a good job of keeping the information flowing. But then they listed some damaged nodes and then claimed "Everything else should be up and running for the UK". This was a mistake because it seems to be inaccurate, it raised expectations and I imagine it flooded them with queries about why things weren't working. Tired people make mistakes. Anyway hope this resolves as quickly as possible for all our sakes. I'm not very happy so my staying with them depends on their response to this exceptional disaster.
I'm still waiting to find out if my VPS was one that got rm'd, but if I'm honest I half suspected (along with some others here) that this might happen one day with HyperVM. Not because I knew anything about HyperVM itself, but because web apps in general pose so many security problems. There are usually many different input methods, all filtered differently and usually all with access to the crown jewels.
The annoying thing here is that if my VPS comes back on line do I assume the hacker(s) left a back-door somewhere, and rebuild it from scratch just to be safe? How very tiresome.
Paris because I saw this coming, ahem.
Jonathan,
You wrote:
"The researcher than found the vulnerabilities in the only gave the developer 2/3 weeks to fix it, looks like to me that he got peed off and released them to the public because lack of response."
I'd like to clarify this. First, I did not give the developer "2/3 weeks to fix" the issues. In fact, I did not give any timeline at all. What I did is what the advisory says. Had the vendor looked at the issues (which was not done) and requested some time to address them, of course, I would have given any amount of time requested before going public with the information.
Second, what it looks like to you and what really happened are 2 different things. I did not release the information out of anger with anyone. It was released so that customers, both current and potential, would be aware of the issues. It is not the job of the person who spends their time finding and documenting the bugs to babysit a vendor, plain and simple.
Third, as it stands, there is nothing whatsoever that definitively connects the current situation with the afflicted webhost with the information that was made publicly available. I audited Kloxo. As I understand it, and do correct me if I'm wrong, but they believe the issue was with HyperVM. I did not find out until later than HyperVM and Kloxo (formerly LxAdmin) share some of the same features/code.
Finally, lxlabs/kloxo/hypervm has been getting hacked for a while now, well before I ever published anything. Read their forums and you will see.
In reply to the above, here is an excerpt from that advisory:
# Timeline :
# 05/21/2009 - sent initial email to vendor with a link to a private resource for viewing various kloxo hiab575 vulnerability info
# 05/23/2009 - received the following: "Thanks for the info. I will review this and let you know." (no signature)
# 05/30/2009 - sent an email asking if there were any updates
# 06/01/2009 - received the following: "Sorry for the delay. I am currently looking into this, and will reply in a couple of hours time." (no signature)
# 06/04/2009 - nothing heard from vendor, and the private resource containing the vulnerability info still does not appear to have been accessed
# 2 weeks have passed since the initial notification. Vendor appears uninterested.
Your qualification to call yourself a white hat is based on the pretence of being interested in giving Kloxo information about weaknesses that they could fix privately. In fact, you failed to give them reasonable time to respond before calling up the hackers. There's no defence.
This post has been deleted by its author
We're a UK and USA based provider who have had to cease all new orders and remove HyperVM from all our servers.
After running several tests on development hardware, the issues still remain -- LXLabs have essentially packed up and gone home.
Bill gates, because he loves hackers.
"you gave people the warning to make a back up"
Where? He didn't even warn the company of what he was about to do. That would have woken them up.
The "full disclosure" cant is just a fig leaf in this case. I agree he can't be expected to babysit these stupid lazy tw*ts at Kloxo, but at the same time he overstepped the mark by a long way in this case. Knowledge gives power and privilege, but where was the _noblesse oblige_? He should learn from this, unless he just wants to create problems for himself in the future, especially given the dubious legal situation.
I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.
I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.
I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.
Vaserv subsidiary a2b2.net has something of a history of providing IP space for phishes and spam/phish maildrops, and as recently as last week was running a site offering turnkey phish kits from the same IP address that had recently hosted several bank and PayPal phishes:
http://tacit.livejournal.com/297618.html
http://tacit.livejournal.com/297775.html
http://tacit.livejournal.com/299317.html
Wouldn't surprise me one bit to learn that the attack was perpetrated by a current or former customer, somehow.
Why are so many people moaning about lost data and then no idea where their backups are? For flips sake I have cruddy little personal webpage on some free provider with photos of me doing stupid stuff, but hell's teeth even I keep a backup of it, so it can be resent! I know you pay for services, but quite often you will find that cut-rate comes at a price.
If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?".
M$ really should take the blame for the disasters they cause, with these zero-day bugs infesting their crapplications.
Oh? Sorry? This wasn't Windows-based applications?
Oh, right, I'll stop ranting then.
They should have used OSX. No security holes in that, no sirree!
But seriously, a total bummer to everyone who was on it....
<blockquote>I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.</blockquote>
The neat thing about mirrored data is that valid data manipulation such as deletion of unwanted data gets backed up to your backup system.
Mirroring data is only valid for guarding against hardware failures that don't cause bad data to be stored and replicated. Unless there is versioned backup to offline storage, there really is no backup at all.
If you don't have regular backups, then you should know you are rolling the dice. If it wasn't this, then any number of other possible events could have taken out your data.
Seems like a lot of people learn this the hard way by losing a lot of data at some point in time.
If you used a hosting service without a backup plan, and then didn't create your own backup plan, you really set yourself up for this kick in the teeth.
"I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.
I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.
I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job."
If your time line is only 24 hours from when you posted I take it you made sure your SLA with them for restoration of service was well within this time-frame when you signed up?
I doubt you did and I can have very little sympathy for someone who appears to be sent under by this because they failed to address the issue of tail events and business continuity.
I hate to break this to you, but it's sort of your fault for relying on one provider.
Even if you're a low-end provider (hosting sites on someone else's VPS setup is low-end IMHO) it's not hard to get at least one box somewhere else and regularly smuggle your data back and forth. Having all your boxes at one provider is stupid for all sorts of reasons what happens to your DNS, mail etc when the provider has down time, or goes bankrupt and gets their cables pulled?
"If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?"." ..... By Anonymous Coward Posted Tuesday 9th June 2009 04:24 GMT
And THE Bleedin' Obvious?! Diamond Geyser Rule, AC, which Personally Guarantees XXXXStreaming Fortunes to iCanny Personnel ...... Joint Venturing Virtual AIMachines into More Sticky Sweet Candy. ....... which would be in Quantum Communications Field akin to New Fangled Entangled Honey Traps/Blooming Flower Powers.
And as for the Virgin CLOUD and ITs Phorming Cloud Bases in CyberSpace, which you will not be surprised to learn Cloak Covers and Host SMARTer AIgents, which can Easily Zero in on Any and All Intelligence Led Operations, to lay Waste and/or overwhelm Systems with the Simple Disclosure of a Falsely Leading Truth which is always a Fatal Systemic Endemic Human Flaw stupidly carried forward into Binary Code in a Vain Bid to maintain a Previous Ineqitable and Positively Discriminatory Analogue Advantage.
However, Please be Cordially Advised, such is Considered and Deemed a Conscious Abusive Act in Virtualisation and Punitive Self Destructive Sanction Automatically Ensues to Purge the Systems of their Failings Preventing Future Travel ..... Magical Mystery Turing ....... and Virtual TelePortation Comunications Control of Global Events.
The SMARTer Operating System will Programme Accordingly to make Full and Beta Use of ITs Novel and Noble Facility/Faculties and all Others will Fail Miserably in the New SurReal Applied IntelAIgents Environment.
Which makes the Future Choice and Path to be Followed something of a No Brainer.
"In fact, you failed to give them reasonable time to respond before calling up the hackers" WHAT?!
Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this.
Vendor was provided with the private link to bugreport, didn't access it, didn't provide any information when this would be fixed, in short he had shown no interrest in fixing the bugs. Now, what would YOU do, mr anonymous smarpants?
"Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this."
On the contrary, in the section of the "security advisory" that I quoted it is clear that the vendor was replying to correspondence, but hadn't got around to dealing with it yet. Only a teenager without business experience (who else has the time to find bugs in other people's software for free?) would assume that this shows "no interrest [sic] in fixing the bugs". It's clear that Kloxo are lackadaisical about security, and I am in no way attempting to exculpate them -- indeed, looking at the vulns being exploited, they're complete t***ers -- but the fact they have problems is hardly unique in the IT industry, is it? That fact doesn't justify releasing these vulns so soon, and without warning. I can understand wanting a bit of kudos for finding all those bugs, but seriously...
You ask me what I would do. I would give the company a bit longer to respond, whilst embarrassing them with a public but non-specific security alert. If it took them more than a few months (let's say 6), *then* I would think about publication, and to hell with them. I would wait more than *3 days* for a follow-up to the last piece of correspondence...
A backup on a second raid disk is great in all intents and purposes but I'd say for that extra bit of security a backup on a completely different machine at a different location maybe even with a different provider would be a good thing.
I keep occasionally badgering one of my customers I support about their backups. They have a server with RAID disks which they backup but the backup never goes off site because they only have one backup drive. No matter how much I tell them that they need AT LEAST one other backup drive so they can alternate the drives and have an off-site backup it just falls on deaf ears because they won't spend the money on a bog standard USB hard drive (they don't have much data, easily under 160GB!).
Rob
"Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there."
If they offer an specific package which does not include backup but comes at a lower price point, then you'd have 0 right to be 'miffed' if there weren't backups.
Such a package is just offering customers what they want: cost savings based on the customer sorting their own backup solution out.
It appears this was all too much for the poor guy and the India Times is reporting that he hung himself yesterday :(
http://timesofindia.indiatimes.com/Bangalore/Techie-hangs-himself-in-HSR-Layout-/articleshow/4633101.cms
puts it all into perspective! This is what happens when a large chunk of the hosting industry base their entire businesses on a chap's "India-style-super-duper" software....
To start with, backups are essential for any web site (in fact anything on a computer), not simply an added bonus feature as Vaserv wishes to treat it. That's a fundamental mistake in their management and now they are paying for their false economy of not properly backing up. Everyone involved should have backups of their own sites as well as Vaserv having mandatory backups. (I always keep my own backups of the entire site for peace of mind as well. That way even if the host burns down I still have my data).
But that said, they have had the misfortune of suffering a shockingly bad hacker with an attitude of pure contempt. The hacker behind this attack is going to do serious time for this. It has to be one of the most expensive criminal hacks of all time?! ... 100000 web sites destroyed! ... the cost of destroying that many is serious money. How many thousands of businesses have been hit?! ... The lawyers are probably lining up around the building wanting to get in on all the ways cases and claims can be generated from so much destruction!
I don't blame LXLabs for being unavailable to most people. Their phones must be running continuously. I feel sorry for the programmers as they are now in the center of a horrific storm.
I don't think the full scale of this criminal action has sunk in yet. The major news channels and papers should be picking up on this.
A number of sites hosted by me for friends and family disappeared as a result of this hack, including a site for a business that I was going to set up last year.
I have a complete backup because it was obvious from the site that backups and data security was not something that they provided.
If you lost data in this, then I'm sorry but you should have read the T's & C's and taken appropriate steps.
Moral: Don't host business critical services on a system that costs less than £20 per month...
I've just had an email from vaserv who have advised me that:
================
Currently, we are enrolling a new platform on the new hardware for our customers who have lost all their data on one of the, unfortunately, lost host machines, and the ones that do have backups and would like to get things as soon as possible. Currently we expect to start deploying these during the night, once servers are prepared and installed.
We would also take this opportunity to outline that we will be issuing full RFO (Reason For Outage) and some other announcements related to this situation once everything is fully operative again. In addition to this, please read what we will be doing for you, our beloved customers, below:
- We will be applying 1 month worth of credits in case you have had a downtime for the day
- We will be applying 2 months worth of credits in case you have suffered lost of your data
========================
They've done a great job on this one, and I've even had an email from BlueSquare advising me that they will transfer VPS server to their own infrastructure ASAP, WITHOUT changing the pricing!
Well done VAServ,
A Very Happy Customer.
"I don't blame LXLabs for being unavailable to most people. Their phones must be running continuously. I feel sorry for the programmers as they are now in the center of a horrific storm."
There was no "they" - only a "he" - which you think might have sounded a few warning bells to large companies.
Now that he's dead and the vuln won't be fixed, I suspect there will be a few IT managers with some explaining to do.
Your comments should have killed him. Ligesh takes an attack on his code as big insult. He never use to release the software based on the same fear/ or did somebody challenge him to hack his code? He should have wandered in room like mad fellow seeing this vulnerability. I know him. He walks through the length of his room so fast scolding you all guys, and would have done this extreme step. He is a genius. At same time he was innocent too. Somebody should have been there to take care of him, You moron hackers.. You killed our Ligesh.
1. If your company can go backrupt because an unmanaged hosting service goes down, clearly you're doing something wrong. Or you are a cheapskate.
2. Mirrors are not backups, and anyone who thinks they are deserve what they get.
3. If you have no recent backups of your company data, you are doing something wrong, whatever other factors might be in play, whether it's hardware failure, remote attack, disclosed or undisclosed holes in the software, death of the developers, flood, fire, plague of locusts, up to and including a meteor strike, whatever. Your data = your responsibility. It's just a bloody web site, it should fit on a single CDROM.
If you trust "the other guy" you SHALL get screwed eventually. He who doesn't backup often, consistently, without FAIL will pay the price for digital ignorance. It's YOUR data, it's YOUR responsibility, and it's YOUR a$$ when (NOT if) things go poof!
Seems to me this is just nature's way of deselecting digital idiots.
"Since last night, I've had probably 40 phone calls from clients saying 'Why is my website down,'" said Daniel Voyce, a web developer for Nu Order Webs who uses Vaserv to host customer sites. "It's making me look bad."
I have no sympathy here, if you are using low cost hosting and not backing up your customers data then it's your own fault.
They explicitly offer a cheaper service without backups, and if you made the conscious decision to use the cheaper service which doesn't offer backups then you are either not storing anything of importance on it, sorting out your own method of backups, or stupid.
I have a cheap VPS with a plan like that, it's sole purpose is as a backup DNS server, which retrieves the data directly from the main server. The main server has regular backups, the backup DNS server is never backed up. Were i to lose that server, it would be a trivial case of copying a new set of data from the master.
"Voyce said the hackers, given the high level of server access they gained, were likely able to intercept a wealth of sensitive data stored on Vaserv's servers. Voyce said his customers are safe because all sensitive information was encrypted."
It may well have been "encrypted", but where were the keys stored and what type of encryption was used? I've seen lots of deployments where data is stored on an encrypted filesystem, but the key has already been entered into the system so that the database can access it, meaning you can still access the data from the running system, the encryption would only protect against someone stealing the physical drives. Sometimes it's even worse than that, the keys are saved on the machine so that it can access the encrypted drives at boot without user intervention required.
I work for a hoster. Backup costs money, this is why we charge extra for the service.
If you don't pay for backup then you don't have any right to complain that your site and data is lost either through the loss of a server or your own finger trouble.
You as a customer/reseller have to assess how critical your sites are and build in the cost of business continuity to the TCO. As a hoster we can provide all the backups and redundancy you require, but it all costs money.
The Real Kev K
Joe.
Backups were taken - both on VAServ and as personal scheduled downloads to another VPS. I also have backups of each site on a local machine. The main problem is that we dont actually have a server to put all of this on at the moment. Of the 5 servers I have with VAServ (and have had happily for over 3 years now - with no previous problems and excellent support). 3 have lost all data and are currently still offline - including the backups with VAServ. Which leaves me with local backups (totalling about 25GB) which I now need to commision / build a server (VPS.NET) and upload these backups aswell as waiting for DNS propagation for the new server.
It is not a case of data lost but rather service delayed as we strive to get these sites back on line.
The encryption was DES encryption with the users password as the key - entered when they login not stored anywhere other than in a salted MD5 hash in the user table. The only sensitive information was name and address.
Its a lot more than most people do and seeing as this was only an SQL injection of rm -rf there is no risk of any data leak full stop - I am secure in the knowledge that my users data would be protected even if they did get hold it - plus going to the effort of cracking a salted MD5 for a relatively small number of names and addreses would seem a bit pointless.
So - Joe, please explain what else you would expect me to do other than the steps I have already taken on this one?
I lost my virtual server there too. I thought, and still think, their price, terms and conditions are ideal for non-mission-critical stuff run by people who have sufficient Clue to organise their own backups. Their network connectivity, for the price, was excellent, the support very quick, and their choice of pre-installed operating systems (notably, Ubuntu 8.04) sold it for me.
One of the few affordable UK VPS hosts, I do hope they stay in the market for a long time.
Just got this email from VASERVE
Hello,
We are sending you this notice to let you know that your credit card ending in **** will be expiring in 10 days. Please be sure to log into your account and update your card to avoid service disruption for non-payment. Your login information is enclosed for you.
I dont think so ;-)
It appears that the attack was targeted more at VAServe than HyperVM.
The following purports to be the message left by the hax0rs of VAServe:
http://www.webhostingtalk.com/showpost.php?p=6227712&postcount=7
It appears that VAServe are putting a bit of a spin on the tale. Besides all the vulns in HyperVM, it was just poor password pol at VAServe:
"Z3r0 day in hypervm?? plz u give us too much credit. If you really really wanna know how you got wtfpwned bitch it was ur own stupidity and excessive passwd reuse. Rus's passwds are:" blah blah blah
Seems like VAServe cannot entirely pass the buck...
Someone wanting to do some damage to a particular website could pressure the programmer for the system password, suicide him then hack the main server.
You have to cover your tracks so you kill the person you got the password from and you kill all the websites so there is no clear motive.
Obviously a Black Op, hence the dark chopper.