Hacked flight sim site in catastrophic crash and burn A popular website for users of flight simulation gear has been felled, most likely fatally, after malicious hackers attacked both of the servers housing more than 12 years worth of content supplied by its 60,000 members. Tom Allensworth, the founder of Avsim.com, said in a statement that that an attack on Tuesday left the site …
I've heard of the term belt+suspenders a lot here on El Reg. They didn't have that. Belt + suspenders would be a hot mirror and then some offline copy (even an external drive with a copy of the data would have saved their necks). Earn bonus points from risk auditors for taking them offsite, and testing them regularly, etc etc etc.
Slashdot has gobs of comments on this and I don't want to start another long-winded discussion of the best way to avoid having this happen again.
If they have 60,000 users and if they can get the site back up then this would maybe help them move to a paid model!? Even $5 per year per sub would get them a backup method that works. If the fans are really that dedicated, etc.
As a moderately enthusistic flight simmer - more enthusiasm than skill - this is a real shame, but what's the motive ? There isn't a lot of profit in knocking off a hobbyist's site. A lot of annoyance, yes; the probable loss of a great deal of mental effort and ingenuity by the members of the community, yes; but no money.
It would seem something of an over reaction to harsh words from the Forum moderators. Was this just a dummy run before attacking a target with more profit potential ?
Chris Cosgrove
Sheeeeit - I'll do it for half that. Hell, just should "One!" "Zero!" at me, and I'll write 'em down by hand for that.
Seriously, half-way sensible offsite backups need cost damn near nothing. The admin guy has a home broadband? Just have a machine at his house that a backup can be automatically copied to in the middle of the flamin' night.
The site didn't die of a hack. It died of an abysmal lack of preparation. The hack was merely a symptom.
They didn't have any offsite backups or any form of disaster recovery, it seems. It doesn't say if the two servers were co-located but I can't imagine someone who's sunk 12 years of their life into a system with 60,000 users not thinking at least once, "what would happen if this place caught fire / flooded / was rendered uninhabitable" and then ignoring that train of thought ...
An SSH public key, a cron job, filled with something like:
ssh user@www.avsim.org pg_dumpall | bzip2 > /home/backups/`date +%s`.db.bz2
run from a home user's machine every night/week/month would have been better than nowt.
Add in an rsync to keep a local copy of the actual web files, and you'd be back up and running in 20 minutes.
What is wrong with people?
No point in having two identical sites running. As every time-traveller knows, you need at least three watches to have a chance ot Byzantine fault tolerance. Two fall too easily (as in this case) or just leave you with no idea as to which has failed.
Hindsight is wonderful, though, and it's probably all too clear to this unfortunate individual that they should have done things differently. My (belated) sympathies.
My cheap little NAS box has a 500GB USB hard drive hanging off it for expanded storage. Only the NAS box internal backup routine can write to it. I can read from it over the network so I use it as a library for all my media files, and an autobackup store for my relatively small NAS based work folder. (Yes, I know that anyone who breaks into my network and can get past the NAS box admin password can delete the USB drive contents, but I could configure my router to prevent that if I knew how; etc, etc.)
Some arrangement like that, or even hard drives that have a 'read only' hardwired switch would prevent major data loss from a website due to external attack. I know my arrangement is homebrew and doesn't have the bandwidth for any public facing use but surely some suitable arrangement can achieve this level of protection for a website which is vulnerable to external attack?
No need for me to say anything about the poor chap who runs this site, but to reinforce the point that we just dont learn......
Just as i was leaving the office yesterday, i overheard that typical conversation to IT support from someone which goes like "... but i have been working on that power point for the last 3 hours, surely it must be saved somewhere..."
I felt smug for myself and pity for mr powerpoint, as i had just watched our weekly backup to blu-ray start... so i know my arse is covered.
Same as everyone else! Can't believe there was no belt and braces (as we Brits call 'em!). Even my own personal stuff, including about 60 gigs-worth of music, is on no less than three separate external hard drives and is also stored elsewhere. Can't be too careful y'know!
Presumably everything is totally lost. No chance of a data recovery company being generous and having a look?
I assume the hard disks are not physically corrupt (smashed with hammer, given a once over with a grinder etc). Given that some data will be lost is there no way to use a hd recovery company to restore at least some of the data? Not that I'm a sim fan but the users seem a mite upset so if they all chip in a £1 they should be able to do something helpful.
As for backups..sounds like the guy was running on a shoe string. Still a bad call but he's just lost his lifes work....give him a break!
I hope someone will point out that the data's probably there on the disks, the inode lookup table's been nuked but unless the attacker actually overwrote the entire data partition(s) with random numbers in the DoD approved manner, there are plenty of people able to recover data from a scenario like that. Some will charge a lot of money, of course, but Google is their friend...
Nobody can "predict" server failure or hacking, that is why there exists a thing called a BACKUP.
Your "approach" was akin to building a house in a flood zone, then counting every year without a flood as a validation of your poor management skills.
Meanwhile, hard disk size has been continuously increasing and prices have been continuously dropping. Worst-case scenario for backups : take the server offline and plug in a bloody USB external disk. I'd wager they have some models with more disk space than the server.
I know hindsight is 20-20, but this is not a case of hindsight. Any admin worth his title knows that without a working backup, any server is just a victim waiting for its killer.
Well, you got yours now. You had six years to prepare for this, and you did nothing.
I don't feel sorry for you.
I know nothing about servers but TestDisk certainly recovered my XP box partitions after I had accidentally deleted them. It's a very powerful little utility, I even sent the author some money and that is rare for me!
http://www.cgsecurity.org/wiki/TestDisk
I have also used his PhotoRec to recover images off a duff SD card.
This guy is the VICTIM. He's made mistakes by not having a backup yes, but is that the problem ?
NO, the problem is that he was hacked, when did the sympathy die?
Now let's hear some derision for the scumbag/s that did this to him.
If it was possible to trace and convict the arseho++ that perpetrated this, the yes maybe $5 ayear for each of the 60000 members for 12 years wouldn't even begin to compensate the members for their loss.
It's easy to look at this story the way most of you have, it's not so easy to realise the problem is actually people like you.
Next, wtf do I care about the next shuttle exploding on re-entry, they had insurance didn't they?
I pay around $100US/year for a dreamhost account to hold my backups. They give full shell access over ssh, including rsync and give you shitloads of space - plenty to backup any website with a simple rsync. Or the admins could have simply rsync'ed the website down to one of their home PCs over their own broadband connections - total cost about 50 quid for a USB external drive to back it up.
Really, a 2nd machine in the same room is not a backup - if it wasn't a cracker it would have been a burst pipe in the datacentre or something similar (stupid stuff like that happens a lot more often that you'd expect).
Better be quick... Google's cache drops dead sites/pages as soon as it notices they've changed or moved. With a site of that size and popularity the chances are it's spidered every day. It may already be gone.
archive.org perhaps?
Even then, the value on these sites is the community - especially a high-tech field which changes as much as flight simulation (where 2+ year old info is mostly useless).
A better route may be to organize a fundraiser from the community to get the site back online (and some proper backups) and pick up where they left off. They'd lose some traffic/members, but the core members of the community would return and it'd rebuild over time.
Firstly for the benefit of the el reg mods - much of what is included here is firmly in the public domain and can be checked out on the temporary avsim forums.
Now onto the show....
For this undeniably catastrophic event - maybe a bit of wayback machine fiddling could help reconstruct the missing data. For the future ? Secure off site back ups.
Avsim was a very large site with gigs of goodies for all manner of flight sims and a vast user base, so significant that many third party FS vendors actually chose to make support for their products available though Avsim rather than in house channels. Any serious back up is going to get large I'm afraid. Backing up two servers to each other is not a professional backup solution and that at some point an alleged professional put this regime in place is stretching my credulity somewhat.
There's more to the story than meets the eye, without risking prejudicing a legal case or risking any liability - A supposedly professional techie has apparently admitted allegedly harvesting the user database for emails and used them to spam users with ads for FS related products on behalf of various companies without the agreement of those who own and run the site, from the avsim server, and allegedly for the benefit of his own company.
He was shown the door as a result, and the failure occurred within hours of his termination. He has stated on his own site which is similar to avsim in terms of content and function is that he believes the cause to be some kind of service/hardware failure. Naturally, speculation and tempers are running high amongst the members.
One thing is for sure though, if I were harvesting a user database for spamming users for my own benefit, from my employers server, I would sure as hell expect to be terminated pretty promptly. My views as to the professionalism of someone who expects such shenanigans to go undetected and unpunished ........... well...... you can imagine.
"It remains unclear who is responsible or exactly how the attacks were carried out."
So the partitions fail on the server, could be a host of reasons other than a hack - and with no proof he was how can he state such - if he's lousey enough to not have an offline back-up (my site has and I only made it three days ago) then he's probably as bad looking after his hardware.
So no sympathy until he's proven it's not his own stupid fault.
FAIL. Not only could this have happened, but the servers could have been stolen or physically destroyed/damaged (datacentre fire, flood, etc most likely), or any number of things where an offsite backup would prevent data loss. Makes me wonder if they even kept backups at all, or just thought 'hmm, its on RAID with redundant hardware, thats safe so we don't need to backup'
How many websites out there are given as much attention?
Do the people running the systems really value what is on them?
Lessons learned in the past 6 months. Mirrored drives are not backing up your system. And now, mirrored servers are not backing up your system.
Next lessons to learn, RAID is not a backup, large attached USB hard drives on the same network are not backup, NAS in the same building is not backup, Wayback Machine is not a backup. Syncing to a home computer is probably half a backup.
Poor minimal backup is transferring the data to something removable, taking it offsite and preferrably storing it in a bank vault.
Since he said both partitions had been deleted, surely (if a bit expensive) someone like Ontrack or some other recovery lot maybe able to pull some of it back?
As for backup yeah, does seem odd, I'm paranoid enough over varied backups of my own personal stuff and thats not even a 100 users. Can't stop feeling sorry for the poor bugger though, and makes no sense sadly.
I use USB hard drives on SBS boxes, one master drive thats perm plugged in for quick and dirty restores, and dailys for offsite rotation. I use a little prog that enables the usb port just before a backup and then disables afterwards. I use robocopy as an incremental mirror and the script is within a data folder.
The idea is that if the data folder becomes unavailable, then so does the script - no mirroring a blank drive.
And by disabling the USB port it becomes invisible to the server should anything happen.
This post has been deleted by its author
and it looks like the hack was (unofficially) made by a lad named "Phil".
It would seem that a month ago they hired this guy to do some upgrades on their system, giving him root access. Then, a week ago while upgrades were ongoing, spam started being sent to everyone on their mailing list, advertising a rival company ("Simfly"), which Phil apparently has some connection to. The owners of the Avsim accused Phil, Phil denied, owners fired Phil. The next day their website is obliterated.
Here's the forum post
http://linux.myalbemarle.org/forums/viewtopic.php?f=32&t=41
If you have limited resources, one relatively inexpensive way to do backups is to get another hosting account at a discount provider and pull a daily backup from the primary site. This allows
rapid recovery over high-bandwith connections, or simply changing the DNS entry for the site to point to the backup provider.
I'll add a vote for Test Disk as a viable tool for recovering the partition. Works very well, and is available on the Ultimate Boot CD, a necessary tool for anyone doing onsite support.
netgeek
Wow.. your comments are astounding. Yep, sure, big mistake not backing up the TB's of data that resided on the site, and how remiss of them not to have someone full time at the NOC there to run the daily backup.. after all, it was only two odd hours out of anyone's way to actually do that.
Its easy to say there should have been a more solid backup strategy, that's the benefit of hindsight, and it does seem a little reckless backing up web server (green) to ftp (purple) server.
That said, I'd like to think that there would be a little more sympathy from you for a site operator who had their site KO'd by some venomous little scrote with an axe to grind. No, I don't believe this was some random hack job, as from the looks of what happened, it appears that the hack was from someone with pretty intimate knowledge of the setup there.
I sure hope, Mr AC (16th May 2009 07:38) that you're just as smug when you need to restore from that Bluray disk to find it's corrupt to buggery.
Whilst their kit is fairly new, (supposedly identical machines),
they've had a long-term instability issue, (going back over many months).
AIUI
they were concurrently backing up, (rather than incrementally )
across both machines.
It's only an assumption that they were hacked.
They were coincidentally investigating those performance issues,
when they observed the processes failing.
Nothing else.
There was previously a spamming incident,
supposedly using their email database.
Several of those claiming to use a unique email address at AVSIM,
have been shown to use that same address on - their blog/facebook/bebo/other websites.
So if there was a hardware or OS misconfiguration
the similar time-frame for both failures
is not an indictor of a hack.
It might even have been a disk wrap-around.
Avsim was never exactly cashed up, and despite what everyone here says, proper backups are DAMN expensive.
Avsim had a couple of terrabytes of data, so even a 'relitively' cheap HP Ultrium ($6000 Australian - that a LOT of beer money) plus maybe $1000 in tapes per annum would be nowhwere near enough.
A full backup solution for a site of that size would run to maybe $25,000 min. and for a donation-run enthusiast site, thats just not possible.
Think of Avsim like your local library - it's not mankind's sole repository of knowledge, so it isn't nuclear-blast-proof secure; it is still a bloody tragedy if some 12-year-old ADHD tnuc burns it down.
Which is what seems to have happened to Avsim.
I was a member, some of my files were on there. The Avsim forums had been pretty closely following developments over the last three weeks, though since most of the discussion was on the forums, we've lost that.
The attacks originated mostly in Poland, and for a while were coming from a single IP. The Polish provider was NOT helpful in shutting it down or taking any action, unfortunately, and pretty soon it was obvious the attacks had spread to a botnet. To the more suspicious of us, it looked like a young punk testing attack vectors, possibly as rite-of-entry to a larger network of criminals. The attacks stepped up after the guys running the site started trying to secure it, but remember this is volunteer run, so the guys running it had day jobs to attend to as well. In hindsight, pulling the network cable for a month or two might have been the solution, but this was a community - imagine if the Reg dissappeared for a month with no reason given.
If nothing, it is a sad indictement on the human condition that things like this happen.
I doubt Avsim is going to be the only victim of attacks like this, and not everyone out there is a BOFH with attendant PFY and a large backup budget.
To me it really seems that action needs to come from the ISP-level, ISPs need to be held accountable when they fail to act on warnings of criminal activity. I don't know, maybe that's a simplistic view, but if the ISP in question had acted a bit more responsibly this might not have happened.
Tom has yet to provide a single shred of evidence that it was a hacker that nuked both servers. Word is that they were busy doing an upgrade and had a new guy on board (the whole tech team as I understand are volunteers for different parts of the US/world).
I recall an upgrade (back in the 90's) by a hardware vendor to a massive parallel processing database cluster we had. They had to take the o/s version through two upgrades. Decided to try a shortcut and then screwed up every shared raw SCSI device (on which the cluster database resided). After the upgrade, the database was trashed. .
Did the backups work? No. As we told them, when forced by management to sign the acceptance certificate for backups, a dd command reading a raw device and dumping it to tape does not suffice as a properly verified backup.
And who was to blame? The vendor of course blamed the Unix sysadmin and me.. despite the fact that we had nothing to do with the upgrade (all done by the vendor) or the backup method decision (granted the backup acceptance certificate was signed by us, but only after we wrote "this may not work" on it).
In Avsim's case.. no robust backup strategy. System goes fubar after work was done on it? Easy to blame hackers. But would a hacker have destroyed both servers (thus being unable to plaster his signature on the home page and give shoutz to the boyz)?
Of course, it may have been a script kiddie. Which is even a bigger indictment of how poorly Avsim was run technically. How the hell do you leave doors opened for script kiddies to gain root access?
As a long time Avsim user, really sad to see it no more. But methinks Tom and his guys need to take a long good hard at how they (technically) ran Avsim.
Don't know much abou the whole thing myself, so I could be totally wrong. However I understand Avsim was a hobbist site, run by an Av Simming fan. Was he an IT professional? if he was then you may have some just cause to snear at his lack of backups. However it is most likely that he wasn't an IT expert (unlike you lot) and didn't realise that the environment that had been built (wasn't it built by an IT professional for him) was structually unsound. Another IT professional worked on the site (the alleged stealing of email addys is another topic) did _HE_ tell the site owner of the dangers?
You probably know precisely how a modern PC works and what is happening in each part of the whole system. You _DO_ don't you? Most users think that a PC is filled with magic faries and you wake them up when you switch the box on. Thats why there are so many computer viruses about, the faries catch them from each other. Mention Backups to a normal user and they'll think that you're talking about getting angry. And copying the data from one machine to another is probably more than most would ever consider doing.
SO WHAT?
Snigger to yourselve about his lack of knowledge while you take your car to the garage for a service at £40 to £50 an hour labour. Because _you_ don't know how to do it. How about a bit of plastering to repair a wall? Gonna throw together a bucket of muck and slap it on the wall then? Oh look, lady over there has broken her arm and _LOOK HOW FAR THE BLOOD'S SPURTING_. What are you going to do? faint at the sight of the bone sticking out?
How about remembering that you get paid to do what you do because others don't have your skills. Just like you don't have other skills that are neaded from time to time.
Give the bloke a break. If you want to help then you offer help. REAL HELP. Go over and help setup his servers. Maintain them for him. Set up the backup routine and archives and mirror sites.
Until then STFU.
Only it's distrubuted amongst the community af AVSIM users. Though the structure of the HTML etc may be retained by places like Wayback, Archive.org and Google's cache, the actual binary files (arguably the most valuable part of the site?) will be sitting on the HDDs of the 60,000 members. After all, these files are valued and in some cases paid for. Can't the community pull together and reconstruct the binaries from the user's own HDDs?
This can be cheaper than you might think.
I found this out the hard way doing a rebuild. Forgot to load the RAID drivers during the boot from install media process, repartitioned and formatted the HDD only to find out that, since the OS couldn't *see* the internal disks, what I'd just hosed good 'n proper was the the USB backup drive. Numb feeling. Cold sweat. Replace underpants.
An hour's squirrelling with Google turned up a nice, if somewhat idiosyncratic in its use of English, Chinese disk recovery utility costing 30 quid that got everything back for me.
Yes, yes, yes, we techie's all know about backups, and offsite, and archives on mutiple volumes into clouds on really cheap or free services.
However do the journalists reporting the story fully understand the technology they are reporting on and the technical implications? The Reg may be one of the better "media", but let's not forget that "Dan Goodwin in San Fransciso" gets paid for what he writes, and anything that raises the profile of the story potentially increases his pay packet. There's little story in "flight sim site hacked, but they should have it back online in 6 hours"
(I'm only using Dan as an example - his credentials might be exceptional. I'm merely pointing out that those who are commenting on how to do backups should make sure they have all the facts)
Yesterday,
All those backups seemed a waste of pay.
Now my database has gone away.
Oh I believe in yesterday.
Suddenly,
There's not half the files there used to be,
And there's a milestone hanging over me
The system crashed so suddenly.
I pushed something wrong
What it was I could not say.
Now all my data's gone and I long for yesterday-ay-ay-ay.
Yesterday,
Need for backup seemed so far away.
Seemed my data were all here to stay,
Now I believe in yesterday.
--------------
-Unknown
"Snigger to yourselve about his lack of knowledge while you take your car to the garage for a service at £40 to £50 an hour labour. Because _you_ don't know how to do it. How about a bit of plastering to repair a wall? Gonna throw together a bucket of muck and slap it on the wall then? Oh look, lady over there has broken her arm and _LOOK HOW FAR THE BLOOD'S SPURTING_. What are you going to do? faint at the sight of the bone sticking out?"
I earned my early life wages as a mechanic, so I can quite happily service my own car thanks, I also put up an extension five years ago and it's still strong, warm and looking good. I also did an advanced first aid course, so I'd be able to handle her until the publicly funded medics turned up. Oh, and I'm a coder.
What about you?
You're absolutely right I don't know a air filter from an oil filter, but that is why I PAY SOMEONE WHO DOES. I see someone in distress as you discribe, I call emergency services, because I am not qualified, AND I KNOW THIS. A bit of platering to my wall wouldn't bother me too much, but I am presently tendering bids on a new heating/Air Cond. unit.
Note something subtle here, when I am not qualified to be doing something I HIRE SOMEONE WHO IS. Everyone and their mom thinks they are quilfied to run servers because they once used word, but this is a profession that requires training, expertese, and experence. The public at large need to be made aware of this.
I've seen a few articles on this now, and the focus of them all has been "poor flight sim site, destroyed by malicious hacker".
Granted, whoever did this certainly seems to have had acted out of malice, and is certainly guilty of criminal acts. It doesn't much matter in the scheme of things why he/she did it. Vendetta? Revenge? (Things can be tough in the flight sim world, I'm sure!) Money? Just because they can and are the sort of sociopath that will do something like that?
What slightly jars for me though is that the injured party avsim.com is portrayed as the victim of the hacker, when surely they have to take some blame for not having any offsite backups. None! As far as I can gather, not a byte of their apparently irreplaceable data was backed up offsite. I know this was a privately run, for-fun site but still. External disk drives with hundreds of gigabytes of storage capacity are pretty cheap these days. DVD-Rs are a matter of pence. If you have the wherewithall to run a website on your own, I'd be amazed if you didn't realise how important backups are.
Yes this was an awful thing for someone to do, but let's not pretend the proprietors of avsim.com were completely innocent victims!
I've read your comment and you make some reasoned points all of which are agreeable, except for one which was your last point.
The only thing it seemes and most people comment on it with no sympathy or understanding is that no backup was made and that somehow means that he deserved what he got.
I disagree with this completely and it's a sad reflection on the IT community and perhaps the general public that this is the consensus view.
In hindsight it was careless not to have a backup, and in the future I'm sure the same mistake won't be made, but this does not somehow make him guilty of anything except an error of judgement.
Very little has been commentated on the perpetrator of the real malice, and this somehow makes it OK to hack, and thence "criminal" not to have a backup and thence he deserved every thing he got.
Is that a true reflection of today's morals?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
