Like stealing data from a kid: LA school pays web scum US$28,000 ransom A Los Angeles school has made a whopping US$28,000 ransomware payment after hackers raided its network. Attackers had encrypted enough to ruin computer services, email, and messaging at the Los Angeles Community College District. The school paid the bitcoin ransom after learning it had no other alternatives by way of backups …
"This writer has argued that paying ransoms is a legitimate if unfortunate last response to ransomware; cries from law enforcement that payments mean shoring up criminal business models is not the primary concern of administrators."
Your argument is wrong, and shows up your thinking as a short term contractor with no concept of long term security. The more people that pay now, the worse the problems will get and the more difficult our job as sysadmins later. Injecting money into a relatively immature criminal enterprise is never a good idea; they'll use the money to upskill, and that's an arms race we don't need.
I have no idea, maybe they didn't have one. But they've certainly painted a decent-sized target on their own backs now, as every ransomware cretin out there now knows they'll pay up. If I were the sysadmin there I would now be bracing myself for a rash of infection attempts.
That's not actually my point though; I take issue with the author's standpoint that paying a ransom is legitimate from a sysadmin perspective because it's not our problem. By paying the ransom, it ultimately becomes our problem, because (a) the ransom gets invested in better and better crypto which makes our job that much harder, and (b) once a ransom is paid, it never stops at one infection.
From the linked PDF:
How did we pay for it?The District has a cybersecurity insurance policy to address these specific types of cyber intrusions and it
was activated during this incident.
I cannot believe that insurance exists for paying out to ransomeware; it'll do is to support the "market" for this kind of attack. I could understand some kind of insurance to payout for some advanced geeky for data recovery, but not for this.
When you buy an insurance policy it will state exactly what it covers and does not cover - in this case they had coverage and the Insurance company paid up. To the insurance company this is just like a flood, a fire or an earthquake - a valid claim was made and they honored their policy. That's what you pay insurance for and why you buy insurance.
Realistically the Insurance company got off lightly - where I live we had a flood back in August, current estimates for the damage in my area are around 4 billion dollars US - the local school system shut down for about a month and we are still working on repairing the damage.
Is like saying fire insurance shouldn't cover arson. You could make the same argument that replacing a building that was burned down with an identical building would be a tempting target for more arson.
OK the comparison is inexact, and I agree that paying ransom should be a last resort, but it sounds like they were down to that last resort in this case! Sure, maybe it will make them a tempting target for phishing attempts to infect them with more ransomware. Hopefully they will improve their backups so that if it happens again they can simply restore from backup, losing at most a day's work, and there will be no further ransom payments.
Well designed crypto ransomware can take out backups as well, usually by only triggering when enough time has elapsed for offline data to be encrypted along with the online version. Given enough time between infection and encryption, an enterprise can lose enough data to make it a seriously painful experience. It might even be one critical file, recently created, that was worth paying $28k to recover. No idea if that was the case here, but getting f*cked by ransomware doesn't necessarily mean your backup strategy is shite. Lessons should definitely be learned though...
Why hasn't something been done at the OS level yet?
Some kind of service that watches for file change patterns and blocks writes when patterns occur, such as one file after the other being changed, or a change every 30 seconds etc.
People don't often make changes to all their documents at once so any kind of activity that causes a pattern in file writes is suspicious and should be blocked by the OS. At least that way people would only lose a few files, not everything.
Also, if not already an option, backup software should report that "more than the average number of files have changed, I'm forcing you to use a different media this time."
Come on MS, you are so keen to force updates on us since Win 10 - I wouldn't mind so much if your updates offered this kind of security.
The problem with something like this is that as soon as it sees widespread deployment, ransomware authors are gonna tweak their software until it doesn't trigger the checks anymore. Writing and re-witing files is after all a basic OS operation that you are normally supposed to be able to do.
It's definitely a good idea to build something yourself that detects this, however.
Why hasn't something been done at the OS level yet?
Because it will slow down the system and people will turn it off.
In any case, if if were added and fell into regular use, the malware would simply work around it. I'd hazard a guess that, to an external observer, the difference between malware encryption and background disk defragmentation is quite small.
I'd hazard a guess that, to an external observer, the difference between malware encryption and background disk defragmentation is quite small.
With disc fragmentation, surely it's just blocks being moved around the disc. I think that zipping files and deleting the originals would be nearer.
> triggering when enough time has elapsed for offline data to be encrypted along with the online version
How would that work? I would expect the attack to be immediately obvious to an enterprise of this size, and the very first thing one would do is to isolate the backups and shut down the network, probably invoking the business continuity/disaster recovery plan at the same time. In the past, when we used to do backups to half-ton tape drives, the backups were 'grand-fathered'. I don't know how modern backup technologies work in this respect.
Not easy to get at backups on a well designed system, but Simplest way;
1. Encrypt backup copy whilst leaving the active copy untouched (infecting shadow copies on the fly).
2. Repeat through a number of backup cycles (hour/day/week/month).
3. Disable shadow copy, disable system restore, delete restore points.
4. Start encryption of active filesystem, starting with least-used files first (to maximise time before detection).
5. Ask for ransom and watch the scramble to find a backup that wasn't encrypted.
More sophisticated ways include RAM overlay (holding an unencrypted copy of the filesystem in RAM whilst fully encrypting the stored version).
It all depends on the crypto variant you got infected with.
Well designed crypto ransomware can take out backups as well
Only on not-well-designed backup systems. For a start your backup machine should not be administrable by any account on any target machine, and ideally be of another OS (so simple privilege escalation tricks can't be reused all the way).
Secondly use frequent snapshots on a copy-on-write file system like ZFS on your fileservers - they take no additional space themselves, and if you do get a crypto attack you see the disk space plummet as everything gets changed. Due to the small space usually taken by snapshots and common file usage patterns you can often leave them for months. Such snap shots also make backing up to tape or rsyncing for replication to another server much easier.
Then if you do get attacked: Isolate infected machines, clean, and make a new snapshot from the good one (just in case it gets hosed a 2nd time), and finally resume.
There are backup strategies that offer a defense against this kind of attack but they add significant complexity to the system.
This is a school system - probably with a system administrator who is getting paid a pittance, so their main efforts are directed to a moderate amount of backup protection and mostly stopping Little Johnny from hacking his grades.
I've said it for years: They are two types of people with computers, those who have lost vital data ... and those who are going to.
There are backup strategies that offer a defense against this kind of attack but they add significant complexity to the system.This is a school system - probably with a system administrator who is getting paid a pittance,
This is a "school" in the American sense, so actually what we rightpondians would call a university.
V1.0 said "This is a school system - probably with a system administrator who is getting paid a pittance"
From TFA: ... the campus' 1,800 staff and 20,000 students
That's twice the size of the university I went to (admittedly a long time ago!) so there will be more than one sysadmin.
In fact, the LA Community College District named in the article comprises NINE colleges with a total enrollment in Fall 2015 of over 130,000 students [1]. The ransomware attack was at Los Angeles Valley College [2].
[1] LACCD Fast Facts
[2] LACCD Chancellor’s Statement [PDF]
@Paul Crawford
Well lets look at a quick cost benefit analysis.
Assume you are a small company with an in house development team of 5 permenant developer, you are in the middle of a big project so you have an additional 5 contract developers too. Due to the nature of your business your developers work on virtual machines hosted on your network (The virtual disks, like everything else are hosted on a SAN), they have laptops but they are basically used as thin clients 90% of the time.
Not too unusual, actually pretty much describes where I am working now.
So you discover the infection three days in, your SAN has been encrypted for the last 72 hours and as a result the 3 backup sets you have for those days are also encrypted, your source control and all of the developer VMs are unavailable.
You have three options, the first is to use a free tool/vunerability in the malware to decrypt - assume this is not possible.
Option 2. Pay the $28k/£23k
option 3. Restore the developer VMs to a good state and loose three days work - have the dev team re-do it.
In your scenario you have ruled out option 2 so we are left with option 3. Re-work.
So we have 5 Permie developers (Lets assume on £45k a year average so around £170 a day each * number of staff * number of days for re-work = £2550)
You also have 5 contractors at £500 a day = £7500 (+ VAT realistically but lets ignore that)
So your total cost JUST for staffing the re-work is £10k (Not counting the down time you have to pay the devs while they wait for the restore), Factor in the cost of recovering the three day old backups and in all likelihood the overtime to catch up with where you were and you're approaching a point where you say, fuck it its close enough to be in our interests to lose as little time as possible, option 1 costs £23k and we can be back up and running in 24 hours option 3 costs £10k in wages for the dev team alone and we will be 3-4 days behind with a potential overtime bill of £10k to get back on track... In that situation I'd take the extra few k hit to stay on track.
The other scenario is that your production database and its backups are affected... in that situation your options are pay or loose 3 days worth of data with no way to recover...
You have to remember that once you're infected its too late, it IS going to cost you money whether that is spent restoring backups, paying wages for re-work etc or paying some scum bag to decrypt the data.. there is a tipping point where one option (even the unpalatable one) becomes more desirable from the perspective of continuation of normal business activities.
As with normal backups, I suspect that this will be a lesson learned, same as when I dropped my NAS down the stairs... loosing a few TB of personal data taught me the value of backups... I suspect the budget allocated to the implementation and testing of the backup solution here might be increased...
So you discover the infection three days in, your SAN has been encrypted for the last 72 hours and as a result the 3 backup sets you have for those days are also encrypted, your source control and all of the developer VMs are unavailable.
So your SAN has no (regular) snapshots? Generally I use/prefer NAS instead of SAN as few things need block storage and there is always iSCSI, and for that FreeNAS, which is free and pretty good, uses ZFS with easy options to enable snapshots and its a very valuable feature indeed.
Or your SAN (or snapshot mounting) is administrable from infected PCs?
While your point is valid - that if your are royally screwed then paying up might be the least-worst option, it is doubtful that having provisions for data protection are more expensive. Also what if your SAN had some KCL-style screw-up?
You say "you discover the infection three days in, your SAN has been encrypted for the last 72 hours". How exactly have your developers been working for three days on an encrypted SAN? That's not possible! Once the virtual disks are encrypted the VMs won't run, the developers can't work, and the infection will be known. Then you restore from the previous day's backup.
If you do nightly backups it is impossible to ever lose more than a day's work. The only way you could have your SAN encrypted for three days without anyone noticing is if it happened Friday night before a three day weekend.
> the unpalatable one becomes more desirable from the perspective of continuation of normal business activities
But paying ransom to cyber-criminals isn't a normal business activity, is it? I agree in principle with your cost-benefit analysis, but you ought to factor in (a) the extra cost of iron-clad protection against another attack, since paying up identifies you as an easy mark, (b) the time and risk involved in undertaking decryption (you'll be running software from a known bad supplier with no performance guarantees), and finally (c) the risk that the scum-bag that you pay may not give you the decryption keys anyway. Good luck requesting a refund.
Bootnote: This writer has argued that paying ransoms is a legitimate if unfortunate last response to ransomware; cries from law enforcement that payments mean shoring up criminal business models is not the primary concern of administrators.
Then, clearly, the law should be changed so that paying ransom is itself a criminal act.
... but perhaps it already is? Could payment be construed as conspiracy?
I take issue with the notion that companies should refuse to pay because it only encourages the bad guys. It definitely does encourage them. However I disagree that people should take a hit for the greater good, whatever that is.
If you find a knife in your ribs and a criminal making demands, are you going to refuse for the Greater Good and to discourage muggers? Hell. No. You will reluctantly give up your clothes, your boots and your motorcycle to protect your own life. One incident of refusal won't push the mugger into another profession, and the public won't help put your orphaned kids through college. The creeps only need to score often enough to make it worth it for them.
Government bodies can better withstand refusing ransomware, since they simply steal more of our money to pay for the damage. Private parties take a hit that they may not recover from. Sorry everyone your job is gone, but we took a moral stand, don't you feel better? Serves them right, the smug ones say? Get real! If targets like Kaspersky Labs can be hacked you can't expect small business to do much better.
And I suppose you claim to be such a not-pussy that you could beat the living crap out of a mugger who has a knife to your ribs without the mugger having a chance to sink it in between them? Or are so amazing you could continue to fight and beat the living crap out of him even with a knife stuck deep between your ribs?
I find that usually people who make brave comments like that are the biggest pussies in the world, and write about what they wish they were like. Those few who are truly capable of such a thing wouldn't feel the need to validate themselves in a public forum filled with strangers.
"Those few who are truly capable of such a thing wouldn't feel the need to validate themselves in a public forum filled with strangers."
Which makes you wonder what the ones that feel the need to state in a public forum filled with strangers they are pussies and wouldn't put up any kind of fight while being robbed are capable of.
Oh, Oh, please take my wallet and my captains hat and here is the stick and of course you can fly towards those towers just please don't hurt me with that little knife....
That worked out well, so they reinforce the cockpit doors and some suicidal fuck sees it as an opertunity to lock out everyone else and take 147 with him.
It takes and abundant supply of sheep to support a population of wolves.
A relatively decent backup system.
The thing is, if the ransoms are set this high then there's traction in convincing people to invest in backups (plus the cost of not paying up for a school that lost all its data would be running in multiple 7 figures and insurers are likely to find reasons not to pay out due to negligence.)
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
