back to article Massachusetts city tells ransomware scumbags to RYUK off, our IT staff will handle this easily

The City of New Bedford, in Massachusetts, has found a way to deal with ransomware without paying: shoring up defenses, restoring from backups, and rebuilding systems. The attack on the American city's systems was identified on July 5, after employees noticed unusual network activity upon returning from the July 4th holiday, …

  1. Paul Crawford Silver badge
    Pint

    A beer for the the MIS staff for doing a good job!

    Hopefully this will make other organisations look to their network architectures and backup arrangements before they get screwed.

    1. big_D Silver badge

      Yes. And paying the ransom isn't necessarily cheaper either.

      If a machine has been compromised, the minimum you need to do is re-image it and restore the data from a known-good backup. You might use the decrypted files for those that can't be restored, but are you really going to trust a computer that was infected?

      A friend of mine is an administrator at another company and their server was found to be listed on a darknet website. The Verfassungsschutz (Federal Office for the Protection of the Constitution) recommended that they destroy the drives and the motherboard of the server and restore to a new system from a secure, offline backup.

      1. steviebuk Silver badge

        Thats massive overkill. Mark Russinovich bought this idea up on one of his talks. That the first port of call people do is wipe a system, but he said its not always required. Showed his tools in use hunting malware. He was the guy that found Sony's rootkit all those years ago.

        Yes wiping is best practice. But destroying a mobo and drives after an attack is massive overkill and a waste.

        1. big_D Silver badge

          On a server, providing company information, that has been infiltrated from outside without any alarm bells going off?

          Can you be sure they haven't root-kitted the UEFI or the management board?

          At one place, a user got hit by Goldeneye. We removed her hard disk, stuck a quarantine seal on it and re-imaged her laptop, she was up and working again in an hour. But for a server, I'd be more cautious.

          1. DontFeedTheTrolls
            Terminator

            "Can you be sure they haven't root-kitted the UEFI or the management board?"

            Is there any known malware that does this? Don't get me wrong, it is possible. Nobody trusts the G-en, but is it likely from criminals deploying malware.

            1. big_D Silver badge

              Yes. If they have remote access to the system, it is perfectly possible and there are known UEFI rootkits, such as LoJax or LightEater.

              1. Blank Reg

                Do server motherboard manufacturers not provide a way to validate that installed firmware is legitimate and unaltered? If not then perhaps it's about time that they did.

                1. Mahhn

                  They do not. These new bios infections do not overwrite the files, they create their own little pocket, the malware runs independent of the OS and cannot be seen by the OS, but it can see the files. This is from a Defcon presentation this year. Several of us asked bout flashing the BIOS, but that only overwrites the existing parts and won't touch the infection. I'm sure every AV company is working on detection methods. I have to say I have an older ROG2 MB and it has and independent Linux install in the bios that you don't need a HD for some basic things. Also heard of infections 5 years ago, but not as scary as todays.

                  1. Sir Runcible Spoon

                    Back to pen and paper I reckon, we're doooooooomed !!!

        2. Anonymous Coward
          Anonymous Coward

          Yes it's overkill, but for a lot of sysadmins it's a good excuse to get some money spent that they otherwise wouldn't be able to.

          1. CrazyOldCatMan Silver badge

            for a lot of sysadmins it's a good excuse to get some money spent

            Boss - that 10-year old server that you won't give me budget to replace? It got rootkitted and so I had to destroy it. Munnies plz.

        3. Anonymous Coward
          Anonymous Coward

          If you have a Russinovich at hand...

          IF you're extremely skilled and can hunt, identify, reverse and fully clean a system you can take the risk. I believe very few companies have such skills available. The safest path is to wipe and reinstall. Destroying motherboards and drivers depends on what they are being used for.

          1. missingegg

            Re: If you have a Russinovich at hand...

            I worked at a particular well known tech company last year. They had a well funded and quite experienced security team. They felt that best practices called for destroying client devices that had even potentially been compromised. It was just too challenging to keep up with all the potential hacks and risks that each device was subject to, and that's not even getting into the issue of zero day attacks.

        4. FlamingDeath Silver badge

          "Thats massive overkill"

          What utter BS

          Mark Russinovich is not some infallible being.

          Finding DRM created by a corporation behaving like maleware is not exactly the same as finding something which is state sponsored where remaining elusive is paramount

          What is malware?

          Something that does harm and is undesirable

          I would argue that if what you're saying is true, malware would not exist, because it would be so easy to find. Finding a smoking gun doesnt mean its the only gun.

          You can go right ahead and declare 'Mission Accomplished'

          1. Anonymous Coward
            Anonymous Coward

            Re: "Thats massive overkill"

            Agreed. In medicine, tissue samples are destroyed to prevent any risk of anything. Whether or not the risk is low is immaterial. If you destroy the samples, you destroy the risk.

            If anyone wants to gamble their career on a potential risk that's fine...me though...nah. If I can remove the risk entirely, I will. If I can't, the client signs a waiver absolving me of consequences.

      2. Balding Greybeard

        Just that Simple

        Offline backups are key to recover from malware attacks.

        FBI’s been saying that for years and organizations are finally listening and doing.

        1. Anonymous Coward
          Anonymous Coward

          Re: Just that Simple

          Trouble is a lot firms see offline backups as a step backwards. Most businesses hated tapes.

          1. Anonymous Coward
            Anonymous Coward

            Re: Just that Simple

            Tape is analogous to re-entering data manually, painful but with thought it can work. There is no excuse for ever losing data during an event.

            1. Anonymous Coward
              Anonymous Coward

              Re: Just that Simple

              Yeah, nah, yeah, nah, yeah.

              In my 20 year experience, I've rarely seen a 100% success from a tape restore. Tapes just don't get replaced often enough and aren't properly tested because...time.

              Restoring terabytes from tape then checksumming everything wastes a hurrendous amount of time.

              That said, your managed service provider loves tape backups because it's money for nothing.

          2. CrazyOldCatMan Silver badge

            Re: Just that Simple

            Most businesses hated tapes

            With good reason. They are time-consuming and pretty unreliable (even doing trial restores every couple of months won't catch all the bad tapes) and are generally a pain in the backside.

            Hands up all those that have had a critical restore fail (or take far, far longer than it should) because the one tape that holds the most recent version of $DEAD_FILE is corrupted or otherwise unreadable?

            [Hand up].

  2. Prst. V.Jeltz Silver badge

    Even after reading that RYUK link, i'm unclear how software that doesent know administrative passwords can tear a network up

    Sure , occasionally a massive vuln is found , like those colourfully named tools that wannacry used , and maybe theres some open network drives it can play in, but....

    know what im sayin?

    the link says at one point :

    step2 - escalate priviledges until is an administrstator

    yeah how?

    I'd have thought ransomewares main area of success would be home computers where everyone and their dog (or teenage son) is an admin and will click anything, and have lots of irreplaceable docs and no backup.

    1. big_D Silver badge

      There are enough vulnerabilities out there to escalate privileges. Malware can use unpatched buffer overflow vulnerabilities etc. to push up their rights for the local machine. If they can use an unpatched CIFS/SMB flaw, they can escalate their privileges on the remote file server as well.

      Even without that, if you infect enough PCs and make it a co-ordinated attack on the network, you will get access to a vast majority of the shared user data on the network drives.

      If one of the infected PCs is being used by a domain administrator, you have already lost, as it will have complete access - it can use the hidden, system level shares on the servers and other PCs to spread itself.

      That is why best practice these days is never to log onto a local PC with domain level administrator rights and to have a separate PC / VM used purely for administration, with no other software on it and not used for email, data transfers or web browsing.

      A home PC just isn't worth it. Most people wouldn't pay, and if they did, they wouldn't pay very much. You would be nickel and diming thousands of PCs to get the equivalent of one corporate take-down. That isn't to say that it can't/won't happen, but they aren't the primary target.

    2. steviebuk Silver badge

      Finds local admin on one machines. Attempts to jump from that machine to another and discovers all local admin passwords are the same. Eventually finds a machine where a domain admin was once logged on. Uses mimicats to get the hash of the domain admin. Domain admin has an easy password. Cracks it. They now own the network with that domain admin account.

      From there they can create dummy accounts and then delete logs. Then edit the odd service or two on some servers that give them readwrite persons on the service for write dac and write owner. This then gives them a hidden backdoor unless you're looking for that you wouldn't see it. They can then use that service, even as a non admin, to restart the service to run their own code. Then change the settings back leaving the write dac and write owner as their secret backdoor.

    3. cmaurand

      Someone got phished. This is how it starts. This is how they get a toehold. We used a system whereby we external snapshots on a device that could spin up any of the snapshots as a virtual machine. The system also kept those snapshots on a private cloud and could be spun up from there as well. Wi Dows has enough vulnerabilities that once on a machine, that's all it needs.

      1. Alien8n

        I can absolutely tell you that it does not need someone to be phished to gain access. We bought a company not long ago and discovered that at some point they'd been ransomwared (they got hit by CrySis). For some reason they'd left the encrypted files on the server under the assumption they'd be able to decrypt them at some future point. They restored from their backups, so I'm not sure what benefit keeping the encrypted files would have. Once we got their hardware setup at our head office we decided to keep the servers separate from ours, turns out that was a wise decision as they then got hit by Phobos. What's interesting though is that both ransomeware attacks use the same attack vector, by brute forcing their way through the firewall using an RDP vuln. After the first attack they'd failed to patch the firewall, leaving the attack vector wide open for a later attack. Also turns out they'd failed to patch the servers, which meant the RDP settings weren't secure either. Hindsight is wonderful, but looking back I'm so glad I took the decision to ensure the 2 domains were on separate networks.

        1. Nunyabiznes

          Did your company fire the entire network staff at the acquired company? I'm thinking they probably should have, especially after the 2nd trashing.

          1. rcxb Silver badge

            It's easy to blame the IT staff, but more often the fault is at the top. When company heads don't see the value in their IT dept, they don't hire enough bodies to do more than just respond constant business-stopping emergencies. Or they may offer the bare minimum salary, and only get kids with no experience and minimal knowledge, so the company ends up getting the mess they deserve.

            1. ecofeco Silver badge

              Exactly.

            2. FlamingDeath Silver badge
              Mushroom

              Great work experience for the kids though

              There is nothing quite like being forged in the crucible of a poorly funded IT department

              1. Anonymous Coward
                Anonymous Coward

                I second that.

            3. CrazyOldCatMan Silver badge

              company heads don't see the value in their IT dept

              Some of the places I've worked have regarded IT is just a waste of budget and only given them the absolute minimum.

              Those companies tend to be run by beancounters and MBAs.

              1. Anonymous Coward
                Anonymous Coward

                > Those companies tend to be run by beancounters and MBAs.

                So, basically all of them, then. Good show.

    4. Peter2 Silver badge

      Even after reading that RYUK link, i'm unclear how software that doesent know administrative passwords can tear a network up

      From personal experience of a polymorphic virus about ten years back i'd say that anything can do a heck of a lot of damage without needing administrative passwords.

      Programs executed in userspace can encrypt everything on file shares that the users have access to. Obvious, since it happens all the time with ransomware. But less obviously they can also alter files to ammend a copy of a virus to it those files; which is worse, especially if it lies dormant for a set time before going live.

      If you get everything on a file share with a virus attached and then an administrator opens anything on that file share then the payload executes, and you gain a new user to infect from. Shit hits the fan when somebody with domain admin privilages opens a file, since it then infects everything, everywhere.

      Of course, it's actually easy to stop with tools available out of the box with windows; put a software restriction policy in saying you can run executables in %windows% or %program files%, but not anywhere else. Virus infections will come to an immediate and abrupt halt since even if an end user tries to run a trojan horse attached to an email it can't actually do anything.

      1. ivan5

        The joy and fun of windows computing - insecure by design.

        1. Anonymous Coward
          Anonymous Coward

          Tell me about it. I was in charge of cleaning up all the firewall drops from systems in a *very* secure environment.

          One of the AD servers was making outbound dns connections to public IP's (or rather trying to) so I alerted the AD admins. They couldn't stop it, so they ask M$ and they said, and I quote, "that's normal behavour". WTF?!

    5. Anonymous Coward
      Anonymous Coward

      My first suspicions would be aimed at AD. So many companies I've worked at use a hellish mixture of versions and open up every port in the book (and then some) to get it to work across the company wan.

    6. Anonymous Coward
      Anonymous Coward

      Credential stealing. It starts with low level credentials, moves to a network share, infects another machine, steals those credentials then onwards and upwards.

      Usually a Trojan/dropper infects the network first before RYUK enters the network. That benign Trojan/dropper does all the recon first. Then when enough machines are infected RYUK is dropped.

    7. Michael Wojcik Silver badge

      step2 - escalate priviledges until is an administrstator

      yeah how?

      Take a look at the NVD. Microsoft have published 125 CVEs so far this year alone that describe privilege elevation vulnerabilities. That's 125 elevation mechanisms from Microsoft alone in the past nine months.

      And that says nothing about the number of unpublished ones; about the number of vulnerabilities published in prior years that many organizations haven't patched yet; about the vast number of vulnerabilities in the vast number of third-party software packages running with excess privilege. Hell, plenty of people out there are habitually running web browsers elevated, because they can't be bothered not to.

      And if an attack just gets local elevation (by compromising a local-admin user account, or a service running as LOCAL_SYSTEM, etc), then it's not difficult to social-engineer a domain admin from logging in (usually by making things malfunction until the user calls the help desk), and then using Mimikatz or similar to harvest the creds.

      Other OSes are for the most part not in much better shape. Even, say, z/OS, where a number of organizations have famously been compromised due to insufficient control over APF-authorized libraries that let attackers elevate. (Mainframe security researchers such as Dominic White and Phillip Young have documented and analyzed these attacks; the details are readily available.)

  3. Christoph

    "systems compartmentalization further limited the reach of the software nasty."

    Just hope you don't have a Big Boss who demands full access to every machine and then gets his own machine infected by ignoring basic safety rules.

    1. Kimo

      And the spread was stopped by turning off machines. My campus with 63,000 students likes to keep all machines on 24/7 for "updates" that could easily run at boot, leaving he entire network exposed.

      1. Trygve Henriksen

        It's for situations like these you have a 'Network Kill' script that shuts down every effing switch on the network by disabling all but the uplink ports on them.

    2. LeahroyNake

      Or the boss that wants domain admin rights for their user account... Just 'because I'm the boss'.

  4. Dwarf
    Thumb Up

    Thanks

    For not paying the scumbags.

    The less the attacks work, the less profitable they become which in turn makes it less likely the attacks will continue.

    The only question is what they will move onto next once they give up on this form of extortion.

    1. Drew Scriver

      Re: Thanks

      That would be nice, but I doubt it.

      If you could get only one or two paying victims a year you'd still get $500k+. Tax free. On top of that, if the criminal is based in some island nation even a lower amount would allow them to live like kings.

      Much like with telemarketing calls and phone scams. The success ratio is staggeringly low, but it still pays off.

      1. Michael Wojcik Silver badge
  5. usbac Silver badge

    I know this will be downvoted, but...

    A while back I got severely down-voted for for posting this, but I still believe in it.

    I think a Federal law should be passed making it illegal (jail time) to pay a ransom for data/systems. This would, hopefully, result in:

    1) IT managers and system admins would know that there is not an option to "just pay the ransom". Maybe they would actually get their shit together and properly secure their systems like this city did?

    2) The criminals would find that their income suddenly stopped coming in from ransomware.

    The big problem right now is that the attitude of "oh well, our insurance will just pay the ransom for us" is perpetuating the problem. And as long as the cyber-criminals keep getting rich from it, it's never going to stop.

    1. Drew Scriver

      Re: I know this will be downvoted, but...

      I can't decide whether I'm in favor of this, but it's an interesting thought.

      It would be better, I think, to criminalize negligence. Not only for the companies, but also for its officers. If it can be proven that they were personally aware (or made aware) and still declined to take action there should be fines and possibly jail time. Too often breaches are blamed on the lower echelons, while they in fact often reported vulnerabilities to management without any action resulting. At one point there was a bill to this effect pending in Massachusetts, but I don't think it passed.

      Secondly, insurance companies should require due diligence and decline the claim if a company has failed in this regard.

      Thirdly, auditors (government, insurance, and third party) need to talk to the people 'on the floor' - below the level of manager. Under oath, preferably. Now THAT would yield some shocking information.

      1. rcxb Silver badge

        Re: I know this will be downvoted, but...

        If it can be proven that they were personally aware (or made aware) and still declined to take action there should be fines and possibly jail time.

        That's been the downfall of many laws. It's extremely difficult, bordering on impossible to prove someone was aware of something.

        No matter how many e-mails, it's easy to say they didn't see them, or (if they replied) skipped over the important parts. It's trivial to insist that verbal conversations never happened, or your recollection of the discussion was very different than the dozens of other witnesses'.

        That's why lawyers make so much money. So many laws can be weaselled out of, with sufficient effort and lack of scruples.

        1. Anonymous Coward
          Anonymous Coward

          Re: I know this will be downvoted, but...

          > So many laws can be weaselled out of, with sufficient effort and lack of scruples.

          Effort generally not even required. When they have no morals, scruples, (soul, etc.) and are willing to compound lies on top of lies, even under oath, it's a tough nut to crack.

          Corporate execs, politicians, etc. have made a killing (sometimes literally) abusing the assumption that laws will be followed, people act in good faith, etc.

    2. Anonymous Coward
      Anonymous Coward

      Re: I know this will be downvoted, but...

      It is human nature to take bigger risks when you believe you have safety mechanisms to catch things. People come to rely on their safety systems - and then push them beyond their limits.

      What you wouldn't do in bare feet - you would do in steel toe-capped boots. Unfortunately that might lead you to either overestimate the strength of your caps - or forget you are wearing ordinary boots.

    3. Michael Wojcik Silver badge

      Re: I know this will be downvoted, but...

      as long as the cyber-criminals keep getting rich from it, it's never going to stop

      It's never going to stop, period. The rate of return can be very low and ransomware will still make economic sense for the attackers. Their costs (including risk) are already very low, and their marginal costs for attempting more infections are nearly zero. So it only makes sense to continue trying to infect systems even if the vast majority of victims don't pay.

      And it's likely that already a significant portion of the attacks are being carried out automatically by botnets. It's even possible that whoever controlled some of the receiving cryptocurrency accounts has lost access to them one way or another, and some ransoms are simply becoming orphaned cryptocoin - lost money. We know this has happened in some other attack categories, where zombie botnets continue to attack systems with no humans still operating C&C for them.

      And historically outlawing behaviors motivated by immediate need (real or perceived) has not been very successful. It hasn't helped with addictive behaviors. It hasn't done much to curb bribery or extortion payments in other domains.

      Prosecuting people for paying ransoms would be politically unpopular.

  6. O RLY

    Cut off one's nose ...

    "For New Bedford, no ransom was paid but Mitchell said he expects further costs in terms of MIS staffing."

    So, the reward for the MIS team that recovered without ransom is reductions in staff? That's the short-sighted decision-making I know and ... well, not love ... from civil servant budget people!

    1. Nunyabiznes

      Re: Cut off one's nose ...

      I think he meant that there would be more costs in terms of overtime, comp time, etc. Possibly even temp help or contractors hired to finish cleaning up the mess and validating security.

      1. O RLY

        Re: Cut off one's nose ...

        You’re absolutely right. I misread that sentence completely.

  7. Sir Runcible Spoon
    Pint

    Like the picture with the article

    Says it all really

  8. Anonymous Coward
    Anonymous Coward

    Pay insurance using the health sector model.

    You were treated out of network (heh) and four specialists were called in - although you knew nothing of that decision. The total cost of treating your malwaria, is $250.000, of which we think $1000 is a fair contribution from us, your insurers.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like