back to article Emergency services call-handling provider: Ransomware forced it to pull servers offline

Advanced, the MSP forced to shut down some of its servers last week after identifying an "issue" with its infrastructure hosting products, has confirmed a ransomware attack and says recovery will be in the order of weeks. The incident was spotted on 4 August and efforts to contain it resulted in server and network connections …

  1. Screwed

    Is it coincidental that I have today received my first fake/scam Covid proximity alert? The same message has already been reported multiple times in the past few hours.

    1. Anonymous Coward
  2. Version 1.0 Silver badge

    National Hacking Survival?

    If you are providing services online then you need to assume that they will be hacked like this, so maintain complete backups that are regularly maintained offline. That can help you restore services after you are hacked but it don't stop anything daily. When the Internet was created it was designed to be universally accessible ... these days we can see that this was a wonderful design feature originally, but nowadays it's a problem - restricted access would solve nothing but it could make the daily hacking attempts a bit harder and the defenses better.

    1. Oglethorpe

      Re: National Hacking Survival?

      Restricted in what way?

      1. tomuk

        Re: National Hacking Survival?

        Only making it available on HSCN would be a good start

        1. EnviableOne Silver badge

          Re: National Hacking Survival?

          HCSN is on the internet.

          It has a dual backbone, it's DNS is provided by Amazon, it's no longer an independent secure network like N3 used to be.

    2. steviebuk Silver badge

      Re: National Hacking Survival?

      They were most likely penny pinching on IT and didn't think it was "important".

  3. Mishak Silver badge

    Recovery plan?

    Surely a service like this had a document recovery plan in place that would have as least some functionality back within a few hours and total recovery within 48 hours? Supported by regular disaster recovery exercises, of course.

    No? Oh.

    Makes you wonder who placed the order without checking stuff like this...

    1. jdiebdhidbsusbvwbsidnsoskebid Bronze badge

      Re: Recovery plan?

      Full recovery within 48 hours? I can bet that conversation would have gone something like this:

      NHS: We want full recovery from any cyber attack in 48hrs please.

      Bidder: ok, that'll cost this much.

      NHS: We don't have anything near that much money.

      Bidder: For that much we can do this.

      NHS: ok. That'll have to do.

      Government Minister: "We're putting not money than ever into public services, creating an NHS fit for the modern age blah blah blah tax cuts for all!"

    2. Angry IT Monkey

      Re: Recovery plan?

      You'd hope that DR plan included a bare-metal rebuild and data recovery after the Wannacry incident, though we all know it usually comes down to budget.

      If you happen to be writing a contract for IT services you should specify that *successful* DR tests are required. I've worked at a large company that did it's contractually-obliged regular DR tests but wasn't required to report the results. They failed every time.

  4. Lon24 Silver badge

    Recovery redundant?

    Yet another case in my experience where recovery is weeks/months.

    Ransomware attacks are expected. No-one can be sure of thwarting every attack. Recovery from a complete network compromise must surely be part of any professional planning nowadays. The plan will have timeframes. Is anyone actually signing off any that don't have something like 48 hour to core re-functioning? A day to flush or replace existing systems - and another day to bring back core data?

    Yet so many times it isn't happening. Some may be explainable because something outside of the expected happens. But not all. I suspect that having redundant hardware/people/licences and practising live recovery is a price many bean counters may pay lip service but when it comes to shove - today's emergency trumps next week's risk when it comes to budget.

    And it's going to be expensive iif you need to retain existing kit for postmortem examination which implies to you need to bring up a parallel system. Redundancy big time,

    1. Peter Gathercole Silver badge

      Re: Recovery redundant?

      Ransomware attacks are awkward. you have to be pretty certain that the recovery systems that you build are not coming from infected backups.

      I'm not saying that they do this, but if I was someone wanting to place a ransomware bomb in a system, I'd probably want to install and spread it but leave the encryption dormant for several weeks, so that it would be copied onto the backups.

      By doing this, you could probably immediately re-infect the environment that is being rebuilt, especially if it is just a timed trigger rather than an instruction from a command and control system external to the environment.

      What I really struggle with is the fact that so many environments appear to be easy to infect. I know that the malware probably involves privilege escalation as well as the ransom encryption, but in a properly segmented environment, you should be able to contain an infection before it spreads. But I suppose the rush to consolidate systems into easy to manage large groups probably works against you there.

      1. katrinab Silver badge

        Re: Recovery redundant?

        I think the answer is, don't rebuild the software from backups, build that from the original source. Only restore the data from backups.

        Of course the software, even if it isn't infected, will still have the same vulnerability that allowed the original attack to happen, so you need to identify and fix it.

        1. Peter Gathercole Silver badge

          Re: Recovery redundant?

          It depends on the nature of the servers.

          If you have large numbers of similar or the same servers (well, built from the same image, with the same installed software), the rebuilding from sources or a gold image is quite possible.

          If you have significant numbers of servers that have a long history, with different programs that may not necessarily be amenable to scripted install, large amounts of customisations, patches and upgrades etc, then rebuilding from source is not an option, at least not without keeping more people in your support teams than your management is prepared to afford.

          Where I currently work, we have systems with over 10 years of history (the systems have been migrated to newer hardware at least once, and it's not Wintel, before you ask), and it would not be possible to 'rebuild from source' in a convenient time. The recovery process involves taking OS system images on a regular basis, and having a mechanism to restore these to the same, or different hardware, and then restore the data.

          From long experience in this business, I know that automated install is desirable, and I've worked on more than one attempt on implementing, but without significant resourcing, it will and does break down over time whenever it becomes easier to do something by hand as a one-off, rather than adding to the installation process. And eventually, you have to start from scratch, and hope that the 'new' method results in an environment that functions the same as the old one.

          But then again, the OS I work with is normally regarded as being less vulnerable to attack than some.

      2. midgepad Bronze badge

        Distinction between data and programs

        would help, no?

        Theres text, and there's pictures, and there's numbers in tables.

        And there are some programs.

  5. jdiebdhidbsusbvwbsidnsoskebid Bronze badge

    Recovery redundant?

    When it comes to redundant, that's not how government understands that word.

  6. katrinab Silver badge
    Mushroom

    They were originally blaming the heatwave for this, which I thought was a bit strange.

    I get that heatwaves can cause servers to overheat and shutdown, but I couldn't understand how they weren't able to just switch it back on once it cooled down.

    Now I understand.

  7. John Savard

    Impatient

    How long is it that we've had ransomware? And yet they still haven't improved the security of Windows so that such attacks are no longer possible?

    1. Trollslayer
      Devil

      Re: Impatient

      Ever heard of evolution?

  8. Missing Semicolon Silver badge
    Flame

    Liability

    Who to I sue to recover the losses I experience when my PII from this hack is used?

    /S

  9. Anonymous Coward
    Anonymous Coward

    I worked at an NHS Trust

    The file servers got ransomewared on a weekly basis.

    Every penny has to be spent on patients, so sod the staff and sod the systems. They can't been seen to spend tens of thousands of pounds on something intangible such as software licences. The buildings are sweaty, depressing and dilapidated.

    There are umpteen Windows domains because the neighbouring NHS Trusts went bust and had to be taken on.

    It's famously impossible to get sacked from the NHS, so the IT department is stuffed full of incompetent people and bullies.

    Services have to be available 24/7 "because of patients", meaning there is literally no downtime, meaning nothing ever gets patched because you can't even take a server offline for fives minutes to do a reboot.

    You're stuck with obsolete third-party software that does things like rely on SMB 1.

    Etc etc

    And that's why I no longer work there.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like