Emergency services call-handling provider: Ransomware forced it to pull servers offline Advanced, the MSP forced to shut down some of its servers last week after identifying an "issue" with its infrastructure hosting products, has confirmed a ransomware attack and says recovery will be in the order of weeks. The incident was spotted on 4 August and efforts to contain it resulted in server and network connections …
If you are providing services online then you need to assume that they will be hacked like this, so maintain complete backups that are regularly maintained offline. That can help you restore services after you are hacked but it don't stop anything daily. When the Internet was created it was designed to be universally accessible ... these days we can see that this was a wonderful design feature originally, but nowadays it's a problem - restricted access would solve nothing but it could make the daily hacking attempts a bit harder and the defenses better.
Surely a service like this had a document recovery plan in place that would have as least some functionality back within a few hours and total recovery within 48 hours? Supported by regular disaster recovery exercises, of course.
No? Oh.
Makes you wonder who placed the order without checking stuff like this...
Full recovery within 48 hours? I can bet that conversation would have gone something like this:
NHS: We want full recovery from any cyber attack in 48hrs please.
Bidder: ok, that'll cost this much.
NHS: We don't have anything near that much money.
Bidder: For that much we can do this.
NHS: ok. That'll have to do.
Government Minister: "We're putting not money than ever into public services, creating an NHS fit for the modern age blah blah blah tax cuts for all!"
You'd hope that DR plan included a bare-metal rebuild and data recovery after the Wannacry incident, though we all know it usually comes down to budget.
If you happen to be writing a contract for IT services you should specify that *successful* DR tests are required. I've worked at a large company that did it's contractually-obliged regular DR tests but wasn't required to report the results. They failed every time.
Yet another case in my experience where recovery is weeks/months.
Ransomware attacks are expected. No-one can be sure of thwarting every attack. Recovery from a complete network compromise must surely be part of any professional planning nowadays. The plan will have timeframes. Is anyone actually signing off any that don't have something like 48 hour to core re-functioning? A day to flush or replace existing systems - and another day to bring back core data?
Yet so many times it isn't happening. Some may be explainable because something outside of the expected happens. But not all. I suspect that having redundant hardware/people/licences and practising live recovery is a price many bean counters may pay lip service but when it comes to shove - today's emergency trumps next week's risk when it comes to budget.
And it's going to be expensive iif you need to retain existing kit for postmortem examination which implies to you need to bring up a parallel system. Redundancy big time,
Ransomware attacks are awkward. you have to be pretty certain that the recovery systems that you build are not coming from infected backups.
I'm not saying that they do this, but if I was someone wanting to place a ransomware bomb in a system, I'd probably want to install and spread it but leave the encryption dormant for several weeks, so that it would be copied onto the backups.
By doing this, you could probably immediately re-infect the environment that is being rebuilt, especially if it is just a timed trigger rather than an instruction from a command and control system external to the environment.
What I really struggle with is the fact that so many environments appear to be easy to infect. I know that the malware probably involves privilege escalation as well as the ransom encryption, but in a properly segmented environment, you should be able to contain an infection before it spreads. But I suppose the rush to consolidate systems into easy to manage large groups probably works against you there.
I think the answer is, don't rebuild the software from backups, build that from the original source. Only restore the data from backups.
Of course the software, even if it isn't infected, will still have the same vulnerability that allowed the original attack to happen, so you need to identify and fix it.
It depends on the nature of the servers.
If you have large numbers of similar or the same servers (well, built from the same image, with the same installed software), the rebuilding from sources or a gold image is quite possible.
If you have significant numbers of servers that have a long history, with different programs that may not necessarily be amenable to scripted install, large amounts of customisations, patches and upgrades etc, then rebuilding from source is not an option, at least not without keeping more people in your support teams than your management is prepared to afford.
Where I currently work, we have systems with over 10 years of history (the systems have been migrated to newer hardware at least once, and it's not Wintel, before you ask), and it would not be possible to 'rebuild from source' in a convenient time. The recovery process involves taking OS system images on a regular basis, and having a mechanism to restore these to the same, or different hardware, and then restore the data.
From long experience in this business, I know that automated install is desirable, and I've worked on more than one attempt on implementing, but without significant resourcing, it will and does break down over time whenever it becomes easier to do something by hand as a one-off, rather than adding to the installation process. And eventually, you have to start from scratch, and hope that the 'new' method results in an environment that functions the same as the old one.
But then again, the OS I work with is normally regarded as being less vulnerable to attack than some.
The file servers got ransomewared on a weekly basis.
Every penny has to be spent on patients, so sod the staff and sod the systems. They can't been seen to spend tens of thousands of pounds on something intangible such as software licences. The buildings are sweaty, depressing and dilapidated.
There are umpteen Windows domains because the neighbouring NHS Trusts went bust and had to be taken on.
It's famously impossible to get sacked from the NHS, so the IT department is stuffed full of incompetent people and bullies.
Services have to be available 24/7 "because of patients", meaning there is literally no downtime, meaning nothing ever gets patched because you can't even take a server offline for fives minutes to do a reboot.
You're stuck with obsolete third-party software that does things like rely on SMB 1.
Etc etc
And that's why I no longer work there.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
