IT 'heroes' saved Maersk from NotPetya with ten-day reinstallation blitz

It's long been known that shipping giant Maersk suffered very badly from 2017's NotPetya malware outbreak. Now the company's chair has detailed just how many systems went down: basically all of them. Speaking on a panel at the World Economic Forum this week, Møller-Maersk chair Jim Hagemann Snabe detailed the awful toll of …

[Anonymous Coward]

'internet was not designed to support the applications that now rely on it'

10 days and only “a 20 per cent drop in volumes”.

Did he say how it ended up costing them $300m?

[Anonymous Coward]

Re: 'internet was not designed to support the applications that now rely on it'

"...in the near future, as automation creates near-total reliance on digital systems, human effort won't be able to help such crises."

Its good that Maersk acknowledge this. Since the media often downplay 'Automation' risks and ignore how breaches, hacks and malware will factor into things. Overall, I suspect many corporations are still just thinking: "Its ok, I won't get that malware 'AIDS' shit, that only happens to others".

[Paul Johnston]

Re: 'internet was not designed to support the applications that now rely on it'

Seems a reasonable figure

(10/365) * (20/100) * annual turnover

Not got an exact figure for the turnover but revenue was over $35 billion.

Add overtime and other sundries and its a big bill.

[Anonymous South African Coward]

Re: 'internet was not designed to support the applications that now rely on it'

So 20% drop = $300m less?

Boggles the mind...

[MMR]

Re: 'internet was not designed to support the applications that now rely on it'

Overtime, lots of takeaways, overtime, consulting and probably lots of overtime.

[rmason]

Re: 'internet was not designed to support the applications that now rely on it'

Imagine signing off on $3.8million for energy drinks and pizza.

:)

[Anonymous Coward]

Re: 'internet was not designed to support the applications that now rely on it'

rmason,

How do you think 'Just Eat' got so big so quickly !!! :) :)

[John Brown (no body)]

Re: 'internet was not designed to support the applications that now rely on it'

...and maybe late delivery penalties or ships overstaying their welcome in port due to the manual process.

[netminder]

Re: 'internet was not designed to support the applications that now rely on it'

I know that FedEx has penalty clauses built into contracts where they provide services. I have no way of knowing if they ever paid a 'fine' but I did hear that they had every available employee had sorting. With a revenue of $50B a cost of 0.6% would not be surprising.

Maybe if all these companies would listen to their security people and patch they could have saved most of that money.

[TonyJ]

I hope

That all the staff that pulled this off were well rewarded.

Because frankly that's a phenomenal effort that deserves it.

[Korev]

Re: I hope

I agree.

Also, I hope for IT's sake that there's a "Let's make sure this doesn't happen again" rather than finger pointing exercise .

Another one -->

[macjules]

Re: I hope

...rather than finger pointing exercise .

Like WPP did.

[Anonymous Coward]

Re: I hope

@ WPP 3 global AD forests, 1000s servers, dozens backup environments, 10000s workstations all encrypted. Networks are still wide open. They will get hit again.

[Anonymous Coward]

Re: I hope

"@ WPP 3 global AD forests, 1000s servers, dozens backup environments, 10000s workstations all encrypted. Networks are still wide open. They will get hit again."

I mean it's not as if WPP hasn't had these type of issues in the past with the constant stream of new companies/offices being taken on or consolidated into existing offices. Of the networks that were hit, some used to have systems in-place to stop this sort of thing however they were probably left to rot or unmanned and unmonitored while IBM waited to employ someone in India a year or two after the onshore resource was resource actioned. Or IBM have offered an expensive solution that WPP don't want to buy and neither side has the expertise to come up with a workable solution...

And there is some network flow data being sent to QRadar systems in-place now to identify issues but whether they would identify issues fast enough to stop senior IBM managers from making bad decisions is a different story. Unless it was a temporary solution and it's been removed pending agreement on costs.

Still, I'm sure WPP wouldn't want to upset such a valuable client in IBM by making public what the damage actually was.

[tonyszko]

Re: I hope

I can't tell all the details as it is covered under many agreements, but as a person who was/is involved in this process with our managed service team (who was part of this recovery process as well) I can just say that there is no finger pointing.

There is a lot of constructive changes and solid plan which is being implemented how to lower the risk (you can't rule it out) of such incident happening in the future.

It was really exceptional to see how Maersk team handled it and how all involved parties (Maersk IT, managed service teams - ours included and external consultants) managed to pull it together and recover from it.

Also it is really exceptional how Maersk is sharing the knowledge about it and what happened and how they handled it.

We will be covering our lessons learned and experience from this event soon (next week) on our blog -> https://predica.pl/blog/ if you want to check it out.

[Kevin Johnston]

Re: I hope

They have at least been acknowledged which is already a huge leap forward from the normal management responses of "why did you not prevent this" and ""that's the IT budget blown for the next 5 years, cancel the refresh program and forget asking for any overtime money"

[Doctor Syntax]

Re: I hope

"cancel the refresh program"

It looks as if the refresh programme was brought forward.

[Halfmad]

Re: I hope

That all the staff that pulled this off were well rewarded.

Because frankly that's a phenomenal effort that deserves it.

Annoys me that companies don't shout about how well their IT departments recover in situations like this. If they'd had a fire etc they'd be thanking those staff who helped PUBLICLY but IT is seen as a shadow department, we can't possibly talk about those people..

[tonyszko]

Re: I hope

But they do - they are speaking about it on multiple events and Maersk IT got prised for what they delivered in this case.

[Anonymous Coward]

Re: I hope

> Annoys me that companies don't shout about how well their IT departments recover in situations like this.

Err... This guy just did. Hence the headline.

[Anonymous Coward]

Re: I hope

Their main IT support is via IBM so you can guess the chances of reward were between Buckleys and none.... (unless you were a manager.)

They had lots of heroes including the local techie who had the presence of mind to turn off one of the DC's once they realised what was happening - that saved their AD.

We'd heard bits and pieces of what had gone on during the recovery (usual stuff you'd expect - block switches, block ports until each host was confirmed a clean build in a segment then slowly remove the blocking/site isolation.) We didn't, however, see any emails publicly acknowledging their efforts.

[Ken Hagan]

Re: I hope

"They had lots of heroes including the local techie who had the presence of mind to turn off one of the DC's once they realised what was happening - that saved their AD."

Hmm. Yes. I imagine the rebuild might have taken more than 10 days if it had included typing in a new AD from scratch.

[tonyszko]

Re: I hope

Main challenge was to have data from AD to start recovery.

Main question here is - how many organisations have procedure for forest recovery, which is mostly logistics task with good understanding of AD as a service.

My consulting experience from last 20 years tells me that 99% of organisations doesn't have it and never thought about it as something which will never happen.

[InfiniteApathy]

Re: I hope

>Main question here is - how many organisations have procedure for forest recovery

You just made my butthole clench so fast I'm missing a circle of fabric from my chair.

[tonyszko]

Re: I hope

And BTW - good forest recovery plan has "typing in a new AD from scratch." planned somewhere along the path of recovery if it is not meeting business requirement of recovery time. It was included in each "GOOD" recovery plan I saw and had a chance to build or read.

[Anonymous Coward]

Worse things happen at sea.

Compared to a ship sinking with possible loss of life, IT isn't their biggest problem. One ship and cargo probably cost more than $300 million.

Their DR plan must be good; comparing how the reality compared to their plan would make a good case study.

[Anonymous South African Coward]

Re: I hope

I had to recover an AD site once, it had only one PDC, no BDC's, but luckily there was a recent backup made of the PDC (Server2k3 and NTBackup).

Process was to reinstall Server2k3 on a clean server, run ntbackup to restore the AD backup, and we were back in business again. Only niggle was hoping that Windows Activation would go through as I was not in the mood to faff around with that - but it went through just fine. I then set up a BDC just in case, but still continue to make backups from the PDC juuuust in case.

And recovering the forest is no biggie as there's about 60 users - but a backup and BDC makes things so much easier.

But yes, forest recovery, especially with multiple sites and domains need to be addressed. Setting it all up from scratch by hand leads to errors and mistakes if due care is not taken.

[Stork]

Re: I hope

My time at Maersk Data (IT subsidiary until 2005) made me paranoid about backups. They considered to make mirrored systems kilometers apart.

[J. Cook]

Re: I hope

I've (thankfully) never had to restore AD from a backup, and Bog as my witness, I hope to never need to.

Pulling the plug on a DC was *definitely* a heroic measure- even if it's not a FSMO, if it's a global catalog server, it can be promoted to one and used to rebuild from.

[tonyszko]

Re: I hope

This wasn't handled through standard IT support. IBM probably had it role there but it was mostly recovered by Maersk IT team, consultants from vendors (you can imagine which one) managed service teams like ours and external consultant.

Lots of people were working there on shifts spending time on-site for several days to make it happen.

I can't say about public e-mails from Maersk, but Maersk IT team is very open on what they did and how, I saw members of the team speaking about it on few sessions.

I said this earlier in this thread but just for the information - I can't provide official information on it, but our managed services team who was part of this recovery effort spending couple of weeks on site is cooking a blog post with details to be published next week on -> https://predica.pl/blog/ if you want to check it out.

[Anonymous Coward]

Re: I hope

> consultants from vendors (you can imagine which one) managed service teams like ours and external consultant.

Job well done, Tony. Congratulations. :-)

[Stork]

Re: I hope

Sounds like the attitude has not changed much from when I was Maersk Data (separate, but very close to The Client). Get it sorted, we do paperworks later. And recognise effort.

[Anonymous Coward]

The last person who fixed a malware outbreak...

... got thrown under a bus by the UK government and then arrested by the US Feds. I wouldn't touch the repair and removal of malware without a waiver signed and a lawyer present.

[Adam 52]

Re: The last person who fixed a malware outbreak...

"thrown under a bus" as in wasn't told he was going to be arrested. That's not really the same thing and would have been a really big ask of GCHQ.

[Sir Runcible Spoon]

Re: The last person who fixed a malware outbreak...

They didn't mind getting him in to consult for them whilst they surely knew about the US intentions.

[Anonymous South African Coward]

How did they managed it? I would love to hear the side of things from an IT techie... it is stunning... the mind just boggles.

[tonyszko]

Lot's of people working on it on multiple fronts. Lots of logistics - for example, you might be surprised how little USB storage pens are on the stock in the shops :).

One important aspect - no panic! Don't do things if you don't fully understand the scope and what hit you.

Besides technical details there are lessons from the management side:

- do not let people doing the work being bothered by people throwing random questions or issues, put some "firewall" to handle it

- good communication where are you with efforts and what is recovery time is crucial. Dedicated team for it might help you A LOT.

As I wrote in other replies here we will be covering it soon on our blog from our managed services team perspective who was on-site to help recover out of it for couple of weeks.

[highdiver_2000]

This is a ransomware, so it is a massive quarantine, wipe, reinstall and safezone effort.

[Anonymous Coward]

Maersk's own experience is that the attack it endured cost it between $250m and $300m.

Now if they had spent only a fraction of that preventing it.

[Lysenko]

Re: Maersk's own experience is that the attack it endured cost it between $250m and $300m.

I'm sure (!?) that after his "Road to Damascus" moment Mr Snabe also directed that a further $300m be invested in DR facilities and redundant systems while also doubling the systems security budget.

Back on planet Earth.....

[Chrissy]

Re: Maersk's own experience is that the attack it endured cost it between $250m and $300m.

Probably not.... now the IT department has un-stealthed itself as still having some staff left then the CEO is asking "how come these people are still on the payroll.... why haven't they been outsourced already?"

[Anonymous Coward]

Re: Maersk's own experience is that the attack it endured cost it between $250m and $300m.

Depends on who was responsible for letting it happen. If the service provider they will lose the contract at renewal. If internal at least they recovered.

[Stork]

Re: Maersk's own experience is that the attack it endured cost it between $250m and $300m.

The company may have changed (I stopped working with the main IT supplier in 2007), but it was then a company that was aware how important IT was to them. Doing any work at their servers at headquarter, even when employed by a subsidiary, meant having one of their guys standing next yo you.

If they think it makes sense to invest further, they will do it.

- I hear a comment from an ext. consutant once: "we sat at this meeting, and as we talked hardware costs went up from 50k to 200k in 20 minutes - and the customer (Maersk) didn't blink".

[HmmmYes]

Youd think, with companies with large PC deployment maybe, just maybe, a light bulb might come on a ping 'Maybe we need to put a bit more diversity into the client OS?'

I mean FFS. Its a company. Most of the applications can sit beyond a browser now.

[Dan 55]

But Office! But Outlook!

[Sir Runcible Spoon]

Both of which work in a VM.

Sure, your VM is toast, but off-line backups and the underlying OS are all good yeah?

[tonyszko]

But even for browser apps you need service to authenticate people and some common services like e-mail. With 100k+ employees it is not that simple.

[Craigie]

20% drop going to manual

Beancounter: 'How much are we spending on IT?'

[Anonymous South African Coward]

Re: 20% drop going to manual

Beancounters are the natural enemy of the BOFH... Death to all things Beancountery! :) :p

[A Non e-mouse]

Re: 20% drop going to manual

Beancounter: 'How much are we spending on IT?'

That's a good question and IT should always be prepared to justify their costs. It's a shame the same can't be said of management.

[Boris the Cockroach]

Re: 20% drop going to manual

Quote: That's a good question and IT should always be prepared to justify their costs.

Horse shit to put it bluntly

IT is the core system that lets the company function, no IT no company

Thats as true for the 20 man band I attend upto the mega-corps like Maersk.

If you are not treating IT as the core section of the business and consider it an incidental expense (like my manager did once*) then you deserve to have your company crash and burn

This should be drilled into every C level exec with a big mallet.

*His eureka moment arrived thanks to a dead PC containing 10 years worth of robot/cnc programs ...

[cantankerous swineherd]

Re: 20% drop going to manual

IT the core system?

the one that sends emails that can't be replied to?

that says we're experiencing extremely high call volumes 24/7?

that turns employees into robots mindlessly reciting words on a screen?

the one that ensures your details can never be updated because the system won't let me?

I'm seeing a huge competitive advantage here, at a modest cost, resulting in a more effective and resilient organisation.

[Anonymous Coward]

Re: 20% drop going to manual

Quote: That's a good question and IT should always be prepared to justify their costs.

Quote Response: Horse shit to put it bluntly

Ker-wrong. They should still justify their costs but as a a core function it ought to be a whole lot easier. I'm pretty sure the story above said Maersk struggled but didn't fall over. Maersk is not IT in the same way that the IT companies with banking licenses are. Maersk moves metal boxes, not ones and zeros.

[Stork]

Re: 20% drop going to manual

You would be suprised how much IT Maersk actually is.

When I started in the group I thought it was quite simple to move a box from A to B, but when you have a million or so boxes (then) and have to keep track of where they come from, where they are going, what is in them, who does it belong to (and this is "legal title to ownership" sort of serious), can it be loaded under deck, how do you stack it on the ship to make as few moves as possible (without making the ship tip over due to wrong balance) you really need IT.

This is before implementing rules such as "do not send ALU containers to certain places as the locals find it too easy to get into them".

[J__M__M]

Re: 20% drop going to manual

Beancounter: 'How much are we spending on IT?'

IT spending is what pays my bills, and that number has me asking the exact same thing.

[The Boojum]

Re: 20% drop going to manual

I suspect that the loss would be an exponential function of time. Another 10 days? Major crisis. 10 more? No company.

[David Hicklin]

Re: 20% drop going to manual

"I suspect that the loss would be an exponential function of time. Another 10 days? Major crisis. 10 more? No company."

Plus some ships could have been at sea during those 10 days, so agree that the loss rate would ride as time went by.

[cantankerous swineherd]

Re: 20% drop going to manual

spending the extra to get rid of the computers, thereby improving the quality of service looks like a good move?

no more robotic emails or dumb chatbots telling you they're passionate about customer service, you'd get to talk to someone who could actually do something. back to the future.

[dubious]

Good job guys.

Don't get company values such as 'uprightness', and 'humbleness' at many other places.

I'd left the Big M group years before this, but we were well into a outsourcing and centralisation program, pushing lots jobs to India. Despite it being as unpopular with the users as these things always are, we went full steam ahead on Helldesk and desktop support, but the plans went much further. We were pushed to an expensive US company for HP-UX, Oracle, and Navis for example, which didn't seem particularly smart since Maersk had plenty of domain knowledge and was big enough to support centralised internal teams. Can't say they didn't know their stuff though.

There was also talk of centralising port infrastructure until the speed, cost, and reliability of Internet/IPLCs in many locations was brought up.

[Destroy All Monsters]

Hmmm...

the country's most popular accounting software

Wasn't it a State requirement? That sure would make it popular.

[Tom 7]

All IT installations should be called emergencies

so the managers can be used to sign cheques and kept out of the decision making process.

They "just did the work to keep disruptions to a minimum". So no progress meetings or getting the design dept in to argue over colours and fonts!

[Anonymous Coward]

This is where Infrastructure as code comes into play. If you can blow away the entire lot and stand back up an entire fresh set of machines where you have an immutable / declarative way of launching everything you could save a massive amount of time. Ok bare metal is a bit harder and getting up the base hypervisor layer if you're on-premise will take a short while, but after that IAC would save you a fair old chunk of time.

Terraform ftw, or Heat if you're an OpenStack shop.

[Doctor Syntax]

"This is where Infrastructure as code comes into play. If you can blow away the entire lot"

Would that be the entire lot as "including whatever tin the infrastructure as code was running on2?

[Peter2]

You know, it sounds horribly bad when you first think of the work requried. But then thinking about it for a few minutes you can see how it could be done relatively quickly for deploying standard builds. It'd be interesting to know how they did it.

Doing a job of this scale, personally i'd think the fastest way of doing it would be to create a new (clean) desktop image via WDS, rebuild the servers from backups and then firewall everything but WDS and AD for joining PC's to the domain. Download the image to each server, and then send somebody around each site to ensure that every PC ends up reimaged and on the network with the correct network ID.

It's a big job, but far from an impossible one (as they demonstrated by doing it in ten days, although I suspect that they had a lot of tidying up to do such as installing odd bits of random software on PC's that wasn't in the standard build.

[Joe Montana]

Firewall rules

If you allow rules for AD, then you allow the very ports that most of this malware uses to propagate.

[JimC]

Standard builds

AIUI desktops were the least of their problems. Umpteen different systems had to be rebuilt from backups, brought up again and then the data from the emergency processes merged back in. Never underestimate that, its one hell of a job.

[cantankerous swineherd]

assuming the backups weren't corrupted...

[tonyszko]

How it will help you with one things - DATA! Recovering infrastructure could be simple but what if you have lost the data in this infrastructure? Content of it? Fresh build is not giving you anything - you need to recover the previous state.

[A Non e-mouse]

Reliance on computers

But he also warned that in the near future, as automation creates near-total reliance on digital systems, human effort won't be able to help such crises.

That's why when J. Lyons and Son bought their first computer and saw how many people it could replace, they didn't go live with it until a spare was installed ready to take over.

That was back in the 50s...

[Tom 7]

Re: Reliance on computers

I think they didnt buy their computer - they builded it themselves. I think every IT person should be made to read 'A computer called Leo' so when they are wrestling with the formatting of some arseholes code in a spreadsheet they can ask themselves how the boys at the tea rooms had offices more automated than the 6000 PCs they are trying to manage, and did it on 6000 valves that wouldnt power the keyboard on the PC they're swearing at.

[jms222]

Hadn't appreciated Lyons and the LEO as much until our event at the Computing Museum. They were absolutely amazing.

[Allonymous Coward]

I'd like to have a C-level exec like this guy

Ours could learn a thing or two about recognition of huge efforts pulling business-critical systems out of the fire.

[Anonymous Coward]

It is not *the* Ukraine!

In case you need it explained.

[SlavickP]

It’s “Ukraine”, not “the Ukraine”.

[Anonymous Coward]

The internet cannot support applications that rely on it?

"Noting that the internet was not designed to support the applications that now rely on it"

What total nonsense and pseudo technical sounding waffle. The Internet performs exactly as designed, it transfers packets from one end node to another end node. The problem relies solely on the "computers" connected at either end.

What would be of interest to your readers is what was the dollar value in costs to Maersk in down time and the expenditure spent on employing people to reinstall all eight thousand plus "systems" and what indemnification did the vendors of the software provide Maersk in the event they were victims of a hacking attack.

"Snabe plans to ensure Maersk .. turn its experience into a security stance that represents competitive advantage."

What you need to do is run your "computers" off of 3½ floppy disks with the write-protected shutter in the enabled position.

[johnnyblaze]

I assume this means once they take into account the cost of the recovery and the money they lost, Maersk will look to recover some of this by cutting overheads, so IT will be the first to feel the chop. It's great to be an IT 'hero'.

[EnviableOne]

Maersk wasn't the only outfit to cop a huge NotPetya bill: pharma giant Merck was also bitten to the tune of $310m, FedEx a similar amount, while WPP and TNT were also hit but didn't detail their costs.

Hmm isnt TNT a subsiduary of FedEx

So the FedEx numbers are TNTs numbers

"2016 - On 25 May, FedEx completed the acquisition of TNT Express." from `https://www.tnt.com/corporate/history