Criminals go full Viking on CloudNordic, wipe all servers and customer data

CloudNordic has told customers to consider all of their data lost following a ransomware infection that encrypted the large Danish cloud provider's servers and "paralyzed CloudNordic completely," according to the IT outfit's online confession. The intrusion happened in the early-morning hours of August 18 during which …

[b0llchit]

Where are the backups?

...told customers to consider all of their data lost...

And the backups? Or,... we forgot to make any backups because our customers did not want to pay or we never tested a restore procedure or do not believe in backups?

Oh yes, it is "cloud" and all will be well and peachy. Guess it rained, then froze and the sun evaporated both ice, water, cloud and the business(es) in one go.

[wolfetone]

Re: Where are the backups?

Why is it the company's responsibility to make backups of the customer's data? It's the customer's responsibility, and any customer who doesn't believe that to be the case deserves everything they get.

[JimmyPage]

Why is it the company's responsibility to make backups of the customer's data?

Depends what they thought they were buying, really.

It's not unfair to want to offload the work and expertise required to to backups to a 3rd party, in exchange for a fee.

Indeed, almost all cloudy storage outfits make this a selling point.

[wolfetone]

Re: Why is it the company's responsibility to make backups of the customer's data?

If CloudNordic provided a back up service then yes, it's on them. But you are not meant to have just one back up for data resilience. In a physical setting you are meant to have an onsite back up, and an off site backup. If you're operating solely in the cloud you should have a back up away from the provider because it's not enough to just rely on the provider's backup.

[SVD_NL]

Re: Why is it the company's responsibility to make backups of the customer's data?

These days there's a plethora of companies offering cloud to cloud backup services. They are fully automated and tend to be rather cheap (think €2-3 per user per month for email and cloud file storage).

It's still the cloud, but what are the odds of two major cloud providers being affected at the same time?

I guess you could extend it with backups to an on-site server on a monthly or weekly basis.

People just don't realize how vulnerable the cloud can be.

[Anonymous Coward]

Re: Why is it the company's responsibility to make backups of the customer's data?

CloudNordic should take their data seriously and have a backup, just like their customers do. Apparently CloudNordic isn't as serious as any 1 of their customers so, bye bye CloudNordic.

[An_Old_Dog]

Re: Why is it the company's responsibility to make backups of the customer's data?

It's still the cloud, but what are the odds of two major cloud providers being affected at the same time?

There are just three major cloud providers: Amazon, Microsoft, and Google. Why do you presume that one of these little in-the-cloud backup companies would not be hosted on the same infrastructure as the (virtual) web- and/or email-hosting company subscribed to by the victimized companies?

(icon to remind you what can-and-has-happened [smoke-cloud-over-OVH])

[unimaginative]

Re: Why is it the company's responsibility to make backups of the customer's data?

It made me think of OVH too.

What that was a good drmonstration of was how many people do not know what they are buying. A lot of people assume cloud means everything is taken care off: redundacy, backups, updates......

[Alumoi]

Re: Why is it the company's responsibility to make backups of the customer's data?

Backup? But... but... the cloud is my backup. Or so Microsoft, Google and Apple keep telling me every freaking time they try to con me.

[Richard 12]

Re: Why is it the company's responsibility to make backups of the customer's data?

OneDrive even uses the word "backup".

It doesn't actually take a backup, though. It deletes everything from your local filesystem and moves it to Azure.

Still cleaning up that mess.

[Bitbeisser]

Re: Why is it the company's responsibility to make backups of the customer's data?

OneDrive is the spawn of evil. It is the worst of all places to have your data at. The sheer audacity of Microsoft to decide what to back up and actually MOVING all YOUR data to the cloud, removing everything from your local device is downright criminal....

[DugEBug]

Re: Why is it the company's responsibility to make backups of the customer's data?

Did you ever bother to right-click on the file/directory and select "Always keep on this device"? Then OneDrive is simply a backup/mirror. This feature is to allow you to have the same data on multiple computers/phones and only use drive space on the ones you want to have a local copy.

[Richard 12]

Re: Why is it the company's responsibility to make backups of the customer's data?

The default behaviour is to delete everything. That's not a backup.

It also instantly moves the "offline copy" of the files in the local filesystem, and doesn't even create a junction point.

Software that expects files not to be moved out from under them then crashes or corrupts, (pretty much everything).

Worse, the new local location is inaccessible to many tools as the path length becomes huge and often adds special characters like comma.

Aside from that, it took more than 48 hours to redownload them. It is incredibly slow at handling large numbers of small files, and regularly pops up a dialog "transfer failed, try again?", stopping everything until the user responds.

OneDrive is not a backup. It's active sabotage.

[Anonymous Coward]

Re: Why is it the company's responsibility to make backups of the customer's data?

Yeah, they lost THEIR backups. What happened to YOUR backups?

Really, I feel for people who got screwed by this, but this is why you need a safe backup you control, regardless of the provider claims of offers.

[Charlie Clark]

Re: Why is it the company's responsibility to make backups of the customer's data?

A lot of the data is stored only on the servers of the service provider which makes. There are all kinds of problems, including legal ones, if you try and make a regular backup of that on your own local systems.

[hedgie]

Re: Why is it the company's responsibility to make backups of the customer's data?

The people who make a lot of the decisions involved with putting "everything on the cloud" are the ones who are taken in by marketing-speak, get sold on the idea that they don't have to worry about any of those sorts of things, and make anyone who would make sure that they have mirrors and backups of their own essential data become redundant.

Backups and mirrors are things that non-experts don't think about until they need them and don't have them. I'll admit to having lost a lot of irreplaceable (but thankfully, not absolutely needed) personal data because I was stupid and didn't verify a backup before nuking the system and trying to restore everything. I've since changed platforms, but now I have a cronjob invoking Time Machine when I'm at work, and also sync everything that I can't bear to lose with online storage.

[pirxhh]

Re: Why is it the company's responsibility to make backups of the customer's data?

Yes.

My irreplaceable data lives on my PC, with an hourly backup (using restic, not a mounted file system) to a local TrueNAS. That in turn is backed up to my brother's server, about 400km away, again using restic over ssh.

All cloud stuff lives in Nextcloud on a rented VPS, with hourly backup to my server - so if the VPS is lost I will lose time and may lose a few hours of emails and such, but nothing essential.

My company, on the other side, has joined the cult of Azure. The best I can do is to manually back up anything I deem important from my laptop to a portable SSD, that I keep disconnected (and encrypted with Bitlocker to go).

[Bitbeisser]

Re: Why is it the company's responsibility to make backups of the customer's data?

Why would anyone need to make backups??? All the data is in the cloud, it is safe.

Or so the marketing lore goes.

Sorry, I have that very fight a lot with clients, with only a small number actually realizing how they could be effected when they do not have a current "offline" backup of their data.

[Filippo]

Re: Where are the backups?

The company is definitely responsible for providing the service they have been contracted for, though. They are clearly not doing it - I don't know what they're supposed to serve, but I doubt wiped servers can serve it. I suspect they are also responsible for storing the customer's data, which they are also not doing.

And the reason they are not providing the service is no (or vulnerable) backups.

The customers' customers, of course, will probably complain about CloudNordic's customers (their own service providers) not having their own backups, and they would also be right, for much the same reasons.

[Michael Strorm]

Re: Where are the backups?

> The customers' customers, of course, will probably complain about CloudNordic's customers (their own service providers) not having their own backups

It's resellers failing to back up all the way down.

[Jedit]

"their own backups as a contingency"

I may be being slightly pedantic here, but if a company is using the cloud for backups that implies there is a primary data store kept elsewhere, probably on site. If a company has lost all its data in the CloudNordic hack then it was using the cloud for primary storage. That's a serious issue, though I cannot say how much of the blame can be levelled at the company for not using the cloud as an offsite backup as was intended. Cloud providers frequently tout their services as an all-in-one storage solution with the implication that it obviates the need for on-site storage.

[John Brown (no body)]

Re: "their own backups as a contingency"

"Edit: The setup can be any combination primary/backup1/...backupn/mirror with storage local/cloud/offline/mirror - where mirror location can be local or remote depending on requirements and feasibility."

I don't know what the mix of CloudNordic customers are, but I'd bet a fair number of them would glaze over after listing to that :-)

Small and medium businesses in particular probably don't have people who understand that. They are buying in a service that they rely on, true, but they are also relying on those experts to tell them what they need to buy. Just as with plumbing, electrics, water, or anything else that's not core business and the company isn't large enough to have on-staff employees dedicated to doing. Some will have been told of the extra costs of separate backups and will have chosen to not pay for it of course, but it's CloudNordics job to make sure the customer understands what they are buying (or not buying), that's not just the good and moral thing to do, it's making sure they DO buy backup or UNDERSTAND the consequences of not doing so. That's a level of self-protection for CloudNordic

[Aitor 1]

Re: "their own backups as a contingency"

Constant backup on cloud premises is neither practical nor affordable.

The cost is huge, and makes the cloud difficult to justify.

Almost everyone that uses Amazon has the data in amazon. In another region as backup, yes, but still on amazon/azure/google.

[Roland6]

Re: "their own backups as a contingency"

So Cloud is not cheaper than on-prem what a surprise not!

If you want cheap then something ha to give.

I suspect a number of CloudNordic customers will cease trading in the coming months…

[Ken Moorhouse]

Re: ...using the cloud for primary storage.

A lot of the companies I've spoken to want to do precisely this, and their insistence is why I tend to bow out of the proceedings.

Email is included in this categorisation, but can be mitigated by having backups of it. (However, do not believe that having an email client with IMAP connection classes as a backup lol).

[Richard Pennington 1]

Re: "their own backups as a contingency"

If they are using the cloud as primary storage, then they have another vulnerability. Any loss of connectivity (e.g. the man in the JCB digging up the road outside and slicing through their cables) means they cannot see their data.

If they have primary storage held locally, they can at least continue to operate (locally) and do the reconciliation and synchronisation later when the connectivity is restored.

[Roland6]

Re: "their own backups as a contingency"

>” If they have primary storage held locally, they can at least continue to operate (locally) and do the reconciliation and synchronisation later when the connectivity is restored.”

Trouble is too many applications don’t support async operation… I suspect it’s partly because it requires a higher degree of design and execution skills than the typical application programmer possesses.

This ability to operate without a network connection and to automatically resolve the data reconciliation and synchronisation later was my first test for any application claiming to support mobile working. My second test was the ability to download to the client the core data necessary to fulfil a day’s (planned) work schedule and thus minimise the times where a field worker had to have a connection to central systems.

[pirxhh]

Re: "their own backups as a contingency"

Nowadays, there's quite a number of companies for which "on-site" is not a meaningful term at all. They don't own/rent any permanent premises, all employees are remote/traveling/working from home. In that case, cloud-first makes sense - but it requires assurances that the data will, in fact, be there when needed.

Some (too many?) believe in vendor assurances and contracts. Sure, you can sue the provider, but what good will it do? A massive hit like CloudNordic's will likely wipe out the provider, making any lawsuit not only too late to save your company but probably fruitless.

[Snake]

Re: Where are the backups?

"If a CloudNordic customer borks their own data, that's on the customer. If CloudNordic borks customer data, that's on the CloudNordic."

Not if it isn't the legaleze, it isn't. Most cloud providers provide only that - cloud access and services. It is your data and they are just handling or presenting it. Many legal documents will say that they have no responsibility for loss incurred from their operations - again, from their [legal] perspective, it is your data and they are just providing an internet-accessible container for it. If their container fails they are only liable for not providing that container, not for its contents (which are yours).

They may have provided an extra cost data redundancy service, as mine does, and if those repositories are also affected then they should be liable for that. In the meantime, my own company's hosting service ToS states:

Storage and Security. You shall be solely responsible for undertaking measures to: (1) prevent any loss or damage to your website or server content; (2) maintain independent archival and backup copies of your website or server content; and (3) ensure the security, confidentiality and integrity of all your website or server content transmitted through or stored on our servers.

so I would be SOL if I didn't have backups. It is common occurrence in today's world to expect everyone *else* to take responsibility for my things, but that's not the way it works.

[sammystag]

Victim blaming

So a customer who fails to have adequate backups deserves to have criminals attacks its IT and destroy its data? You actually think that? All the hobbyists running little niche websites in their spare time to support some community or other deserve to have it all destroyed and lose all their email because they didn't have the money, time or expertise for off site backups? Yes hardware failures can happen but they are relatively rare and the storage, one would hope, would be resilient and replicated. People make a choice whether to accept that risk or not. Doesn't mean they deserve to have criminals destroy it on purpose.

[Dave_A]

Re: Where are the backups?

Because cloud providers charge you for every MB sent outbound....

Also because it's likely to be in the SLA.

[Filippo]

Re: Where are the backups?

According to the article, the criminals managed to also get all the backups. If I understand correctly, there was an insecure server trasfer procedure that resulted in a window during which everything was connected at the same time, and the attackers exploited that. Apparently, there were no offline backups.

The cloud: someone else's computer.

[b0llchit]

Re: Where are the backups?

Then they have misunderstood the principle of backup, which is supposed to be immutable!

[Norman Nescio]

Re: Where are the backups?

Then they have misunderstood the principle of backup, which is supposed to be immutable!

That's an awful lot of BD-R disks.

Or do you mean offline rather than immutable? Tape storage is not immutable, but tends to be offline unless the tape has been loaded into a tape-drive and mounted*. The problem is ensuring that files being backed up to tape have not already been encrypted with a key you don't know (You are encrypting backups, aren't you?). There is nasty malware out there that encrypts your backups with the malware key for a while before triggering on the source data, so you find your backups are useless. Which is why a regular check to see that you can restore from backup on an independent system is important.

For me, the principles of backup include:

1) Backup data to backup media. Twice.

2) Disconnect backup media from system being backed up.

3) Attach backup media to independent system and check you can restore successfully.

4) Move successfully checked backup media to independent off-site locations.

5) Repeat frequently enough that restoral of backed up data doesn't give you a system with unusably old data on it.

Copying data from one network-attached system to another, leaving them permanently network-attached does not give you a secure backup. Verifying an offline copy of the backed-up data on an independent system is important.

*Note that automated tape silos are vulnerable to shenanigans, as are automated shingled-disk silos. Erasing one medium of the set from the previous couple of cycles of full-backups will kill the utility of the backup. Offline means - unable to access by means of programmatic control, and you need to be very sure that your backup-media inventory control system can't be compromised to 'accidentally' request operators to mount (for erasure) critical backup media.

[Richard 12]

Re: Where are the backups?

That malware behaviour is one of the reasons why your regular incremental backups need monitoring.

If the amount of incremental backup starts rising, better find out why.

[Anonymous Coward]

Re: Where are the backups?

Tape is immutable if you use WORM media. Many disk/flash based storage systems offer immutability and there are immutable cloud options.

See this article from VEEAM for backup strategies and storage options.

https://www.veeam.com/blog/immutable-backup.html

[Norman Nescio]

Re: Where are the backups?

This is IBM's implementation of a WORM tape:

IBM Documentation: 3592 Rackmount: WORM tape cartridge characteristics

Note that it is not WORM media - but various software and hardware controls are used to simulate WORM behaviour using normal tape.

'immutable' object storage is a set of software controls for access to data stored in 'the cloud'. AWS clarifies it thus: Amazon S3: Object-level immutability

S3 Object Lock blocks permanent object deletion during a customer-defined retention period so that you can enforce retention policies as an added layer of data protection or for regulatory compliance. With S3 Object Lock, S3 Versioning is automatically enabled, and these features work together to prevent locked object versions from being permanently deleted (accidental or intentional) or overwritten using a write-once-read-many (WORM) model. S3 Object Lock is the industry standard for object storage immutability for ransomware protection and is used in cloud storage, backup and data protection solutions by AWS Storage partners such as Veeam, Veritas, Rubrik, Cohesity, Commvault Clumio, and Druva.

There are further details here: Amazon AWS: Protecting data with Amazon S3 Object Lock

Note the following:

Note: The only way to delete an object under the Compliance mode before its retention date expires is to delete the associated AWS account.

I presume it is not easy to delete an AWS account, especially via programmatic means, otherwise malware purveyors would be doing it.

It looks very much like WORM and immutability are being implemented by software (and possibly hardware) controls around the use of traditional rewriteable media. It's great that somebody has thought about this, and it should make life more difficult for people who want to cause damage. Of course, the control mechanisms will come under attack, but have a degree of confidence that clever people have looked at the risk model and taken the necessary and appropriate care.

[Anonymous Coward]

Re: Where are the backups?

Its very easy to delete an AWS account.

[Confucious2]

Re: Where are the backups?

I was at a small company once and we did full grandfather/father/son backups onto tape.

It didn’t help us when some oiks walked in and nicked our server.

The director, who was supposed to take the tapes home each night and lock them in a fireproof safe, had got complacent and left the tapes on top of the server. Of course the oiks nicked them as well…

[Missing Semicolon]

Re: Where are the backups?

So, if the users paid for backups, and they are gone too, I'd be interested to see what the liability terms are in CloudNordic's Ts&Cs about consequential losses.

[Anonymous Coward]

Re: Where are the backups?

Likely 1 month free, or a refund for the current month :-)

[pirxhh]

Re: Where are the backups?

TBH, any sort of financial liability would not help the clients much.

The provider is likely to be bankrupt from an incident like this (so the final payout will be negligible), and in any case, the legal system will take too long to save the clients' business should they rely on the data.

It's a bit like insurance against a meteorite strike: They may pay your heirs a princely sum, but you'd still be dead.

[Aitor 1]

Re: Where are the backups?

It also makes it look like an insider job..

[Dave_A]

Re: Where are the backups?

Doesn't have to be an insider job if the hackers are good/patient....

Hang out inside the targets systems for a while identifying processes, weak points, and ability to pay....

Then sic ransomware on them after your recon has been completed....

Although in this case it doesn't seem to have worked out since the only reason to burn it all down is if they didn't pay....

I mean, if you get known for shooting hostages AFTER receiving ransom, NO ONE will pay you...

[Bebu]

Re: Where are the backups?

I am guessing there were before the migration there were carefully separated backup and/or archive systems but unfortunately the front door was left open to Mr Cock-Up...

Technically I imagine you might hold that the data and backups were still there (just add decryption key) although about as useful as dehydrated water.

Some good will have come from this cock-up if those, who ought to have known better, now understand the backup you don't directly control and can test, and do test, is no backup at all.

[Doctor Syntax]

Re: Where are the backups?

"now understand the backup you don't directly control and can test, and do test, is no backup at all."

Understand also that if it's not physically protected and maintained read-only until such time as it's been superseded by another backup it's also not a backup. That includes being kept read-only even while and after being restored. If it isn't read-only it's a copy but it isn't a backup.

[Peter Gathercole]

Re: Where are the backups?

It really depends on how the backup is taken. With something like TSM or Amanda, the physical media is not writeable by anything other than the server doing the backups. Sure, that still allows for the backup server to be compromised, but it's an extra level of indirection that the intruders have to understand and know how to compromise, especially if it has it's own authentication domain.

What is important is to make sure you have offline copies of the database that tracks what is on the backup media (especially in an incremental forever backup solution), as without that, you really are stuffed.

[Norman Nescio]

Re: Where are the backups?

Understand also that if it's not physically protected and maintained read-only until such time as it's been superseded by another backup it's also not a backup. That includes being kept read-only even while and after being restored. If it isn't read-only it's a copy but it isn't a backup.

While I take your point, making a backup on a mutable medium 'read-only' is not trivial. The bit of sticky-tape on the Philips cassette after you broke off the plastic tab, removing the 'write-protect' ring inserted on the hub of 9-track tape, or flipping the movable 'switch' on SD-cards makes supposedly 'read-only' media writeable again. Disk controllers that don't allow writes to attached disks are somewhat specialised. Software 'read-only' flags can be bypassed or ignored.

What you can use is independently-stored cryptographically-signed data integrity checksums to assure yourself that the data restored from a medium is the same (to a high degree of probability) as the data on the medium it was copied from. It doesn't stop the backup from being modified; but you can at least detect it. Maintaining the physical integrity and security of the backup medium then becomes the problem. There are 'immutable' backup media - such as CD-R and BD-R, but they don't scale to cloud data volumes.

'Read-only' is a proxy for data integrity, and if you are storing data on rewritable media, relying on software labels to tell you it can't have been changed is 'brave'.

Doing backups properly is hard, which is why many people and organisations don't do it properly.

[Michael Strorm]

Re: Where are the backups?

> I am guessing there were before the migration there were carefully separated backup and/or archive systems but unfortunately the front door was left open to Mr Cock-Up...

From the article:-

> Some of the machines were apparently infected before the move, and during the transfer servers that had been on separate networks were all connected to CloudNordic's internal network. This gave the intruders access to both the central administrative systems, storage, replication backup system and secondary backups, all of which they promptly encrypted for extortion.

[Peter Gathercole]

Re: Where are the backups?

Interesting that the servers just being on the same networks made them vulnerable.

I know, there's no such thing as a totally secure system, but what that probably meant was that there was a shared namespace and authentication domain, so once a certain level of access was obtained, the entire environment became vulnerable.

And what's this about the internal network? Was there just one all pervasive internal network?

Segregation of function, and of security is vital to at least slow down intruders.

[milliemoo83]

Re: Where are the backups?

"I am guessing there were before the migration there were carefully separated backup and/or archive systems but unfortunately the front door was left open to Mr Cock-Up..."

"Baldrick, you had better make the explaination you are about to give... phenominally good."

[Lars]

Re: Where are the backups?

I don't think this article included all information, and the backup part is certainly important.

While I have no experience of clouds I must admit i have assumed a cloud provider always have backups.

[Graham Cobb]

Re: Where are the backups?

It''s not just about backups because of hardware faults, fat-fingers, etc. In the case where you are using any cloud services, that provider could stop working at any second, without warning, for no apparent reason!

This case is one way that could happen but there are many others. The most likely, in my opinion, is a commercial issue: the company collapses, without warning to any customers, and ceases trading instantly. All data instantly inacessible - including any backups they hold.

For that reason, it is essential that if you use a cloud service, you have a disaster recovery plan which handles the cloud provider effectively disappearing into a wisp without warning.

Most importantly, if you contract with company A to run a service for you, make sure that you contract with a different company for the DR backups.

[Richard Pennington 1]

Re: Where are the backups?

Another possibility is the MegaUpload vulnerability. When the Feds seized the MegaUpload server in New Zealand, they wiped out as "collateral damage" all the innocent customers whose data was on the same server.

[Alumoi]

Re: Where are the backups?

Backup of their files? Maybe. Backup of your files? Who cares, you already paid to have your files held for ransom... erm, montlhy payment.

[teknopaul]

Re: Where are the backups?

Read the article.

It states clearly that backups were encrypted with ransomware.

[Doctor Syntax]

Re: Where are the backups?

A backup is a copy that's held off-line and transferred off-site or to some physically protected storage ASAP. It's there as a last resort to protect against as many possible failures as possible, If it remains online it's a copy but it isn't a backup. Once taken it should be write-protected so that even if connected to a compromised system it won't itself get compromised.

[Orv]

Re: Where are the backups?

By that definition practically no one does backups anymore. You're talking about hiring someone full time just to sling tapes around, and tape sizes have not kept up with disk sizes, so it's going to be a LOT of tapes.

[seven of five]

Re: Where are the backups?

We (and many others) actually have bought a slave just to handle the slinging around of tapes. We tend to call them "library". But yes, tape size has not kept up with disk size, a large (enterprise) drive is around 15-20TB capacity, while tape is only about the same. Uncompressed, which tape drives can do better than disk controllers.

[Orv]

Re: Where are the backups?

Sure, but if you're just leaving them in a tape library they're not a "backup" by Doctor Syntax's definition, since they could be loaded and written to at any time. To meet their full definition the tapes have to be removed, write protected, and then hauled to another site.

[Peter Gathercole]

Re: Where are the backups?

Different types of backup protect against different things.

Taking data offsite mainly gives extra protection from physical destruction of the environment, but you can achieve this with cross-site copying of data, and if you have sufficient access control, this can be relatively safe.

The secret to protecting against the type of problem described here is mainly a problem of access to the media. If you use an additional writeable storage accessible by the systems that were compromised for your 'backup' (or, in fact, by any other compromised system), then this is in reality not a backup but a copy of the data. This is vulnerable to this type of attack (and is, sadly, how many modern environments believe they have a backup solution).

In a properly designed enterprise backup solution, the systems at the front end using the data have no access to the backup media, and the systems at the back end which run the backups and do have access are heavily protected, in at least a different authentication domain. And the route between them is guarded to restrict one getting to the other except to transfer the data.

I do think that in this case, during a physical move of the infrastructure, I would probably have suggested a full offline copy of the data, transferred between the sites using different transport, just as a fall-back.

[Zippy´s Sausage Factory]

Re: Where are the backups?

A lot of cloud services "back up" by just using ftp to copy across to another server in the same room.

Another good reason to avoid cloud services, if you ask me. Because you know every cloud service - even the big players like AWS and Google Cloud - are going to be doing it this way.

[Anonymous Coward]

Re: Where are the backups?

No no no

They've moved beyond that. FTP is old tech, they use rsync now.

And the other server is sharing the same SAN to maximise efficiency.

[Anonymous Coward]

Re: Where are the backups?

Had this.

Management were sold that SAN server was so resilient that loss of data was near on impossible...

... until a total loss of the SAN server and all the data happened. And all the systems, including the management infrastructure that installed the other servers, and the server doing offline backups were also on the same SAN... (it wasn't my design!)

Building the environment back from the ground up using first principals just to get to the point where you can use the tape backups is no fun. Fortunately, once the tape backup server was rebuilt, the tape library became accessible, it then became a lot easier. Still had a week of downtime though (although partly because of a bug in the restore process of the tape storage manager software).

In my ideal world, the server managing the tape backups, and the image deployment server should always be on separate infrastructure (not just server hardware, but storage hardware as well) from the rest of the environment, each with their own re-installable, offline backups taken regularly. This way you can get the deployment server and the backups available quickly (if it's even necessary), and then move on to the rest of the environment.

[Aitor 1]

Re: Where are the backups?

Most of my big customers always had the backup network port and network, plus servers. Everything independent.

But it was expensive.

So they recreated that as virtual appliances.

Still expensive.

So they "went to the cloud" as it seemed less expensive. And threw caution to the wind.

Now it is more expensive than the initial situation and less resilient, but they are trapped.

[John Brown (no body)]

Re: Where are the backups?

"And the other server is sharing the same SAN to maximise efficiency."

...and enables SAN-wide de-dupe to save space.

[Peter Gathercole]

Re: Where are the backups?

I have always been a bit sceptical about de-duplication at the SAN level.

Sure, it you're rolling out multiple system images at the same level, many of the files on the systems will be identical, and these are in scope for being de-duplicated. But for most environments, the size of the OS, large as it may have grown, is insignificant compared to the dataset sizes that these systems now manage. The large Oracle systems I have some involvement in have something like 20GB set aside for the OS components, and 50TB for the data, three orders of magnitude more than the OS.

But as soon as you get any client side encryption involved, or you're handling unique data, especially if using, say, a SAN server as a first stage storage pool in an encrypted backup media hierarchy, the chances of actually getting duplicated data are essentially zero. And you are going to want to keep 2 copies, or have a good RAID setup, to protect against physical media failure.

I would love to see a rigorous independent study of de-duplicated storage systems (with a description of the environments they are serving) to see how much space has actually been saved, and then compare this to the cost, both monetary, processor usage and power that has been expended to achieve the de-duplication. If anybody has links to studies like this, I would be very interested in reading them, because I'm prepared to have my mind changed.

[John Brown (no body)]

Re: Where are the backups?

It was a tongue-in-cheek comment re storing the "backup" on the same SAN as the original data. Unless the backup is separately encrypted and is probably just using something like rsync "on the cheap" copying, then the backup will be identical to the original and de-duped out of existence :-)

[sketharaman]

Where are the backups?

As the article says, backups also got hit by ransomware when its servers were being moved from one datacenter to another: "Some of the machines were apparently infected before the move, and during the transfer servers that had been on separate networks were all connected to CloudNordic's internal network. This gave the intruders access to both the central administrative systems, storage, replication backup system and secondary backups, all of which they promptly encrypted for extortion."

[John Robson]

Re: Where are the backups?

If you'd read the article you'd know

[Dave_A]

Re: Where are the backups?

I would guess they had backups, but to a NAS or something similar that's networked with the rest of their systems and read-write accessible....

Where were the offline, off-site backups....

[jmch]

Offline backups??

"This gave the intruders access to both the central administrative systems, storage, replication backup system and secondary backups, all of which they promptly encrypted for extortion."

So they had multiple backup systems, but all of them were online???

[Jon 37]

Re: Offline backups??

This is sadly common. It makes backups easier and faster. It makes restoring from backup easier and faster.

It also means that your backups offer no protection against ransomware or a hacker.

[Cliffwilliams44]

Re: Offline backups??

It does if no single credential has access to every backup. You certainly should not be using some form of long term access keys to access those backups. It should be an account with MFA.

[mattaw2001]

Re: Offline backups??

I wonder if they made the mistake of confusing access to the backups with access to the unencrypted content of the backups.

It's easy to assume that because your backup files are encrypted they're safe, but they're only "safe" against being read, not destroyed or in this case encrypted again!

[hertz]

Re: Offline backups??

It appears that yes, all of them were online or at least all networked with internal systems during the move because no one could apparently forsee that could cause problems.

[Doctor Syntax]

Re: Offline backups??

"So they had multiple backup systems, but all of them were online?"

In realiy that makes them multiple copies but not backups as they have now discovered.

[Aitor 1]

Re: Offline backups??

That is the standard now.

On different vlans ordifferent virtual environments.

But looks like the miscreants still had c&c servers abke to access everything or it was a time bomb

[Pascal Monett]

So, another cloud company that screws it up for all of its customers

Hey, CloudNordic, if I was one of your customers I wouldn't be worrying about getting my site back online with your help.

I'd be getting it back online with the help of a different provider.

You screw up in that magnitude, I vote with my wallet.

[fg_swe]

And Then ?

Do you seriously believe other providers are immune, by means of magic ?

Just a few days ago one of the megaCloud providers had their "master key" stolen, which meant ALL servers could be read and changed.

A few years ago a smaller, but non-trivial cloud provider had their "management console" hacked and essentially ALL servers in the open.

The entire "cloud" idea looks questionable.

[Anonymous Coward]

Re: And Then ?

Cloud services are increasingly commoditized and compete on price. Offline backups involving LTO tape libraries or hard drives will be one of the first things the bean counters will cut.

[Anonymous Coward]

Re: And Then ?

No no no. Backups to tape will be offered as 'archive storage'!

[fg_swe]

All Eggs in One Basket

Well, having a "master key for all machines" is effectively the same problem as the one in this affair, isn't it ? Same with the "Master Console controlling all servers".

Efficiency is not the same as security...

[Doctor Syntax]

Re: All Eggs in One Basket

"Efficiency is not the same as security..."

Even worse is mistaking convenience for efficiency.

[Cliffwilliams44]

Re: All Eggs in One Basket

And what credentials did the black hats use? Obviously one that had full access to everything. Why was there not MFA on those credentials? I'll venture a guess, because they were being used by multiple people and that would be "inconvenient"!

[Doctor Syntax]

Re: And Then ?

"Unfortunately? Try negligently."

I'd say stupidly.

[Anonymous Coward]

Re: So, another cloud company that screws it up for all of its customers

They have already commented in Danish media, that while they don't expect the company survives, they are standing by to provide whatever support they can.

They aren't foolish enough to think the company will survive.

[Mike 137]

Segmentation, anyone?

"Some of the machines were apparently infected before the move, and during the transfer servers that had been on separate networks were all connected to CloudNordic's internal network."

Quite apart from the $64k question -- why the infected machines were not detected before or during the move, did anyone consider network security at all or was the 'internal network' effectively a giant hub? And if the machines were originally segregated (presumably for good reason), why was this not maintained?

[Anonymous Anti-ANC South African Coward]

Ouch, surely this must count as a very expensive lesson for both customer and service provider...

We do use online backups (quicker backups and restores) but we do store our data on offline media as well, but financial data is also stored online, offline and in the cloud.

Gone are the days when a simple backup to tape was good enough.

[fg_swe]

"Simple Tape"

Also sounds risky. What happens when the building burns down ? Even without cloud, there should be at least two physical storage places for you tapes.

[Orv]

Re: "Simple Tape"

Back in the 1990s, I used to work at a bank that dealt with this by having the sysadmin put the previous week's backup tape set in the trunk of his car every Friday. This did technically create off-site backups, but it always seemed to me that there were security implications. It also didn't do those fragile DDS tapes much good.

[Anonymous Coward]

Re: "Simple Tape"

Yeah, old style backups on manually extracted/rotated tapes.

Until you discover said DDS tapes are actually no longer extracted, or as I discovered once: there has only been a single poor DDS tape for the last 2 years !

The poor thing was probably no longer holding anymore useable data ! I was as shit when I discovered this !

[Orv]

Re: "Simple Tape"

Our DDS drive seemed to need cleaning almost constantly; I'm boggling at the idea that someone could just leave one tape in the thing for 2 years.

[Cliffwilliams44]

Re: "Simple Tape"

I worked for a company where the IT manager did the same thing. This is Florida where it can reach 100+ degrees in the summer, 140+ degrees inside a car! Needless to say those tapes were useless!

[Doctor Syntax]

Everyone looking after data needs to realise one thing: the entire worth of a business will reside in its data. If it's lost the business will be extremely lucky to get back on its feet again. If it's gone it's gone.

Anyone entrusted with the task - and that includes everyone from tape jockeys to CEOs making strategic decisions and beancounters controlling the budgets, businesses looking after their own data to businesses looking after other peoples' data - should be paranoid about it.

[Stuart Castle]

I find it's best to assume that unless a cloud provider actively pushes that they back up your data, it's probably best to assume they don't, and maintain your own backup. Even if that backup is just the same thing on someone else's cloud.

[Yet Another Anonymous coward]

Blaming Vikings

There is absolutely no evidence that iron age Northern European seafarers were responsible.

Vikings have always been big fans of IT services. In fact Loki, god if fsck-ups, is the patron saint of IT.

[The Vociferous Time Waster]

Design for failure

Organisations using third parties should always design for failure and assume stuff like this can and indeed will happen. All your eggs in a single vendor basket is a recipe for a CIO on the chopping block.

[AVR]

Re: Design for failure

No, no, the CIO's job is to manage people in one area of the business. Only very rarely will any responsibility for design problems reach up to the CIO.

[Norman Nescio]

Re: Design for failure

Organisations using third parties should always design for failure and assume stuff like this can and indeed will happen. All your eggs in a single vendor basket is a recipe for a CIO on the chopping block.

Well, yes, but...

Organisations that do things properly have higher expenditures. Things like having a proper disaster recovery plan/business continuity process cost money, as spare data centres or cloud offerings are not cheap, and require resources to manage. Less scrupulous organisations undercut you on price while offering similar looking (or identical) service guarantees to the customers you are competing for. Their 'guarantees' are, of course, worthless, but customers find out only after the fact. There are always ways in which charlatans can undercut you on price by skimping on infrastructure. It's a risk/reward equation - if they can get away without a proper backup service for long enough to put you out of business, then that is what they will do. If they have an 'unfortunate event', well, the directors shut down the company and go to another startup and blame 'unforeseeable events' for their failure on their CVs. They might even believe it is true.

Your competitors are rewarded for taking risks that you might deem to be unreasonable. It's gambling, pure and simple, and some people become undeserved winners.

It's actually a systemic problem generated by having limited liability companies, and lack of personal liability for directors. When you are gambling with other people's money and livelihoods, it matters less. It's the way current capitalism works, which, to be fair, has generated some major successes on the way, You make money by taking risks. You make more money by using other people's money to take risks with. If you fail, limited liability allows you to walk away and try again.

Some companies work hard to manage business risks and allocate investments in anything (including infrastructure) accordingly, but they tend to be large and boring as it requires a fair amount of resources to do it properly, and even the best of them can get it wrong when new technology is involved.

[Ken Hagan]

Re: Design for failure

I wonder how many customers have a plan B for when Microsoft 365 goes belly-up (if only for a week or so).

Given that they apparently lost a fairly significant signing key a few weeks ago, and given that several significant nation state actors currently feel themselves to be at war with the US, this possibility *ought* to be on everyone's radar, but I fear that most MS customers probably couldn't make a backup of their own even if they wanted to.

[Throatwarbler Mangrove]

Ironic

"Once you pay the danegeld, you will never get rid of the Dane."

[Yet Another Anonymous coward]

Re: Ironic

Speaking as a former inhabitant of the Danelaw, and a former Eu citizen, can I pay to have the Danes in charge again?

[Anonymous Coward]

"We have seen no evidence of a data breach"

Wait, is letting the criminals leak your data a data recovery strategy?

[Ribfeast]

Re: "We have seen no evidence of a data breach"

Unintentional cloud backup :)

[Anonymous Coward]

Re: Wait, is letting the criminals leak your data a data recovery strategy?

No, monitoring the network to see what volume of data is flowing through it is what made them say "no evidence of a data breach". It's in the article....

[Anonymous Coward]

The Cloud

Other people's computers you have no control over

[Reginald O.]

Re: The Cloud

Yup. And even on a good day you have no clue who is rummaging through the data copy, pasting, slicing, dicing, selling and sharing the data completely without any clue at all.

[Anonymous Coward]

Re: Other people's computers you have no control over

No control over? What, none at all? No way for me to ask the cloud to store something, do something, or delete something?

What the actual fuck are you talking about?

Did you mean to post "Other people's computers you have limited control over"?

[Reginald O.]

So, offline backups aren't a thing?

I suppose it's possible any backups were infected long before the demands were made. Doesn't anyone read logs anymore?

[Piro]

Immutable backups? Immutable storage snapshots? Offline backups?

My guess is there's a backup guy who was constantly telling the bosses that investment needed to be made, only to be shot down.

Sad. Listen to your backup guy.

[sabroni]

Great stuff guys!! - bogbrush

[Cliffwilliams44]

God Credentials

So, the miscreants, got a hold of credentials that had access to EVERYTHING! Why is that? Why is there some GOD credentials that someone in the company apparently has daily access to? Did these credentials not have MFA?

Yes we have root credentials for your cloud accounts, no one uses them, no one remembers the password, that is kept in a safe, along with a hardware MFA device.

Yes I know, security is inconvenient, IT'S SUPPOSED TO BE!

Did the companies financial system get whacked? This company should file bankruptcy and shutter their doors as they are obviously too incompetent to stay in this business.

[DugEBug]

Re: God Credentials

"Why is there some GOD credentials that someone in the company apparently has daily access to? "

Which leads one to wonder if this was an inside job.

[Ken Moorhouse]

Cloud Provider's Backup Responsibility

I've heard stories where two people in the same office see two different versions of documents freshly downloaded for editing from the cloud, presumably because of replication issues at the cloud end. Hopefully such differences are fleeting in nature, but could be longer if internal links within the cloud are degraded. Backing up from a point of presence to somewhere (another cloud service, or on-premise), might therefore not be as deterministic as one might think. The cloud provider must necessarily be aware which is the most recent version of a document, because it needs to store it uniquely as such.

With databases, there is the additional need to pause execution to take the backup, as there may be integrity issues at stake if a transaction is in course of execution (in short: ACID) and is captured in mid execution. It is presumably much more efficient for the cloud host to determine at what point the backup should be executed.

These two factors point to the Cloud Provider taking on the necessary responsibility for backups, or to spell out how to ensure the latest version is captured. (It would be extremely bad practice for a cloud provider to allow a database to be backed up in mid-execution).