Reply to post:

'No BS' web host Gandi lives up to half of its motto... Some customer data wiped out in storage server meltdown

Robert Morgan
Pint

Others have summed this up very well. My thoughts are - Gandi should have system level backups to get their VMs/systems back up and running again - irrespective of billing for it - those services generate revenue and being available/not being entirely lost is a big part of a hosts reputation. That alone is worth making sure system level/VM restores are available.

Customer based backups - i.e. give me file X from Y time and date is different and should be billable. Additionally, customers should backup their critical files to another provider/platform in the event they need to restore, because that's just sensible.

Either way, using ZFS arrays and relying on ZFS Snaps to restore from isn't the best strategy, ZFS is great until it goes wrong and then it goes really wrong - and that can be admin or hardware failures.

The 3-2-1 backup rule would have helped here. Three copies, two different devices/storage media types and at least one of those being offsite.

From a Gandi perspective, that might have looked like ZFS Snapshots (1), Snapshots mirrored to another independent array (2), that array rsync'd offsite to another site (3). With that, if one and two failed, they could have restored the (3) to another active array and got back up and running again.

Customers could have happily had a VM on Gandi with backup to S3/Backblaze with a lifecycle to delete old files and a copy to their homenas/dropbox/something else. That'd have given them the ability to get back up and running anywhere they could aim DNS records to.

In reality, large scale arrays for just storing large volumes of sync'd VMs to are cheap now, they don't need hugely fast disk i/o, the access pattern is sequential, so a low'ish cost box filled with 8/10TB drives gives a low cost of restore. A host "shouldn't" need it, but in times that this, spending that few K gives them another option. When getting back up and running, options are exactly what you want.

A final point would be, never make a backup plan, make a restore plan, work out what you need to restore to get backup and running, then protect the hell out of it!

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2021