'No BS' web host Gandi lives up to half of its motto... Some customer data wiped out in storage server meltdown

Customers of web hosting outfit Gandi.net have been left less than impressed by its handling of a data-destroying storage crash. The France-based hosting provider on Thursday disclosed it had lost some customer data after a ZFS storage box in Luxembourg broke down and had to be replaced using a backup. Efforts to restore the …

[YourNameHere]

Partial Credit?

But we restored our platform and we will not bill you while you rebuild your site from scratch and while you ask your customers for a copy of their latest bill you sent them last month. I am sure they will tell you what they owe you. Do we get at least partial credit for that??? Maybe if you go look in the way back machine, you can find a copy to start from... I don't understand why your so mad, it could happen to anyone...

[Anonymous Coward]

Can I be the first to say...

Can I be the first to say: the cloud is just someone else's computer. But clearly not someone else's backup...

[Bendacious]

It could happen to any other web host, who have also never tested their Disaster Recovery plan. I've worked for companies who never test their backups because no one wants that job and it's time consuming but for a web host to do that I'd consider a serious failure of duty. Personally the SQL servers I'm responsible for tell me that their daily backups are verified but I don't trust them and do test restores at least a couple of times a year. The current company I work (just a typical technology SME) does a full cut-over to a mirrored DR site for essential services twice a year. Except for Exchange, which is apparently too complicated, or the Exchange team are lazy, so we are trusting the untested DR plan for email when the meteor hits the server room but at least we don't provide email services to customers.

[J. Cook]

If it's an Exchange DAG, performing a full cut-over may cause a disruption or outage, depending on several factors. (How the DAG is configured, how your DR/ Business continuity testing is configured, etc.)

It's not necessarily that your exchange team is lazy, it may more be a case of it's a giant pain in the nethers to perform.

In my case at [RedactedCo], we have a 4 node DAG (2 nodes in prod, 2 in DR), with a witness server and an alternate witness at our DR site, which is inline with the minimum for the Preferred/Reference architecture. Cutting over is a semi-automatic operation, takes ~15-20 minutes, and is NOT transparent to the business. Failing back or ending the test is about as painful and non-transparent to the business.

[Anonymous Coward]

No load balancers sitting in front of your internal Exchange?

Having done similar tests, removing back-end servers from the load balancer is used for patching/DR testing. Admittedly we don't perform a full shut down of the load balancers in order to avoid disruption but we do monitor load balancer connections as part of the test to see what is going through them and decide whether systems have failed over successfully based on that information to avoid surprises. It resulted in a lot of misconfigurations being found and corrected.

To be clear, we would probably see some disruption in a full failover across all of our estate but have a fairly good idea what will give us problems based on our tests and how to fix them....some apps are just "special"

[The Oncoming Scorn]

No BS

Clearly meant No Backup Solution\Storage\Strategy.

[The Oncoming Scorn]

TITSUP

Total In-ability To Safeguard Users Platforms

[Phil Endecott]

I was only using them for domain registrations, but this is still pretty unimpressive especially the “could happen to anyone” bit.

Can anyone recommend any alternatives?

[AdamWill]

ditto - I like their web UI and their early adoption of 2FA and dnssec support and stuff, but this is pretty poor. Also interested in recs.

[Anonymous Coward]

> but this is pretty poor.

It is a hardware failure. Don't tell me you don't plan for that.

> Also interested in recs.

What do you need? Data integrity or availability?

In the first case, have a properly designed and well-exercised data back up and recovery plan¹, or pay someone to do it, but it is *your* data after all.

In the second case, redundant servers with automatic failover is probably a good starting point.

Combine both as needed if necessary to meet your needs.

If you are using shared hosting (as I do: Gandi in fact), consider it a best-effort resource that can fail at any time in a number of more or less spectacular ways. If using dedicated hosting, make sure to go very carefully through your service level agreement.

What I look for in a hosting provider is a) that they don't fail too often or in too mysterious ways and b) that wenn die Scheiße den Fan trifft they are at hand to help sort things out.

Right now, I'd recommend Gandi. Once bitten twice shy, etc. :-)

¹ For the data that needs to be preserved. Not all of it does, usually.

[Ragarath]

Nice to see a Gandi employee backing up the company line.

IMO it all depends on IF they sold this as a backed up solution. If they did, fault is on them if not then yes you are welcome to have a go at the users of the service for not having their own backup.

[Anonymous Coward]

> Nice to see a Gandi employee backing up the company line.

There is always a tinfoil hat wearing retard that's going to come up with an asinine comment like that. It states clearly in my comment that I am a user.

As a matter of fact, a dead smug user right now, because even though my servers were not affected I actually do have a well conceived and regularly practised backup and recovery plan. Which, if one were to judge by the "reporting" and by the comments on this site, you would think it's a patentable novelty.

Even so, the previous comment was meant to be a helpful answer to an in principle legitimate question ("what can I do to minimise the risk of loss?")

The hosting contract (which one is required to accept when buying or renewing) states in § 3.3 that you are responsible to back up your own data. Here's an excerpt, bolded and outlined in the original:

"The backing up of Your content and technical settings shall be Your sole responsibility. You must take all the necessary steps to ensure regular backups and protection of Your Content and all Your data and configurations, as well as their updates, on media outside Gandi."

This is the same as every other hosting provider that I'm aware of, which is why I can't bring myself to have any sympathy for anyone who ventures into the somewhat technical land of shared hosting not being aware of this or just simply counting on their luck.

[Phil Endecott]

That’s all completely irrelevant; you’re replying to my post, and AdamWil’s reply to that, where we say that we are using Gandi ONLY for domain registration. Stuff about backups really doesn’t apply.

[Phil Endecott]

> What do you need? Data integrity or availability?

The posts you’re replying to are about domain registration.

[Anonymous Coward]

This incident is about shared hosting.

The two are completely different beasts so I don't follow you. What sort of recommendations do you want then?

[AdamWill]

Uh, I think you misunderstood. Neither I nor the person I replied to is using Gandi for hosting, or asking for advice on hosting.

[phuzz]

"It is a hardware failure. Don't tell me you don't plan for that."

Depends if I'm paying another company to specifically take care of that for me.

[Doctor Syntax]

"Can anyone recommend any alternatives?"

I only use them for domain and email: Mythic Beasts.

[Anonymous Coward]

Porkbun is bloody brilliant

[Anonymous Coward]

'could happen to anyone' refers to suffering a hardware failure, which can and does happen to anyone.

However they're stretching this pretty thin to try to explain away their entirely separate failure to test their backups or have any other plan B which should mitigate a hardware failure.

[Anonymous Coward]

> However they're stretching this pretty thin to try to explain away their entirely separate failure to test their backups

There aren't any backups. When you buy shared hosting, it does not include backups unless you explicitly contract such a service (if available, not everyone offers that) and pay for it. I wish this article was a bit more clear about that industry fact that I would have thought rather obvious but clearly a lot of people seem to think that because they're buying space on a disk someone else is going to take responsibility for what they put in it.

[Anonymous Coward]

especially the “could happen to anyone” bit.

Should read as "will happen to everyone, one day".

[Anonymous Coward]

Talking about bullshit

This is exactly the sort of bullshit "journalism" that's certainly not needed. Complete with Twatterati comment and all.

Yes, it is regrettable but these things do happen with certain regularity.

And as can be seen from their status page, it is not quite clear yet whether data may be recoverable or not: https://status.gandi.net/timeline/events/2109

Name one shared hosting company that backs up their filers. And their prices, please.

This is what I take Gandi to mean by "could have happened to anyone". Perhaps you prefer the canned "a small number of our customers were affected" PR crap. I don't.

Gandi customer for well over a decade. They have always been very decent I haven't found anyone else equally competitive.

[Phil O'Sophical]

Re: Talking about bullshit

I haven't found anyone else equally competitive.

It's easy to compete on price if you skimp on quality and reliability.

[Anonymous Coward]

Re: Talking about bullshit

> It's easy to compete on price if you skimp on quality and reliability.

I wasn't talking about price or I would have said cheap instead.

Please name a more competitive provider, along with your explanation why. I am sure some people will be genuinely interested.

[Anonymous Coward]

Re: Talking about bullshit

> Please name a more competitive provider, along with your explanation why. I am sure some people will be genuinely interested.

So? Anyone? One provider that includes full remote backups at a similar price, in Europe? Go on, just one.

[teknopaul]

Re: Talking about bullshit

atlantic.net and linode.de both provide backups. I have not tried any others but I don't think I would put data on a system that does not even claim to run backups.

[teknopaul]

Re: Talking about bullshit

FYI Atlantic just confirmed that backups are onsite by default but can be offsite they have multiple regions. I wonder if there is anyone apart from Gandi who does not do backups?

[teknopaul]

Re: Talking about bullshit

OVH, not that I would recommend them, but they do provide backups.

[teknopaul]

Re: Talking about bullshit

hetzner do as well

[Phil O'Sophical]

Re: Talking about bullshit

I wasn't talking about price or I would have said cheap instead.

"Competitive" is generally taken in terms of "value for money", so price is a part of it. It doesn't have to mean 'cheap', you can be underfunded without being cheap.

Please name a more competitive provider

No idea, I avoid hosting providers for exactly this reason. I run my own hosts, although I do use a provider for personal email. I make my own backups, since their SLA is clear that they don't backup my user data. I do expect them, though, to backup their own data including configuration related to the service they sell me, so that they can have their hosts up again ASAP after a failure.

[Phil Endecott]

Re: Talking about bullshit

AWS Lightsail makes an interesting point of comparison.

One observation is that AWS is probably one of the few places where, due to their sheer scale, hardware failures like borked disks must happen quite regularly. So whereas a small provider will have a theoretical disaster recovery plan that you hope they have rehearsed, AWS will have a routine and frequently-used process to deal with failures.

The main storage on a Lightsail instance is EBS, which is replicated within the same “availability zone”. So it should be resilient to a single disk failure, similar to how RAID is. You can then configure it to make incremental snapshots on whatever schedule you need, and these are distributed around the different availability zones in the same region.

Pricewise there are various options; compare Gandi’s “M” plan (120 GB bandwidth, 20 GB storage, about €10/month IIUC) with Lightsail’s $10/month plan : 3 TB bandwidth, 60 GB storage. You can have that in London, Dublin or a load of other places.

Lightsail’s (and AWS’s) weak point, in my experience, is the support; it seems to use Mechanical Turk and/or chatbots, rather than actual support humans. But compared to my experience with Gandi (and another European provider, Scaleway), things generally don’t go wrong so you don’t need support.

Of course this is an apples-and-oranges comparison in many ways.

[katrinab]

Re: Talking about bullshit

Dear Gandi spokesperson,

Maybe if you spent more money in a technology called a "backup"[1], and less on astroturfing PR bots, you wouldn't have this problem.

[1] https://en.wikipedia.org/wiki/Backup

[Anonymous Coward]

Re: Talking about bullshit

"Catrina", those comments make you look pretty stupid. I make no apologies for laughing at morons who can't take responsibility for their own data and I despise people who cannot accept blame for their own mistakes.

Entitled crybabies like that are nothing but a waste of this world's limited resources.

[katrinab]

Re: Talking about bullshit

So presumably you despise yourself given that you evidently think that "responsibility" only applies to other people, and not the company you work for?

[Anonymous Coward]

Passing the blame

That guy Andrea is a bit of a cretin. "We move out today" (/rolls eyes)

What was *his* recovery plan, I wonder?

As someone else said, the "cloud" is someone else's computer. Which is fine, as long as you keep that in mind and proceed accordingly. Hoping that your inexpensive shared hosting is somehow going to be immune to hardware failure is not really my idea of proper prior planning.

[Anonymous Coward]

I am a bit concerned

Looking at the comments here, that people seem to be under the impression that hosting providers systematically keep back ups of customer data.

That is not the case, chaps. For a start it'd be uneconomical and uncompetitive; you'd be paying for the back ups whether you actually needed them or not¹. Then there is that whole box of worms with GDPR and similar laws, where if you do not control your back ups you might end up in a position where you cannot actually comply with legal or industry regulations concerning data rights, security, etc.

¹ For instance, I have used shared hosting to process one terabyte of data twice a day. The data is obsolete after about twelve hours and expensive enough as it is, without having to pay for it twice.

[Anonymous Coward]

Re: I am a bit concerned

> Looking at the comments here, that people seem to be under the impression that hosting providers systematically keep back ups of customer data.

No one expects them to keep backups in the sense of of "Can you retrieve file X.DAT as it was two weeks ago" but people do expect them to be able to restore their systems to operational order within the stated SLA RTO and RPO. That is what seems to have failed here.

[Anonymous Coward]

Re: I am a bit concerned

> people do expect them to be able to restore their systems to operational order within the stated SLA RTO and RPO. That is what seems to have failed here.

Precisely!

Can't see what part of their SSCH CSC they are failing to meet. Section 3.3 clearly states the backup policy too.

[Doctor Syntax]

Re: I am a bit concerned

"For instance, I have used shared hosting to process one terabyte of data twice a day."

That may not be a use case typical of most of their customers.

[Anonymous Coward]

Re: I am a bit concerned

> That may not be a use case typical of most of their customers.

I don't know what their typical use cases are, but I trust people to know where the boundaries of responsibility lie.

[Intractable Potsherd]

Re: I am a bit concerned

You'd be more credible if you weren't posting anonymously. A good rule of thumb is that AC+ad hominem = troll, and you are ticking all three boxes.

[dnicholas]

Backups in 2020 are still important

The Cloud: Just Someone Else's Shitty Computer

[yoganmahew]

Re: Backups in 2020 are still important

The cloud: still you own data and your own processes. PAAS, the hint is in the P... it's not a managed service, it's a managed platform.

[Anonymous Coward]

Re: Backups in 2020 are still important

it's not a managed service, it's a managed platform.

and in this case, not a very well managed one since it all seems to have evaporated. Just like a cloud on a sunny day.

[Tom 38]

Re: Backups in 2020 are still important

Or just like any other host, including self-hosting. In this day and age, you should expect your servers and the data you've stored on them to disappear, and be able to handle the loss of servers and/or data without losing service or customer data.

[Anonymous Coward]

sad

they could have avoided this by having snapshots and storing them on 2-3 other servers that aren't the production server.

[Anonymous Coward]

Re: sad

> they could have avoided this by having snapshots and storing them on 2-3 other servers that aren't the production server.

That is almost exactly what I do as part of my recovery strategy. There is the option of doing that with Gandi, it's called "active repulsion" or some such.

The advantage is that it's very easy to spin a new server up if the old one dies but at least part of their infrastructure is still available. Don't forget to do the same to your OS disks, as well as data, as reconfiguring a new server is easily the most time consuming (and error-prone) part.

If your provider doesn't have a similar option, you can simulate it by snapshotting the data and copying it to another provider and/or downloading to your own infrastructure.

In any case, if you are on the "cloud" and you only have one copy of your non-disposable data, you are doing something seriously wrong.

[Robert Morgan]

Others have summed this up very well. My thoughts are - Gandi should have system level backups to get their VMs/systems back up and running again - irrespective of billing for it - those services generate revenue and being available/not being entirely lost is a big part of a hosts reputation. That alone is worth making sure system level/VM restores are available.

Customer based backups - i.e. give me file X from Y time and date is different and should be billable. Additionally, customers should backup their critical files to another provider/platform in the event they need to restore, because that's just sensible.

Either way, using ZFS arrays and relying on ZFS Snaps to restore from isn't the best strategy, ZFS is great until it goes wrong and then it goes really wrong - and that can be admin or hardware failures.

The 3-2-1 backup rule would have helped here. Three copies, two different devices/storage media types and at least one of those being offsite.

From a Gandi perspective, that might have looked like ZFS Snapshots (1), Snapshots mirrored to another independent array (2), that array rsync'd offsite to another site (3). With that, if one and two failed, they could have restored the (3) to another active array and got back up and running again.

Customers could have happily had a VM on Gandi with backup to S3/Backblaze with a lifecycle to delete old files and a copy to their homenas/dropbox/something else. That'd have given them the ability to get back up and running anywhere they could aim DNS records to.

In reality, large scale arrays for just storing large volumes of sync'd VMs to are cheap now, they don't need hugely fast disk i/o, the access pattern is sequential, so a low'ish cost box filled with 8/10TB drives gives a low cost of restore. A host "shouldn't" need it, but in times that this, spending that few K gives them another option. When getting back up and running, options are exactly what you want.

A final point would be, never make a backup plan, make a restore plan, work out what you need to restore to get backup and running, then protect the hell out of it!

[Anonymous Coward]

Good summary above.

> Gandi should have system level backups to get their VMs/systems back up and running again

Note that the VMs did not go down. It's just a storage unit that crashed. From talking to an affected user, their VM is still running happily, except that it cannot "see" the data disk.

> Either way, using ZFS arrays and relying on ZFS Snaps to restore from isn't the best strategy, ZFS is great until it goes wrong and then it goes really wrong

I have no experience with ZFS but from their communications, that is indeed the problem they're facing at the moment with attempting data recovery.

Agreed with the rest of your points, especially good point about the restore plan. It is like in aviation, where landing is your plan B; your plan A is going around. It puts you in the right mindset.

[katrinab]

zfs is a file system, not a backup system. It guarantees that any data you read from the disk will be exactly the same as what you (or a virus etc) wrote to it. It does not guarantee that you will get it back. Other file systems will give you corrupted data in situations where zfs will know it has been destroyed and not give you anything.

[Anonymous South African Coward]

Somebody asked me why I don't back my company data up to the cloud.

This is one reason why. Imagine you'd get hit by crypto-malware, and just before you're ready to start restoring data, your backups goes TITSUP.

Second reason is - I want full control over my backups. Who have access to it, where it is, and how old it is.

And the third - $$$. It works out cheaper long-term to store it yourselves than having it out on somebody else's purdy compoota (or RAID array).

[katrinab]

What's your recovery strategy in the event of a fire/flood at your location?

Cloud can be a useful component of your backup strategy because it is in a different location to you.

[Anonymous Coward]

A backup strategy is only worthwhile if there is a recovery strategy.

[Rainer]

Interesting

A while (months at least) ago, somebody on the freebsd-fs mailinglist had an unrecoverable 36T pool after repeated crashes (due to power-failures, IIRC) and the subsequent (uncompleted) resilvers. There was also some sort of metadata corruption.

It was a huge thread that petered out with no solution - until the author came back a month or so later to claim she was able to access the pool again with the help of a commercial Windows-only tool made by a 3rd-party company.

[Anonymous Crowbar]

Re: Interesting

Could you point me at it? I had a 10tb 4disk array fail a disk. During resilver second disk failed. Powered it off to deal with when i have time as it is not super critical.

[Rainer]

Re: Interesting

Not sure if el reg allows links:

https://lists.freebsd.org/pipermail/freebsd-stable/2019-June/091254.html

Here you go.

[Robert Carnegie]

Re: Interesting

I'm going to imagine that the Microsoft-Windows-based tool is an imaginary one called "LongJohnSilver" - no connection if there is a real software product of that name (or a, um, specialized film actor) - and that it's slightly possible that the claim of recovered data is not authentic and is originated by the imaginary seller of the imaginary software. And endorsed by the, ahem, actor. It's just something to check.

And, having said that, the technique exists of simply scanning the disk surface for data regions that appear to contain data intended to be a file (not encrypted). So, without repairing the file system, you can get your data back. Even in Windows. Obviously an array of disks may be more challenging to investigate if a file is distributed amongst them. But if the distribution follows a simple pattern, it's not impossible. Or if you have small files that actually don't split across disks.

[katrinab]

Re: Interesting

If I was to write a tool for recovering zfs volumes, I don't think Windows would be my operating system of choice.

The number of computers that boot up to windows and contain zfs volumes is remarkably small. I actually have such a system - Hyper-V host, drives directly assigned to a FreeBSD virtual machine, but I don't think that's a common setup.

The number of computers that have corrupted disk volumes and are unable to boot up to anything at all is remarkably large; so usually these tools come on a bootable USB stick or similar. Distributing Windows in such a form would be a challenge from a licencing point of view, and Windows on a USB device is not a great experience.

[Wanting more]

me too

We also had a corrupted ZFS filesystem this week. We had a 40 hour outage of our main business database. Diagnosing the issue, then perform rebuilds / database restore and roll forward the db transaction journals, then checking / testing. Fortunately we had the required backups etc.

[keb]

Re: me too

Is there any reason to suspect a ZFS issue as the culprit?

I did see some discussion on Linux kernel-related boards that ZFS would never be officially supported due to its license.

[Mr Dogshit]

Gandi?

That Indian fella?

[katrinab]

Re: Gandi?

The Indian fella is Gandhi

[Cynic_999]

No sympathy whatsoever

Anyone who relies on a 3rd party to protect vital and/or irreplaceable data really gets what they deserve if that 3rd party loses it. How much money and effort does it take to make a local backup?

"The cloud" is convenient for sharing or accessing your data from many different locations, but it should never be considered a place for safe or reliable data storage. And a HDD is usually cheaper per GB storage than most online services anyway, even over just a year (not to mention being at least an order of magnitude faster).

[Anonymous Coward]

Re: No sympathy whatsoever

As a Gandi customer, I completely agree with you.

Some people will learn from this (and I'm certainly wanting to see the post-mortem), others will complain and will carry on living their miserable lives for a while longer until evolutionary theory takes care of them.

I also agree that having a "physical" copy of the data in your possession is a good idea, plus being able to see and touch that disk has something of a soothing psychological effect. :-)

[Anonymous Coward]

Re: No sympathy whatsoever

> Anyone who relies on a 3rd party to protect vital and/or irreplaceable data really gets what they deserve if that 3rd party loses it.

Mind, I'm not about to pull my money out of the bank, get paper certificates for my investments, etc., I suppose that there has to be a balance there between risk / likelihood / convenience / impact.

And with that said, I did pull everything out of a certain country in southern Europe during the last crisis, so yeah, still agreeing I guess.

[SuperGeek]

Re: No sympathy whatsoever

"Some people will learn from this (and I'm certainly wanting to see the post-mortem)"

Hmm, just like all the MegaUpload users then? How many of those uploaded their *only* copy of data to see it wiped out when the servers got swiped, and erased?

[Phil Endecott]

Re: No sympathy whatsoever

> How much money and effort does it take to make a local backup?

Possibly rather a lot if, like me, you have a system with hundreds of gigabytes of data and work from a home office with a bog-standard internet connection. “Local” backups just aren’t an option. Multiple “cloud” backups with different providers are the best available option.

[zb42]

Moxie Marlinspike, known for signal messenger and entertaining blogging about ocean sailing, has a bit of history with Gandi.

Marlinspike make a browser plug-in that anonymized google searches. Ghandi broke it by cancelling the SSL certificate without warning.

A decade ago he found that they accepted null characters in SSL certificates, allowing the issue of certificates that some browsers accepted as being for someone else's website. Ghandi locked his account without warning and customer service later told him he was personally banned.

When the certificates were approaching expiry Ghandi sent him emails suggesting renewals, with links that worked to renew the certificates

El reg passim links

https://www.theregister.co.uk/2010/04/05/googlesharing_cert_revoked/

https://www.theregister.co.uk/2009/07/30/universal_ssl_certificate/

[Huw D]

3-2-1 rule always applies, regardless of where your data is.