'No BS' web host Gandi lives up to half of its motto... Some customer data wiped out in storage server meltdown Customers of web hosting outfit Gandi.net have been left less than impressed by its handling of a data-destroying storage crash. The France-based hosting provider on Thursday disclosed it had lost some customer data after a ZFS storage box in Luxembourg broke down and had to be replaced using a backup. Efforts to restore the …
But we restored our platform and we will not bill you while you rebuild your site from scratch and while you ask your customers for a copy of their latest bill you sent them last month. I am sure they will tell you what they owe you. Do we get at least partial credit for that??? Maybe if you go look in the way back machine, you can find a copy to start from... I don't understand why your so mad, it could happen to anyone...
It could happen to any other web host, who have also never tested their Disaster Recovery plan. I've worked for companies who never test their backups because no one wants that job and it's time consuming but for a web host to do that I'd consider a serious failure of duty. Personally the SQL servers I'm responsible for tell me that their daily backups are verified but I don't trust them and do test restores at least a couple of times a year. The current company I work (just a typical technology SME) does a full cut-over to a mirrored DR site for essential services twice a year. Except for Exchange, which is apparently too complicated, or the Exchange team are lazy, so we are trusting the untested DR plan for email when the meteor hits the server room but at least we don't provide email services to customers.
If it's an Exchange DAG, performing a full cut-over may cause a disruption or outage, depending on several factors. (How the DAG is configured, how your DR/ Business continuity testing is configured, etc.)
It's not necessarily that your exchange team is lazy, it may more be a case of it's a giant pain in the nethers to perform.
In my case at [RedactedCo], we have a 4 node DAG (2 nodes in prod, 2 in DR), with a witness server and an alternate witness at our DR site, which is inline with the minimum for the Preferred/Reference architecture. Cutting over is a semi-automatic operation, takes ~15-20 minutes, and is NOT transparent to the business. Failing back or ending the test is about as painful and non-transparent to the business.
No load balancers sitting in front of your internal Exchange?
Having done similar tests, removing back-end servers from the load balancer is used for patching/DR testing. Admittedly we don't perform a full shut down of the load balancers in order to avoid disruption but we do monitor load balancer connections as part of the test to see what is going through them and decide whether systems have failed over successfully based on that information to avoid surprises. It resulted in a lot of misconfigurations being found and corrected.
To be clear, we would probably see some disruption in a full failover across all of our estate but have a fairly good idea what will give us problems based on our tests and how to fix them....some apps are just "special"
> but this is pretty poor.
It is a hardware failure. Don't tell me you don't plan for that.
> Also interested in recs.
What do you need? Data integrity or availability?
In the first case, have a properly designed and well-exercised data back up and recovery plan¹, or pay someone to do it, but it is *your* data after all.
In the second case, redundant servers with automatic failover is probably a good starting point.
Combine both as needed if necessary to meet your needs.
If you are using shared hosting (as I do: Gandi in fact), consider it a best-effort resource that can fail at any time in a number of more or less spectacular ways. If using dedicated hosting, make sure to go very carefully through your service level agreement.
What I look for in a hosting provider is a) that they don't fail too often or in too mysterious ways and b) that wenn die Scheiße den Fan trifft they are at hand to help sort things out.
Right now, I'd recommend Gandi. Once bitten twice shy, etc. :-)
¹ For the data that needs to be preserved. Not all of it does, usually.
> Nice to see a Gandi employee backing up the company line.
There is always a tinfoil hat wearing retard that's going to come up with an asinine comment like that. It states clearly in my comment that I am a user.
As a matter of fact, a dead smug user right now, because even though my servers were not affected I actually do have a well conceived and regularly practised backup and recovery plan. Which, if one were to judge by the "reporting" and by the comments on this site, you would think it's a patentable novelty.
Even so, the previous comment was meant to be a helpful answer to an in principle legitimate question ("what can I do to minimise the risk of loss?")
The hosting contract (which one is required to accept when buying or renewing) states in § 3.3 that you are responsible to back up your own data. Here's an excerpt, bolded and outlined in the original:
"The backing up of Your content and technical settings shall be Your sole responsibility. You must take all the necessary steps to ensure regular backups and protection of Your Content and all Your data and configurations, as well as their updates, on media outside Gandi."
This is the same as every other hosting provider that I'm aware of, which is why I can't bring myself to have any sympathy for anyone who ventures into the somewhat technical land of shared hosting not being aware of this or just simply counting on their luck.
> However they're stretching this pretty thin to try to explain away their entirely separate failure to test their backups
There aren't any backups. When you buy shared hosting, it does not include backups unless you explicitly contract such a service (if available, not everyone offers that) and pay for it. I wish this article was a bit more clear about that industry fact that I would have thought rather obvious but clearly a lot of people seem to think that because they're buying space on a disk someone else is going to take responsibility for what they put in it.
This is exactly the sort of bullshit "journalism" that's certainly not needed. Complete with Twatterati comment and all.
Yes, it is regrettable but these things do happen with certain regularity.
And as can be seen from their status page, it is not quite clear yet whether data may be recoverable or not: https://status.gandi.net/timeline/events/2109
Name one shared hosting company that backs up their filers. And their prices, please.
This is what I take Gandi to mean by "could have happened to anyone". Perhaps you prefer the canned "a small number of our customers were affected" PR crap. I don't.
Gandi customer for well over a decade. They have always been very decent I haven't found anyone else equally competitive.
> It's easy to compete on price if you skimp on quality and reliability.
I wasn't talking about price or I would have said cheap instead.
Please name a more competitive provider, along with your explanation why. I am sure some people will be genuinely interested.
I wasn't talking about price or I would have said cheap instead.
"Competitive" is generally taken in terms of "value for money", so price is a part of it. It doesn't have to mean 'cheap', you can be underfunded without being cheap.
Please name a more competitive provider
No idea, I avoid hosting providers for exactly this reason. I run my own hosts, although I do use a provider for personal email. I make my own backups, since their SLA is clear that they don't backup my user data. I do expect them, though, to backup their own data including configuration related to the service they sell me, so that they can have their hosts up again ASAP after a failure.
AWS Lightsail makes an interesting point of comparison.
One observation is that AWS is probably one of the few places where, due to their sheer scale, hardware failures like borked disks must happen quite regularly. So whereas a small provider will have a theoretical disaster recovery plan that you hope they have rehearsed, AWS will have a routine and frequently-used process to deal with failures.
The main storage on a Lightsail instance is EBS, which is replicated within the same “availability zone”. So it should be resilient to a single disk failure, similar to how RAID is. You can then configure it to make incremental snapshots on whatever schedule you need, and these are distributed around the different availability zones in the same region.
Pricewise there are various options; compare Gandi’s “M” plan (120 GB bandwidth, 20 GB storage, about €10/month IIUC) with Lightsail’s $10/month plan : 3 TB bandwidth, 60 GB storage. You can have that in London, Dublin or a load of other places.
Lightsail’s (and AWS’s) weak point, in my experience, is the support; it seems to use Mechanical Turk and/or chatbots, rather than actual support humans. But compared to my experience with Gandi (and another European provider, Scaleway), things generally don’t go wrong so you don’t need support.
Of course this is an apples-and-oranges comparison in many ways.
"Catrina", those comments make you look pretty stupid. I make no apologies for laughing at morons who can't take responsibility for their own data and I despise people who cannot accept blame for their own mistakes.
Entitled crybabies like that are nothing but a waste of this world's limited resources.
That guy Andrea is a bit of a cretin. "We move out today" (/rolls eyes)
What was *his* recovery plan, I wonder?
As someone else said, the "cloud" is someone else's computer. Which is fine, as long as you keep that in mind and proceed accordingly. Hoping that your inexpensive shared hosting is somehow going to be immune to hardware failure is not really my idea of proper prior planning.
Looking at the comments here, that people seem to be under the impression that hosting providers systematically keep back ups of customer data.
That is not the case, chaps. For a start it'd be uneconomical and uncompetitive; you'd be paying for the back ups whether you actually needed them or not¹. Then there is that whole box of worms with GDPR and similar laws, where if you do not control your back ups you might end up in a position where you cannot actually comply with legal or industry regulations concerning data rights, security, etc.
¹ For instance, I have used shared hosting to process one terabyte of data twice a day. The data is obsolete after about twelve hours and expensive enough as it is, without having to pay for it twice.
> Looking at the comments here, that people seem to be under the impression that hosting providers systematically keep back ups of customer data.
No one expects them to keep backups in the sense of of "Can you retrieve file X.DAT as it was two weeks ago" but people do expect them to be able to restore their systems to operational order within the stated SLA RTO and RPO. That is what seems to have failed here.
> people do expect them to be able to restore their systems to operational order within the stated SLA RTO and RPO. That is what seems to have failed here.
Precisely!
Can't see what part of their SSCH CSC they are failing to meet. Section 3.3 clearly states the backup policy too.
> they could have avoided this by having snapshots and storing them on 2-3 other servers that aren't the production server.
That is almost exactly what I do as part of my recovery strategy. There is the option of doing that with Gandi, it's called "active repulsion" or some such.
The advantage is that it's very easy to spin a new server up if the old one dies but at least part of their infrastructure is still available. Don't forget to do the same to your OS disks, as well as data, as reconfiguring a new server is easily the most time consuming (and error-prone) part.
If your provider doesn't have a similar option, you can simulate it by snapshotting the data and copying it to another provider and/or downloading to your own infrastructure.
In any case, if you are on the "cloud" and you only have one copy of your non-disposable data, you are doing something seriously wrong.
Others have summed this up very well. My thoughts are - Gandi should have system level backups to get their VMs/systems back up and running again - irrespective of billing for it - those services generate revenue and being available/not being entirely lost is a big part of a hosts reputation. That alone is worth making sure system level/VM restores are available.
Customer based backups - i.e. give me file X from Y time and date is different and should be billable. Additionally, customers should backup their critical files to another provider/platform in the event they need to restore, because that's just sensible.
Either way, using ZFS arrays and relying on ZFS Snaps to restore from isn't the best strategy, ZFS is great until it goes wrong and then it goes really wrong - and that can be admin or hardware failures.
The 3-2-1 backup rule would have helped here. Three copies, two different devices/storage media types and at least one of those being offsite.
From a Gandi perspective, that might have looked like ZFS Snapshots (1), Snapshots mirrored to another independent array (2), that array rsync'd offsite to another site (3). With that, if one and two failed, they could have restored the (3) to another active array and got back up and running again.
Customers could have happily had a VM on Gandi with backup to S3/Backblaze with a lifecycle to delete old files and a copy to their homenas/dropbox/something else. That'd have given them the ability to get back up and running anywhere they could aim DNS records to.
In reality, large scale arrays for just storing large volumes of sync'd VMs to are cheap now, they don't need hugely fast disk i/o, the access pattern is sequential, so a low'ish cost box filled with 8/10TB drives gives a low cost of restore. A host "shouldn't" need it, but in times that this, spending that few K gives them another option. When getting back up and running, options are exactly what you want.
A final point would be, never make a backup plan, make a restore plan, work out what you need to restore to get backup and running, then protect the hell out of it!
Good summary above.
> Gandi should have system level backups to get their VMs/systems back up and running again
Note that the VMs did not go down. It's just a storage unit that crashed. From talking to an affected user, their VM is still running happily, except that it cannot "see" the data disk.
> Either way, using ZFS arrays and relying on ZFS Snaps to restore from isn't the best strategy, ZFS is great until it goes wrong and then it goes really wrong
I have no experience with ZFS but from their communications, that is indeed the problem they're facing at the moment with attempting data recovery.
Agreed with the rest of your points, especially good point about the restore plan. It is like in aviation, where landing is your plan B; your plan A is going around. It puts you in the right mindset.
zfs is a file system, not a backup system. It guarantees that any data you read from the disk will be exactly the same as what you (or a virus etc) wrote to it. It does not guarantee that you will get it back. Other file systems will give you corrupted data in situations where zfs will know it has been destroyed and not give you anything.
Somebody asked me why I don't back my company data up to the cloud.
This is one reason why. Imagine you'd get hit by crypto-malware, and just before you're ready to start restoring data, your backups goes TITSUP.
Second reason is - I want full control over my backups. Who have access to it, where it is, and how old it is.
And the third - $$$. It works out cheaper long-term to store it yourselves than having it out on somebody else's purdy compoota (or RAID array).
A while (months at least) ago, somebody on the freebsd-fs mailinglist had an unrecoverable 36T pool after repeated crashes (due to power-failures, IIRC) and the subsequent (uncompleted) resilvers. There was also some sort of metadata corruption.
It was a huge thread that petered out with no solution - until the author came back a month or so later to claim she was able to access the pool again with the help of a commercial Windows-only tool made by a 3rd-party company.
I'm going to imagine that the Microsoft-Windows-based tool is an imaginary one called "LongJohnSilver" - no connection if there is a real software product of that name (or a, um, specialized film actor) - and that it's slightly possible that the claim of recovered data is not authentic and is originated by the imaginary seller of the imaginary software. And endorsed by the, ahem, actor. It's just something to check.
And, having said that, the technique exists of simply scanning the disk surface for data regions that appear to contain data intended to be a file (not encrypted). So, without repairing the file system, you can get your data back. Even in Windows. Obviously an array of disks may be more challenging to investigate if a file is distributed amongst them. But if the distribution follows a simple pattern, it's not impossible. Or if you have small files that actually don't split across disks.
If I was to write a tool for recovering zfs volumes, I don't think Windows would be my operating system of choice.
The number of computers that boot up to windows and contain zfs volumes is remarkably small. I actually have such a system - Hyper-V host, drives directly assigned to a FreeBSD virtual machine, but I don't think that's a common setup.
The number of computers that have corrupted disk volumes and are unable to boot up to anything at all is remarkably large; so usually these tools come on a bootable USB stick or similar. Distributing Windows in such a form would be a challenge from a licencing point of view, and Windows on a USB device is not a great experience.
We also had a corrupted ZFS filesystem this week. We had a 40 hour outage of our main business database. Diagnosing the issue, then perform rebuilds / database restore and roll forward the db transaction journals, then checking / testing. Fortunately we had the required backups etc.
Anyone who relies on a 3rd party to protect vital and/or irreplaceable data really gets what they deserve if that 3rd party loses it. How much money and effort does it take to make a local backup?
"The cloud" is convenient for sharing or accessing your data from many different locations, but it should never be considered a place for safe or reliable data storage. And a HDD is usually cheaper per GB storage than most online services anyway, even over just a year (not to mention being at least an order of magnitude faster).
As a Gandi customer, I completely agree with you.
Some people will learn from this (and I'm certainly wanting to see the post-mortem), others will complain and will carry on living their miserable lives for a while longer until evolutionary theory takes care of them.
I also agree that having a "physical" copy of the data in your possession is a good idea, plus being able to see and touch that disk has something of a soothing psychological effect. :-)
> Anyone who relies on a 3rd party to protect vital and/or irreplaceable data really gets what they deserve if that 3rd party loses it.
Mind, I'm not about to pull my money out of the bank, get paper certificates for my investments, etc., I suppose that there has to be a balance there between risk / likelihood / convenience / impact.
And with that said, I did pull everything out of a certain country in southern Europe during the last crisis, so yeah, still agreeing I guess.
> How much money and effort does it take to make a local backup?
Possibly rather a lot if, like me, you have a system with hundreds of gigabytes of data and work from a home office with a bog-standard internet connection. “Local” backups just aren’t an option. Multiple “cloud” backups with different providers are the best available option.
Moxie Marlinspike, known for signal messenger and entertaining blogging about ocean sailing, has a bit of history with Gandi.
Marlinspike make a browser plug-in that anonymized google searches. Ghandi broke it by cancelling the SSL certificate without warning.
A decade ago he found that they accepted null characters in SSL certificates, allowing the issue of certificates that some browsers accepted as being for someone else's website. Ghandi locked his account without warning and customer service later told him he was personally banned.
When the certificates were approaching expiry Ghandi sent him emails suggesting renewals, with links that worked to renew the certificates
El reg passim links
https://www.theregister.co.uk/2010/04/05/googlesharing_cert_revoked/
https://www.theregister.co.uk/2009/07/30/universal_ssl_certificate/
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
