* Posts by wcpreston

23 publicly visible posts • joined 31 May 2018

Firm fat-fingered G Suite and deleted its data, so it escalated its support ticket to a lawsuit

wcpreston

That timeframe doesn't apply

I know this is a reply to a really old question, but I was doing research for my podcast. (We're covering this event next month on The Backup Wrap-up podcast.)

So just in case someone comes across it, your question is about why they don't keep the data for 51 days, as it says in their policy. That policy applies to deleting your subscription. It's similar to when you delete your subscription to, say, Netflix. The actual deletion will be some time later (usually the end of your billing period). They're even saying that if your subscription lapses, they'll keep your data for 51 days to give you a chance to renew it.

But in this case, the customer deleted the account itself, not the subscription. That is a purposeful act they are then immediately honoring.

wcpreston

Google Vault is not a backup. "Data retention" is for e-discovery and archive purposes. It would not have been able to restore this account if it was deleted. In addition, Google Vault is part of your account. Delete your account and it goes with it.

wcpreston

Story update: Musey dropped their lawsuit two weeks later

I can't believe I didn't find this out until now. Musey quietly dropped its pro se lawsuit 12 days later.

https://dockets.justia.com/docket/california/candce/4:2019cv03864/344456

I knew the lawsuit was without merit because there is nothing in the Google contract that says they provide backup.

I still wish that any service contract (and service description on the website) with an IaaS/PaaS/SaaS vendor would specifically address the situation of backup. Salesforce is probably the most forthcoming, but most SaaS vendors never address the FACT that they are not providing backup for your account. In fact, none of them (as of this writing) even offer it as an additional service.

And if they did, I would question its validity due to what happened to OVH's customers who paid for what they thought was an offsite backup. The service description said backups were triple-replicated and the service agreement said backups were "physically isolated." Turns out that meant the server was in a different part of the datacenter. You know... isolated.

Please make backups of your IaaS/PaaS/SaaS data, and know exactly where those backups are going.

Disclaimer: I do work for a backup as a service vendor, but that doesn't decrease the validity of what I said above.

wcpreston

Re: So...

Can you provide any links on this extra-cost G Suite backup?

wcpreston

Re: What use is Google's statement about backups?

That is not what that google statement says. It says they can restore a deleted USER, because a deleted user goes into a recycle bin type area. This is not a deleted user; it's a deleted account. BIG DIFFERENCE.

This is not a cloud computing problem. This is a "user assuming their SaaS service was backing them up w/o verifying that" problem. I've been preaching this for quite a while. This will now be my test case every time I talk about why you should backup your SaaS data.

wcpreston

Your SaaS vendors aren't backing up your data

Please check your service agreements. You might be surprised. Neither Salesforce, Office 365, or G Suite have what I would consider a backup for your data. I would define that as something I could use to restore all my data when the caca completely hits the rotary oscillator.

Salesforce has KIND OF a backup, but it takes 6-8 weeks to get ahold of it and costs $10K. And it doesn't restore everything.

MS and google has recycle bin type features, but those are all stored w/in your account. They go away if you (or a bad actor) delete your account.

What I don't like is that they are not open about this. Either they should 100% cover you and support it, or they should very plainly state that backups are your responsibility. Right now they do neither.

wcpreston

Re: Under a cloud

It's not in the service agreement for them to do so, so why should they? None of the major SaaS vendors (Salesforce, Microsoft, Google) have backup your data in a way that any backup person would recognize as a backup.

Salesforce has a restore service that generates CSVs of your account. It takes SIX TO EIGHT WEEKS, costs $10K, just to get a bunch of CSVs that you can then use to restore your data. Metadata is gone forever. Even they don't recommend you to use it.

All of MS and Google's documented data protection features leverage the recycle bin or versioning, and thus are not a backup. MS does have a delayed replicated copy of your entire account they can use in case of THEIR disaster, but they do not make it available in a scenario like the one in this article.

Please look at your service agreements. Look for the words backup, restore, recovery, data protection, etc. I think you'll be surprised.

wcpreston

Re: Conflicted who and what to bash

I do agree that Google should have notified them quickly that there was no hope. Google does not have a backup of your entire account. As with other SaaS vendors, backups are your responsibility. Feel free to consult your service agreement.

PC store told it can't claim full cyber-crime insurance after social-engineering attack

wcpreston

Re: Closer to being insurance fraud

Yes, because the "hack" was actually executed by an employee of the company when he clicked on that email.

The attacker used social engineering to get them to open the email and/or attachment and/or click the link.

Had they not opened the email, the attack would not have happened.

Ukraine's secret cyber-defense that blunts Russian attacks: Excellent backups

wcpreston

Or... you could have off-site backups stored in a completely different computing environment that none of your employees have admin access to. This is what you get using a SaaS data resilience solution.

As you already alluded, the only truly secure computer is one not connected to anything. But having backups in a completely different cloud provider, different account, different design, and different authentication mechanisms, with NO logins whatsoever to said infrastructure makes it extremely unlikely that some hacker attacking your production network would be able to use your network to also attack your SaaS backups.

Disclaimer: I work for such a company, Druva. I'm also an expert in backups, having written four O'Reilly books on the topic. You can get an ebook copy of my latest one (Modern Data Protection) here: https://www.druva.com/ebook.

wcpreston

Re: Insightful

As someone who has focused on data protection and resiliency for 30 years, it warms my heart to see you say that. Backups have always been afterthought.

The wrong guy: Backup outfit Spanning deleted my personal data, claims Cohesity field CTO

wcpreston

Re: this guy is a STORAGE EXEC

Cohesity does offer data protection of some SaaS products. They do not appear to offer one for G-Suite (now Google Workspace). They also, like my employer, Druva, do not market to the "prosumer" space.

I therefore see nothing wrong with him using another company's service to back up his personal data. This isn't a case of "he didn't eat his own dogfood."

AWS celebrates Labor Day weekend by roasting customer data in US-East-1 BBQ

wcpreston

Re: Using the cloud doesn't absolve you of the need to design your platform

"Made to believe." Not by the product descriptioin or the SLA, that's for sure. The product description comes right out and says you will lose 1 or 2 volumes a year if you have 1000 volumes -- so backup. The SLA does not imply anything about backup.

So if they were "made to believe" that, they didn't get it from Amazon.

wcpreston

Re: re: if you use EBS, that sort of failure is to be expected.

"Manglements" should at least know how to read an SLA. That's what they are good at. Someone somewhere should mention that this doesn't include backup, and if they're doing their job then they would double-check that. And if they double-checked it, they would find out they are responsible for backup.

wcpreston

Re: Using the cloud doesn't absolve you of the need to design your platform

Yes, you can migrate your on-premises systems to EC2 in a 1:1 mapping -- as long as one of those servers is the backup server. ;) . (Or, of course, use the cloud equivalent of such.) . These customers migrated but left backup out of the equation.

wcpreston

Re: But everything's OK.

No. Customers who lost data on their EBS volumes did NOT have it backed up to anywhere – including the cloud. EBS is nothing more than very resilient block storage. Nowhere in its service description or SLA does it imply that you do not need to back it up. In fact, in the service description it blatantly states you can expect to lose 1 or 2 volumes a year if you have 1000 volumes. ... so you should use EBS snapshots and back it up.

Customers that lost data had no backup of their EBS data.

The glorious uncertainty: Backup world is having a GDPR moment

wcpreston

Re: Backups are GDPR compliant trough retention.

Waiting for stuff to expire from backups is a great response. The problem is too many companies keep backups for years, decades, or even forever.

wcpreston

Re: Not a problem

We brought it up before. It's getting coverage now because the law is now in effect. Such is life with news.

There are actually sections in the GDPR that speak to technical infeasibility and undue burden as a defense against certain requirements of the law. In addition, the need to keep the data for other valid business purposes is also a defense.

As to what you're proposing (restore, delete, backup again) for every single request? The cost is so high that most companies would just pay the fine if the law were to be enforced that stringently. We're talking costs in the tens of millions every single time you get a request. My opinion is that is never going to happen. Not to mention the risk of doing something wrong and doing damage to the company.

The ICO said they will provide guidance on this soon, and I for one am looking forward to it. I'm willing to bet the advice is going to be closer to what Robert Wassall said in the article. The data needs to not be accessible to production systems, not be used for any decisions, etc. To that i would add that a company must commit to deleting it if it ever DOES come out of the backup system via some kind of restore.

My opinion so far.

wcpreston

Re: Snapshots

No, I am NOT suggesting anyone do that with their snapshots. But your suggestion has another issue. The snapshot you're suggesting the company delete has other legitimate purposes. It's very common, for example, to keep all snapshots for 30, 60, 90 days, or even longer.

wcpreston

Re: Not a problem

I'm not sure the filing room analogy works here. In this analogy, the filing room represents the main "production" copy of the data. And yes, you WOULD have to delete it in that case.

But imagine, if you will, you had a scanned JPG of every piece of paper in each drawer, stored in a completely separate system. Continue to imagine that you don't have OCR, so you can't scan the contents of each JPG without physically pulling each one up, reading all the words in it, then moving on to the next image. Now imagine being asked to redact info from those JPGs without the ability to search them. Now you're a bit closer to what we're talking about here.

I agree with your last comment. I look forward to further guidance from the ICO.

wcpreston

Re: Not a problem

I can only speak for myself. My blog was a response to comments I was seeing out there that suggested that GDPR RTBF was absolute and everyone should be able to delete personal data from production AND backup systems. So I suggested that wasn't possible given current technology (nor do I think full RTBF from backups is coming any time soon), and suggested the kind of process you mentioned in your comment. So I feel like I'm trying to clear up FUD, not create it.

wcpreston

I had a person reply to one of my twitter comments that this exact thing happened to him. The company "forgot" him, then a while later he started getting emails from them. Upon investigation, they realized the restored the marketing database to before he was forgotten. They had to institute the kind of process we're talking about.

wcpreston

Re: Not my field of expertise

Well this IS my field of expertise, so I'll chime in. :)

Many backup systems have added (or are in the process of adding) features to delete personal data from the backup -- to a certain extent. For example – depending on how your backup is stored – it is technically feasible to delete all spreadsheets, word processing files, or PDFs with a certain person's identifier in them. But asking that same backup product to delete the person's data from within a file or database – while keeping the rest of that file intact – is venturing into extremely dangerous waters. If that's what we're asking, I'll have to agree with the quote in the article form Linus Chang, "deleting data from a backup is a terrible idea because it risks corrupting the backup, breaking referential integrity, breaking applications that were expecting that data to be present, and importantly, breaking any checksums on the data that would prove that a restore was successful."

That leaves the "delete on restore" option. My opinion is that a RTBF journal/database that stores ONLY the unique identifiers (and no other data) – while it sounds on the face a direct defiance of the RTBF – is the best way to ensure the person "stays forgotten" if there is ever a restore from older backups. It's even possible to have the backup system trigger the "make sure these people stay forgotten" process after a restore.

The RTBF article of the GDPR says you can keep data required to defend against a legal claim. In addition to being used for this "make sure they stay forgotten" process, this database I'm proposing can also be used to prove when someone asked to be forgotten, when they were forgotten, etc, in case of a GDPR claim. In addition, the use I'm proposing is also to protect against a legal claim – that you said you forgot somebody that you ended up restoring from backup. Ergo, I think it should be OK to have a RTBG journal/database. I am not a lawyer and am not giving legal advice on GDPR. I'm just spitballing here.