Firm fat-fingered G Suite and deleted its data, so it escalated its support ticket to a lawsuit An interior design tools startup called Mosss on Wednesday sued Google to get it to restore its data after someone at the startup accidentally deleted the firm's G Suite account. In a pro se lawsuit [PDF] filed in US District Court in Oakland, California, Mosss, under its previous corporate name, Musey Inc., asked Google to …
Cloud systems are no better or worse than in-house systems - probably better than most in-house systems I have seen, actually, and def. way more secure.
You still have the responsibility to make sure everything is correctly setup and have recovery procedures for emergencies, it is not the responsibility of the cloud provider. If the story was "in-house IT system crashes, deletes all data with no backups" it would be a non-story as that is SOP for most small businesses.
In this case, it's just a PEBKAC.
a) I'm referring to the people running the in-house systems that the poster was commenting on being amateurs "probably better than most in-house systems I have seen, actually, and def. way more secure". However, seeing as you seem to have misinterpreted my comments then
b) Google are merely the hosts, providing the software and infrastructure. There still has to be an administrator, well, administrating. That's the problem not Google.
I have never heard of anyone filing a lawsuit to get an off-site data storage company to deliver a back up. Never once has an LTO tape library asked to see a court order before it restored data from a tape. I am very confident that the AES256 encryption algorithm has never consulted a lawyer before encrypting a file.
I am sure the people at Google get paid. That is all they need to be called professionals. That doesn't mean relying on a single account from a sole cloud provider to store or backup all of your company's data is a good idea. We are supposed to know the tools of our trade and then use the right tool for the job. Cloud computing is a great resource but if it is the the tool in your box then I say you are doing IT wrong because doing IT right will never create a single point of failure.
Professional (paid to do IT) <> Professional (paid to do IT and actually knows WTF they're doing, has standards)
AKA - the MCSE[1] effect..
[1] Must Call Someone Experienced. I'm sure that there are some people with MCSE qualifications who know what they are doing but they are far outweighed by people who have gone through boot-camp[2] style MCSE courses and don't actually have a clue about IT.
[2] Not the Mac thing..
Yes, I've done a lot of MCSE training and exams in the last couple of years, after 35 years working in IT* and using Windows since 1987. The MCSE level of training and the exams are for n00bs, the course touched on about a third of the knowledge required to effectively run an MS installation.
It doesn't teach you to think laterally, it doesn't teach you to think outside the box and it is no replacement for experience.
* I was looking for a new job and decided that having a couple of qualifications would look good on the CV, then I actually took the courses and exams and was shocked how basic the knowledge required was.
Here is an employment tribunal judges view (not mine I just happen to know them) on "professional". If there is not a professional standards body that can kick you out of the profession and stop you ever working in the profession again if you turn out to be an incompetent numpty you are not a professional.
> PEBKAC
I agree, but how would you go about actually deleting a Gsuite account - surely there must have been multiple "are you sure/are you really sure/you know it's gone forever if you click YES" messages? And the unnamed muppet still went ahead?
They didn't necessarily actively "delete" the account.
I'm not sure about Gsuite specifically, but these sort of linked accounts can end up orphaned or mangled in certain edge-cases, such as removing the wrong user, etc. especially at smaller scales where it may have evolved from someone's personal account.
You sure don't know about G Suite.
The only way this story makes sense is if they actually DID delete their entire account
If it was just a user account you would have plenty of time to restore the user and wouldn't need to call Google and escalate it it to try and get the user back - the built in admin can do that just fine.
I ALWAYS tell my customers they should be paying for a third party backup solution for 0365 and G Suite - because despite those guys having extremely good infrastructure and security, they can't protect against internal stupidty/and or malicious actions of either an employee or a bad actor.
That is where you are wrong, why would an enterprise cloud solution automatically give you backups? If delete stuff in, on your own servers or in a cloud offering without any form of backup then unsurprisingly it is gone. This is where many companies (including really large corps) fall down. There is a belief that just because your data is on a cloud service it is protected. There will be some sort of protection in place but only so that the provider can sort out their own problems, not yours.
I have never understood why it is deemed to be acceptable to use Azure with the Azure Backup. If you subscription gets compromised it is all gone, just is if you have no off-site backup provision for a real data centre.
"you'd expect an enterprise cloud solution to be equivalent to an in-house system... So backed up and recoverable within a day of catastrophic failure / mistakes."
Depends, are you paying for the backup service? If not, you might not be backed up.
Still, you did check the small print before you signed up right?
you'd expect an enterprise cloud solution to be equivalent to an in-house system... So backed up and recoverable within a day of catastrophic failure / mistakes.
It is, if you do it right. We have several availability zones all synchronised, acting as hot backups, we use the cloud providers backup for data (CI/CD for code with github and local hardware acting as code backups), and we still retain our own daily data backup in our own equipment.
Of course, if you have inexperienced dreamers or fools setting up your backup and retention policies, it simply doesn't matter if you're in the cloud or using local hardware, it won't end well. Experience cannot be replaced with keenness and ego, or talent as the kids describe it because there's no short cut to experience.
The service people think they're paying for IS the security of having the date in the cloud, where it will always be backed up by service provider policy. Google is simply behaving like the near-monopoly that it is. We've also had cases of inadvertently deleted data that "cannot be recovered." Anybody else would have to recover it. Not Google.
They only do that with your personal information they use for advertising. Your own data is of no value to them so of course it gets binned the moment you delete it!
Timely article on CNBC showing exactly this - if you delete your Gmail Google keeps the purchase history they stole out of your emails: https://www.cnbc.com/2019/05/17/google-gmail-tracks-purchase-history-how-to-delete-it.html
No. That CNBC article is making an issue over nothing.
The "purchase" section is based on receipts in your *current* gmail inbox. Delete the email, the receipt goes.
As the article itself says:
"But there isn’t an easy way to remove all of this. You can delete all the receipts in your Gmail inbox and archived messages. But, if you’re like me, you might save receipts in Gmail in case you need them later for returns. There is no way to delete them from Purchases without also deleting them from Gmail -- when you click on the “Delete” option in Purchases, it simply guides you back to the Gmail message."
It just seems the person who wrote the email wants it to not exist in "purchases" but still exist in his gmail, despite "purchases" simply being a live summary of gmail receipts.
It's quite possible that there are compliance aspects to consider inasmuch as it Google may well be obliged to delete all data, with a reasonable period, when such an account is closed. If so, I'd expect this to be in the contract or the T&C's. If it is, then I wouldn't reckon much for their chances.
Google's contracts were written by Google, do you think they are obligated to do anything under the terms of their contracts? The only obligations they'll have would be legal ones that depend on which country's laws are applicable.
It isn't like you can call up Google and tell them you want to negotiate a contract for GMail services. They will tell you it is on the web site, and if you don't like the terms you can get elsewhere.
So what? Nobody forced the company to use Google and the principle ofcaveat emptor applies, especially when you're potentially betting the house on a service.
If you feel that a contract is one-sided, you normally also have recourse the challenge the contract in court. But basically this is a SNAFU that the company hopes Google will fix for them.
The countercase is, of course, you use GSuite for your very important and confidential data and then decide to move to another service. Surely, in such a case you'd expect that Google would make the data irretrievable within a short period of time after the contract was cancelled and that you'd take them to court if that wasn't the case?
This is a story, where I get so very much conflicted of who and what to bash.
- A business having no backups of its own business data; must be SNAFU.
- A simple click to delete all access to whom exactly?
- A non-responsive customer service; is there any other?
- A lawsuit to determine who to blame for stupidity; lawyers must make a living too.
- A cloudy venture entrusted with your data; its no longer your data.
At least, they used google. Then they may probably find all of their designs, conversations and sales eventually used in advertising, somewhere. That is actually a good idea, they should write a bot scraping google for their data. There must be something! Maybe in the cookies of their browser and certainly loads of incriminating data in the (google-shared and transparent) cache of Chrome.
A lawsuit to determine who to blame for stupidity; lawyers must make a living too.
The lawsuit's pro se; ie, the plaintiff is self-representing. They were definitely being idiotic(the request for subpoena mentions private medical documents as amongst the lost data, for instance), Google's a bit out of line here if they took weeks to inform a paying customer their data was deleted(during which time they probably charged the company at least once while giving the impression access would be restored).
if they took weeks to inform a paying customer their data was deleted
Well they didn't. The customer terminated the account, and thus became a non-customer. While Google gives you a grace period when you accidentally delete a document, they may not extend the courtesy to the accidental deletion of a paying account.
Gosh.. If only there were 3rd party backup products that can back up an organization's cloud data... If only.
"Put it in the cloud," they say. "It's always online," they say... Well it's not backed up apparently!
Fact, cloud providers make your data available as a best effort. They don't fix stupid.
...There was a way to do on-site backups. If only there was a way to then make a backup of that backup. If only there was a company that could store that copy for you off-site and return it to your business as needed.
If only this has been the standard model for backing up data for over 30 years. Then perhaps this news story would never have been written.
All sarcasm aside: You data is like your child in that you are ultimately responsible for it's well being. Choosing not live up to that responsibility doesn't make it go away.
'if only there was a company that could store it off site and return it to your business as needed'.
Around 15 years ago I went as part of a group to support a third-party customer who had hired a dedicated DR company.
The DR company's dedicated DR site was in another country in a site that was entirely surrounded by a grass-covered embankment - bomb proof(?). The 3rd party who used the DR company wanted to test the end-to-end process. They picked a nice simple NT server with a couple of DBs on it as the first step.
They allocated a whole weekend to this - did they know something we didn't?
On Saturday morning, the DR folk who *knew* we were coming did not have the required tapes ready.
They then took around two hours to find them.
They then realised after more hours of faffing about that they could not provide the contractually required server until late the same day. Despite having dozens and dozens of servers 'ready' on site.
Long before they found and allocated us one we had left in disgust.
On Sunday morning we returned and restored the NT server, recovered the DB's and completed local testing to our customer's satisfaction.
After a short booze-cruise by the driver, the boot fully stocked and the car's front wheels making occasional contact with the ground, we headed back to the UK.
The driver generously handed over some of his liquid stock to us passengers, so not an entirely wasted trip.
... a cloud is just vapour ...
Indeed ...
I've been saying this very same thing for years now and been mocked and criticized for it.
But it'll be -30º C in hell before I even think about putting anything in the cloud, be it Google's or anyone else's, no matter how convenient it may be.
Old?
Yes, that's why I know better.
O.
O, If you choose not to use a cloud solution then I am assuming you would use an on premise solution. If you delete files from an on premise solution there is a strong chance that you won't be able to get them back.
As someone with a long history in IT you would know that this story indicates two failings which apply equally to on-premise or cloud scenarios:
1. How did they "accidentally" delete their account? This is like del *.* on a root account for an on premise system - which has been done many times.
2. Why did they not have a backup that they could restore from? HA and DR principles don't change because you move to the cloud.
Hope you have a nice day.
"O, If you choose not to use a cloud solution then I am assuming you would use an on premise solution. If you delete files from an on premise solution there is a strong chance that you won't be able to get them back."
The only comment I have is that 'premise' is a supposition, 'premises' is a location.
Me too....and then I dropped my IPhone and discovered that to use ‘Find my phone’ I was forced to sign (for the first time ever) to sign into something called ICloud, and now It tries to steal my data or divert my instructions pretty much all the time. It’s like dealing with ransomware merchants.
No, they're both invisible gases. Steam is water directly converted to a gas. Water vapour is water dissolved in air whereupon it assumed a gaseous state.
If you're thinking of the stuff you see coming out of e.g. your kettle that is microscopic droplets of liquid water as the vapour begins to condense. They are small enough to remain airborne via Brownian motion but the reason you can see it is due to the difference in refractive index between the water drops and the surrounding gas.
"I have always suspected google has no viable backup and recovery process"
The surprising thing is that they don't have one and charge extra for it. You'd have thought they'd have had loads of business for a built-in backup system. (Amazon are quite happy to sell their customers a backup service.)
No, the real fuckup here is assuming your data is being backed up, without actually checking the contract.
Actually....
A cloud isn't vapour, water vapour is transparent. The cloud is actually tiny liquid water droplets.
On a hot day you can see "little fluffy clouds" forming above thermals. You cannot see the water vapour rising but you see the cloud where that vapour has cooled enough to condence back into water.
It's not in the service agreement for them to do so, so why should they? None of the major SaaS vendors (Salesforce, Microsoft, Google) have backup your data in a way that any backup person would recognize as a backup.
Salesforce has a restore service that generates CSVs of your account. It takes SIX TO EIGHT WEEKS, costs $10K, just to get a bunch of CSVs that you can then use to restore your data. Metadata is gone forever. Even they don't recommend you to use it.
All of MS and Google's documented data protection features leverage the recycle bin or versioning, and thus are not a backup. MS does have a delayed replicated copy of your entire account they can use in case of THEIR disaster, but they do not make it available in a scenario like the one in this article.
Please look at your service agreements. Look for the words backup, restore, recovery, data protection, etc. I think you'll be surprised.
That's hyperbole. I've seen a several businesses lose everything due to fire ... paper records, computer records, inventory, tools and tooling, the building, the company car & delivery vehicle, name it ... and still stay in business.
The businesses? A bakery, a machine shop, a boat dealership/repair shop and a startup making an application specific piece of medical gear. Insurance helped for all four.
I've never felt comfortable about "The Cloud", for exactly this reason. Who's responsible for backing up and insuring a restore from backup policy and procedure? Trusting that to some outsourced clown, just isn't going to work. There should always be a backup and restore process that is internal and verified to be working. Trusting Google with all of your data? HUMPH....
I worked at a place where the CEO wouldn't approve a backup agent for the Oracle *spit* DB the LoB ran on once he knew we could sqldump and restore the DB for free. The business accepted a 24hr recovery point, so every night cron would dump the data to an nfs share in the backup server room where it would be archived to tape. Simple as you like.
Logs and email confirmation were configured, a restore was tested every 6 months and visually checked on an adhoc basis. Worked fine for a few years, then one day when it needed to recover for real, it didn't work.
Looking at the SQL everything looked fine, but by about halfway in, hints of file corruption started. By the end, it was just line noise.
Lots of overtime for hundreds of the guys manually re-entering just shy of 3 months' worth of records, and operating at a reduced capacity for almost a month on the backup paper system. The IT manager got the old 'choose the sword or get the bullet' and the CEO was out within the year. Plus we made the national news, and international industry-specific news.
Winning bigly!
“The Cloud” is broad definition though. This little start up was using G-Suite which is nothing like an Enterprise cloud service where a business might run their own IaaS infrastructure(s) with their own ring-fenced network(s) containing hundreds of their own VMs or even their own hypervisor to create their own VMs. Along with their own Enterprises applications and BYOL licences. They would also have their own firewalls, routers etc, so “the Cloud” is effectively the new data centre, but without the need to have people racking and patching.
I've written all sorts of scripts and blocks to disable/remove tracking things on my system, but still, every time I get the (almost constant) "You must review and acknowledge our privacy agreement" popup from google, I'm not convinced:
"Nice try Google, but you still know who I am - don't try and pretend that my simplistic tidyups fool you", I mutter to my screen with a grating sneer.
Oops, did I write that last paragraph out loud? :-)
The company should have had backups. That said, they were likely not technically minded, so probably thought that the data being held on Google's servers should be enough. After all, nothing would be able to take out Google, right? The problem is, it doesn't need something to take Google out. All it needs is for someone to push the wrong button, either inside Google or (as in this case) outside it.
That said, do Google allow backups of G Suite? I don't see why they wouldn't, but I don't have access to the admin console of a G Suite account, so don't know what they offer..
do Google allow backups of G Suite?
Not sure about the rest of gsuite, but for email they have imap, so you can back up using that. It's pretty easy to set up. And I'd be surprised if their file storage thing didn't have an API that you could use to scrape/backup your stuff (I'm assuming they're not going to allow things like rsync).
I use multcloud to move data between cloud accounts. At the moment, I just move stuff between personal and business OneDrive accounts, plus a little bit to DropBox. Free tier enough for me, but the paid thing is only a few pounds per month to shuttle your data all over the place. You still have to pay for the storage with MS, DropBox or Google, but it's almost as good as backup if you really don't want anything local.
thought that the data being held on Google's servers should be enough
When you start a business, there are lots of "non-core" responsibilities you have to take on: tax, employment law, health & safety, planning, consumer rights, etc. Even if you don't immediately run into them, you have to understand them them to the extent they affect your business plan. Even if you decide to ignore them, you need to fully understand the risk of doing so. If your approach to critical business data is "What could possibly go wrong?", I'm not sure you should be in business at all. Certainly, if I'd been asked to contribute to the $1.5M investment, I'd have wanted to find out a bit more about how the business intended to operate: the role of investors is not just to provide money, but scrutiny and experience.
That said, do Google allow backups of G Suite?
Good question. And even if they do, how easy are the backups to do? I recall that about the third thing Google did when in bought Blogger many, many years ago was to delete the capability to load Blogger websites by ftp. So much for any possibility of easily maintaining a local copy of your website. (Or of moving it quickly to another host).
"That said, do Google allow backups of G Suite? I don't see why they wouldn't, but I don't have access to the admin console of a G Suite account, so don't know what they offer.."
Yes, various options. To automate it, the best way is to use their API.
There are plenty of services out there that are very reasonably priced that will do it several times a day and let you restore email and drive contents right back where they were with a couple of click. ackupify (which was bought out by Datto) is the one I put people on. There is also SpinBackup which is popluar
Yes, Google allow you to backup G Suite, there are many such 3rd parties that will do it for you. Most people we speak to have no idea that there aren't any backups for them to access on G Suite. They assume Google will take care of it all for them if they cock something up. Same with Office 365.
The service provides an option for backups.
It's not free but it's an option.
The problem is they deleted their master account which would include whatever back up service they were using.
Most cloud providers make you step through multiple steps to delete an enterprise account up to and including multi factor validation.
Not sure how Google would win in this situation, ignore the delete request and be nailed for not deleting.
Except the delete request and be nailed for accepting the request.
> Accept the delete request, lock the account, and put a stay on final execution for a month or two.
And be responsible for the data not being deleted ?
Suppose I pay Iron mountain to destroy some paperwork and I find that they have been hanging on to it for a couple of months "just in case" I might get a little unhappy
All this nonsense about running entire business on cloud with no backups should worry more IT folks given how frequently cloud platforms tend to fail.
Even multiple cloud providers can experience near simultaneous regional outages and having things in a S3 bucket and github project doesn't cut it as being "backed up".... it is how data exposure and breaches happen....
Sounds like the olden days grey and white bearded druids were right about managing your own IT inhouse if your business depends on it!
Google will have a backup, however it will probably be a daily tape archive. They might not have the ability to easily restore a single account, and even if they do it will probably be a lot of effort/cost.
If it had been a major account they likely would have restored it from the last daily snapshot, possibly sending a $XXX,XXX bill... however they certainly will not want to set a precedent saying that they will do a tape restore of a small account for free.
Why do they have to go to tape? I am sure that a lot of the time when you delete a file or email it is just marked as deleted then only garbage collected after a certain period of time (hence them saying they might be able to restore), however doing that for an entire domain and all sub-accounts is a lot more complex and prone to error. If they were able to do a partial restore... the lawsuit would be over the fact that they agreed to restore the data but something was missing!
Of course the next question will be, how long do they keep their daily tape backups for before overwriting them? If they wait a month before responding that there is no Federal jurisdiction and it needs to be referred to a State court then the data problem could disappear by itself.
Google keeps everything forever. They just don't want to admit it.
My other halfs Gmail had a brain fart a year ago. Suddenly 10 years of deleted emails all came back, including all the deleted binned spam.
This was all stuff that was fully deleted as far as she knew. Took ages to clean up.
So, we have two options, here, either:
1. "other half" knows how to delete and expunge email in a mailbox AND Google keeps everything, even deleted stuff - shit, this would mean they have extended their IMAP protocol implementation to "not actually delete" stuff, simply move it elsewhere.
2. "other half" DOES NOT know how to delete and expunge email in a mailbox.
I am very sorry for your other half, but option 2. is so much more likely ...
So she clicks 'Delete' then she go to the Bin and clicks "Delete Forever". Some emphasis on the FOREVER part there.
Well I never, seems she does know how to delete stuff from Gmail!
And then 10 years later it all comes back.
You don't have to be such a 'typical IT guy' about it.
This is the kind of thing that can happen with a misconfigured mail client. The mails were most likely being copied elsewhere, like an old forgotten outlook express, but without the deletions on Gmail being properly mirrored, and then suddenly resync'd for some reason. I've seen this kind of thing happen a few times... It's an absolute pain in the ass to tidy up, as you found.
Hmm deleting data accidentally is one thing, and one could reasonably expect Google could recover that. But the suggestion here, is that someone has deleted the actual Google account in its entirety (accidentally or otherwise). Expecting that an entire account can be recovered - in spite of the warnings that use words like "irreversible" and "permanently" is a bit unreasonable in my opinion.
Google states that it is able to recover data when you get hold of them quick enough, but like all bigger US companies they use self-help and forums to explicitly avoid talking to customers.
Ergo, their backup promises are worth sweet FA, because by the time you have dug through all the defensive layers to NOT to talk to customers, your data will be gone anyway.
Cloud computing - for when you want your data to evaporate...
There are backups. If you delete mails or drive files you have 25 days to recover them. if you delete a user account you have 20 days. Drive files which have been edited have some 100 versions with them.
Deleting an entire account has to be done on purpose (see post below).
That is not what that google statement says. It says they can restore a deleted USER, because a deleted user goes into a recycle bin type area. This is not a deleted user; it's a deleted account. BIG DIFFERENCE.
This is not a cloud computing problem. This is a "user assuming their SaaS service was backing them up w/o verifying that" problem. I've been preaching this for quite a while. This will now be my test case every time I talk about why you should backup your SaaS data.
to delete an entire GSuite account you first need to delete all user accounts, then go to a different page, click on "DELETE MY ACCOUNT", click on two or three boxes which say "ARE YOU REALLY SURE? THIS CAN'T BE UNDONE" and only then everything is gone.
Single user accounts can be recovered for 20 days after being deleted.
So no, this wasn't a "mistake".
Blame the admin, not Google.
This post has been deleted by its author
If you put your data in cloud, do make a backup - either on premises or at a different cloud provider. (like the other commentards above posted).
And ensure that you have the option to recover from accidental deletions.
And should you do backups, do test for recovery, preferably on a different/separate system.
It all costs money, but compared it to total data loss. I'm sure total data loss will be much more expensive. Unless you're a fly-by-night operator.
First, store data in the local machine. Is already there and the company is paying for it, but check the company rules first just in case.
Store the data in at least two cloud services from different providers. Again, check company rules.
And for the love of Pi do offline backups! Even if it just the files you changed on the day, in some external storage device! Just don't take the offline backups home without asking because company rules.
Offline backups can be the difference between paying millions in ransomware or not doing it and just losing the work that was done that day.
But of course remember the golden rule, ask tech support and be polite while doing it.
Zeroth law of working on a company: Don't break the company rules and policies.
Elsewhere other comments make the point about JUST KNOWING about backups (as opposed to ASSUMING that "the cloud takes care of all that").
*
But as someone who has had to set up and manage a regular disaster recovery test from scratch, it's pretty clear that backups, although important, are a long way from the whole risk management story! I wonder if any of the people described in this article have even bothered to read their contract with Google, never mind think about backups or disaster recovery or user involvement in risk management.
*
I know....I'm a dinosaur....all that cool cloud stuff is the answer to all business problems.....until it isn't!!!
No, the dinosaur bit might be more for actually reading a contract.
I can't begin to describe how inept most tech companies are at reading supplier contracts and terms - make them long enough and you could probably ask for their first born and still get sign-off. TL;DR is well alive, with few realising that the "TL" part of that may not be accidental.
That said, you can push it too far. If Adobe ever gets into court for issues with terms they may very well lose the case because of how difficult they have made it to read the terms that effectively define the contract - having to wade through two mazes of links to find the relevant data could be construed as deliberately obstructive..
Nearly everyone here is completely missing the point: the problem is not that the startup's data is gone, the problem is Google doesn't care and is non-responsive.
Had Google just told them in a timely way that they cannot restore their data, we'd all be talking about something else. But Google initially talked like maybe something could be done, and then did nothing.
The startup is just hoping the law suit will make the corporate giant care just enough to act — or at least communicate.
Gsuite admin here. They do have recovery tools if user data in gmail or google drive etc gets accidental deleted - the user-accessible 'bin' being the simplest! But admins can recover that data from the last 30 days regardless, along with user accounts - presumably it's not actually gone from disk till after that. Beyond the 30 days though, they warn the data is likely gone for good.
To delete the entire gsuite account though, that's not a simple fat-finger error, that's a full on failure to read what's in front of you, and doing it anyway including clicking through the warnings. I've never actually done that particular step, but the admin interface is clear when what you're doing will cause permanent data loss, including disabling certain services, and has 'are you sure?' confirmations. I'm not surprised that deleting the whole shebang meant google couldn't just turn it back on and have all the data linked back with user accounts etc. there's a huge amount of different services linked together under the 'gsuite' brand, I'm sure the bits were still there somewhere, but it's a bit like deleting the partition table on a disk, or formatting your last AD server, and expecting the restore to be simple without your own backups.
As a long-time sysadmin, with great power comes great responsibility etc...
And yes, we back up important data via external backups. There's a bunch of cloud-to-cloud backup services, as well as API access to download things, and on the individual level 'google takeout' lets you back up literally everything - including a bunch of stuff you didn't even know google had, though a lot of the data analytics/advertising stuff is turned off for gsuite business/education customers.
Loving all this cloud bashing folks :)
People also don't realise how important backups are when you look at the cloud provider's SLA - with 'cold' Azure or Google storage you're only guaranteed a 99% chance that your read/write will actually work - https://azure.microsoft.com/en-in/support/legal/sla/storage/v1_5/ / https://cloud.google.com/storage/sla
If I had a RAID array or similar only guaranteeing 99% success of a read/write operation I'd be sending it to landfill immediately!
Please check your service agreements. You might be surprised. Neither Salesforce, Office 365, or G Suite have what I would consider a backup for your data. I would define that as something I could use to restore all my data when the caca completely hits the rotary oscillator.
Salesforce has KIND OF a backup, but it takes 6-8 weeks to get ahold of it and costs $10K. And it doesn't restore everything.
MS and google has recycle bin type features, but those are all stored w/in your account. They go away if you (or a bad actor) delete your account.
What I don't like is that they are not open about this. Either they should 100% cover you and support it, or they should very plainly state that backups are your responsibility. Right now they do neither.
As a few have said, using a cloud service (whether SaaS, PaaS or IaaS) doesn’t take away the need to think about recovery processes etc, although recovery testing can be more complicated.
The top level admin who had the rights to do this probably would have had similar abilities in an ‘on premise’ world - delete all the VMs/clusters, delete everything in AD, delete all the passwords, delete the encryption keys used for backups etc etc. Only a delegation of rights that separates rights over separate areas prevents this, but its not uncommon for the top admins to have rights over everything.
I guess maybe the difference is that you could have someone with very little experience as your top level Gsuite administrator. But if it was done deliberately by someone who, say, just got sacked, the fact its a cloud service is not that relevant. The company just has someone else they can try to put the blame on, and possibly less chance of scavenging around for fragments of data...
I know this is a reply to a really old question, but I was doing research for my podcast. (We're covering this event next month on The Backup Wrap-up podcast.)
So just in case someone comes across it, your question is about why they don't keep the data for 51 days, as it says in their policy. That policy applies to deleting your subscription. It's similar to when you delete your subscription to, say, Netflix. The actual deletion will be some time later (usually the end of your billing period). They're even saying that if your subscription lapses, they'll keep your data for 51 days to give you a chance to renew it.
But in this case, the customer deleted the account itself, not the subscription. That is a purposeful act they are then immediately honoring.
I can't believe I didn't find this out until now. Musey quietly dropped its pro se lawsuit 12 days later.
https://dockets.justia.com/docket/california/candce/4:2019cv03864/344456
I knew the lawsuit was without merit because there is nothing in the Google contract that says they provide backup.
I still wish that any service contract (and service description on the website) with an IaaS/PaaS/SaaS vendor would specifically address the situation of backup. Salesforce is probably the most forthcoming, but most SaaS vendors never address the FACT that they are not providing backup for your account. In fact, none of them (as of this writing) even offer it as an additional service.
And if they did, I would question its validity due to what happened to OVH's customers who paid for what they thought was an offsite backup. The service description said backups were triple-replicated and the service agreement said backups were "physically isolated." Turns out that meant the server was in a different part of the datacenter. You know... isolated.
Please make backups of your IaaS/PaaS/SaaS data, and know exactly where those backups are going.
Disclaimer: I do work for a backup as a service vendor, but that doesn't decrease the validity of what I said above.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
