back to article Firm fat-fingered G Suite and deleted its data, so it escalated its support ticket to a lawsuit

An interior design tools startup called Mosss on Wednesday sued Google to get it to restore its data after someone at the startup accidentally deleted the firm's G Suite account. In a pro se lawsuit [PDF] filed in US District Court in Oakland, California, Mosss, under its previous corporate name, Musey Inc., asked Google to …

  1. _LC_
    Facepalm

    Place files in cloud - surprise - can't reach'em anymore

    *aww*

    1. big_D Silver badge

      Re: Place files in cloud - surprise - can't reach'em anymore

      Yeah, that's what backups are for.

      All your eggs in one basket. That is a lesson nobody seems to learn. Just because it is on a cloud somewhere doesn't mean it is safe and doesn't mean you don't have a duty to back it up.

      1. Law

        Re: Place files in cloud - surprise - can't reach'em anymore

        While I agree putting all your eggs in one basket is a bad idea, you'd expect an enterprise cloud solution to be equivalent to an in-house system... So backed up and recoverable within a day of catastrophic failure / mistakes.

        1. Anonymous Coward
          Anonymous Coward

          Re: Place files in cloud - surprise - can't reach'em anymore

          "While I agree putting all your eggs in one basket is a bad idea, you'd expect an enterprise cloud solution to be equivalent to an in-house system"

          and you would be wrong.

          1. ckm5

            Re: Place files in cloud - surprise - can't reach'em anymore

            Cloud systems are no better or worse than in-house systems - probably better than most in-house systems I have seen, actually, and def. way more secure.

            You still have the responsibility to make sure everything is correctly setup and have recovery procedures for emergencies, it is not the responsibility of the cloud provider. If the story was "in-house IT system crashes, deletes all data with no backups" it would be a non-story as that is SOP for most small businesses.

            In this case, it's just a PEBKAC.

            1. chuckufarley Silver badge

              Re: Place files in cloud - surprise - can't reach'em anymore

              "Cloud systems are no better or worse than in-house systems - probably better than most in-house systems I have seen, actually, and def. way more secure."

              Perhaps you should consider finding a new group of people to work with.

            2. Huw D
              Trollface

              Re: Place files in cloud - surprise - can't reach'em anymore

              "Cloud systems are no better or worse than in-house systems - probably better than most in-house systems I have seen, actually, and def. way more secure."

              You've obviously not seen systems run by professionals then ;)

              1. sabroni Silver badge

                Re: You've obviously not seen systems run by professionals then

                You're calling Google amateurs?

                Get over yourself.

                1. Huw D

                  Re: You've obviously not seen systems run by professionals then

                  a) I'm referring to the people running the in-house systems that the poster was commenting on being amateurs "probably better than most in-house systems I have seen, actually, and def. way more secure". However, seeing as you seem to have misinterpreted my comments then

                  b) Google are merely the hosts, providing the software and infrastructure. There still has to be an administrator, well, administrating. That's the problem not Google.

                2. chuckufarley Silver badge

                  Re: You've obviously not seen systems run by professionals then

                  I have never heard of anyone filing a lawsuit to get an off-site data storage company to deliver a back up. Never once has an LTO tape library asked to see a court order before it restored data from a tape. I am very confident that the AES256 encryption algorithm has never consulted a lawyer before encrypting a file.

                  I am sure the people at Google get paid. That is all they need to be called professionals. That doesn't mean relying on a single account from a sole cloud provider to store or backup all of your company's data is a good idea. We are supposed to know the tools of our trade and then use the right tool for the job. Cloud computing is a great resource but if it is the the tool in your box then I say you are doing IT wrong because doing IT right will never create a single point of failure.

                  1. Huw D

                    Re: You've obviously not seen systems run by professionals then

                    Professional (paid to do IT) <> Professional (paid to do IT and actually knows WTF they're doing, has standards),

                    I'm using the latter. I think we're on the same page.

                    1. CrazyOldCatMan Silver badge

                      Re: You've obviously not seen systems run by professionals then

                      Professional (paid to do IT) <> Professional (paid to do IT and actually knows WTF they're doing, has standards)

                      AKA - the MCSE[1] effect..

                      [1] Must Call Someone Experienced. I'm sure that there are some people with MCSE qualifications who know what they are doing but they are far outweighed by people who have gone through boot-camp[2] style MCSE courses and don't actually have a clue about IT.

                      [2] Not the Mac thing..

                      1. big_D Silver badge

                        Re: You've obviously not seen systems run by professionals then

                        Yes, I've done a lot of MCSE training and exams in the last couple of years, after 35 years working in IT* and using Windows since 1987. The MCSE level of training and the exams are for n00bs, the course touched on about a third of the knowledge required to effectively run an MS installation.

                        It doesn't teach you to think laterally, it doesn't teach you to think outside the box and it is no replacement for experience.

                        * I was looking for a new job and decided that having a couple of qualifications would look good on the CV, then I actually took the courses and exams and was shocked how basic the knowledge required was.

                    2. jabuzz

                      Re: You've obviously not seen systems run by professionals then

                      Here is an employment tribunal judges view (not mine I just happen to know them) on "professional". If there is not a professional standards body that can kick you out of the profession and stop you ever working in the profession again if you turn out to be an incompetent numpty you are not a professional.

            3. Flywheel
              Facepalm

              Re: Place files in cloud - surprise - can't reach'em anymore

              > PEBKAC

              I agree, but how would you go about actually deleting a Gsuite account - surely there must have been multiple "are you sure/are you really sure/you know it's gone forever if you click YES" messages? And the unnamed muppet still went ahead?

              1. 142

                Re: surely there must have been multiple "are you sure/are you really sure/you...

                They didn't necessarily actively "delete" the account.

                I'm not sure about Gsuite specifically, but these sort of linked accounts can end up orphaned or mangled in certain edge-cases, such as removing the wrong user, etc. especially at smaller scales where it may have evolved from someone's personal account.

                1. solv

                  Re: surely there must have been multiple "are you sure/are you really sure/you...

                  You sure don't know about G Suite.

                  The only way this story makes sense is if they actually DID delete their entire account

                  If it was just a user account you would have plenty of time to restore the user and wouldn't need to call Google and escalate it it to try and get the user back - the built in admin can do that just fine.

                  I ALWAYS tell my customers they should be paying for a third party backup solution for 0365 and G Suite - because despite those guys having extremely good infrastructure and security, they can't protect against internal stupidty/and or malicious actions of either an employee or a bad actor.

                  1. ridley

                    Re: surely there must have been multiple "are you sure/are you really sure/you...

                    As the account was on one name and the company is now called by another I suspect they were trying to change the company name in the account.

              2. LewisRage

                Re: Place files in cloud - surprise - can't reach'em anymore

                Dave : "Steve I need you to delete the old/test/spar/unwanted GSUITE account completely"

                Steve : "No problem Dave, I'm just about to leave for the weekend but I'll get it done right quick now before I leave"

          2. Law

            Re: Place files in cloud - surprise - can't reach'em anymore

            "and you would be wrong."

            Judging by your comment and all the down votes, I guess not.

            I am but a humble dev, forgive my ignorance. :)

            1. jake Silver badge
              Pint

              Re: Place files in cloud - surprise - can't reach'em anymore

              Just promise you won't try to do ops ... and relax, have a homebrew.

        2. hoola Silver badge

          Re: Place files in cloud - surprise - can't reach'em anymore

          That is where you are wrong, why would an enterprise cloud solution automatically give you backups? If delete stuff in, on your own servers or in a cloud offering without any form of backup then unsurprisingly it is gone. This is where many companies (including really large corps) fall down. There is a belief that just because your data is on a cloud service it is protected. There will be some sort of protection in place but only so that the provider can sort out their own problems, not yours.

          I have never understood why it is deemed to be acceptable to use Azure with the Azure Backup. If you subscription gets compromised it is all gone, just is if you have no off-site backup provision for a real data centre.

          1. Probie

            Re: Place files in cloud - surprise - can't reach'em anymore

            For that very reason you should have at least two subscriptions, one for use one for backup/bc

        3. phuzz Silver badge

          Re: Place files in cloud - surprise - can't reach'em anymore

          "you'd expect an enterprise cloud solution to be equivalent to an in-house system... So backed up and recoverable within a day of catastrophic failure / mistakes."

          Depends, are you paying for the backup service? If not, you might not be backed up.

          Still, you did check the small print before you signed up right?

        4. LucreLout

          Re: Place files in cloud - surprise - can't reach'em anymore

          you'd expect an enterprise cloud solution to be equivalent to an in-house system... So backed up and recoverable within a day of catastrophic failure / mistakes.

          It is, if you do it right. We have several availability zones all synchronised, acting as hot backups, we use the cloud providers backup for data (CI/CD for code with github and local hardware acting as code backups), and we still retain our own daily data backup in our own equipment.

          Of course, if you have inexperienced dreamers or fools setting up your backup and retention policies, it simply doesn't matter if you're in the cloud or using local hardware, it won't end well. Experience cannot be replaced with keenness and ego, or talent as the kids describe it because there's no short cut to experience.

      2. k9gardner

        Re: Place files in cloud - surprise - can't reach'em anymore

        The service people think they're paying for IS the security of having the date in the cloud, where it will always be backed up by service provider policy. Google is simply behaving like the near-monopoly that it is. We've also had cases of inadvertently deleted data that "cannot be recovered." Anybody else would have to recover it. Not Google.

    2. 9Rune5

      Re: Place files in cloud - surprise - can't reach'em anymore

      If you're unable to reach your data in the cloud, you just need a bigger ladder.

  2. Anonymous Coward
    Anonymous Coward

    They expected Google to keep it after it was deleted?

    They only do that with your personal information they use for advertising. Your own data is of no value to them so of course it gets binned the moment you delete it!

    Timely article on CNBC showing exactly this - if you delete your Gmail Google keeps the purchase history they stole out of your emails: https://www.cnbc.com/2019/05/17/google-gmail-tracks-purchase-history-how-to-delete-it.html

    1. Jamie Jones Silver badge

      Re: They expected Google to keep it after it was deleted?

      No. That CNBC article is making an issue over nothing.

      The "purchase" section is based on receipts in your *current* gmail inbox. Delete the email, the receipt goes.

      As the article itself says:

      "But there isn’t an easy way to remove all of this. You can delete all the receipts in your Gmail inbox and archived messages. But, if you’re like me, you might save receipts in Gmail in case you need them later for returns. There is no way to delete them from Purchases without also deleting them from Gmail -- when you click on the “Delete” option in Purchases, it simply guides you back to the Gmail message."

      It just seems the person who wrote the email wants it to not exist in "purchases" but still exist in his gmail, despite "purchases" simply being a live summary of gmail receipts.

    2. Charlie Clark Silver badge

      Re: They expected Google to keep it after it was deleted?

      It's quite possible that there are compliance aspects to consider inasmuch as it Google may well be obliged to delete all data, with a reasonable period, when such an account is closed. If so, I'd expect this to be in the contract or the T&C's. If it is, then I wouldn't reckon much for their chances.

      1. Anonymous Coward
        Anonymous Coward

        Re: They expected Google to keep it after it was deleted?

        Google's contracts were written by Google, do you think they are obligated to do anything under the terms of their contracts? The only obligations they'll have would be legal ones that depend on which country's laws are applicable.

        It isn't like you can call up Google and tell them you want to negotiate a contract for GMail services. They will tell you it is on the web site, and if you don't like the terms you can get elsewhere.

        1. Charlie Clark Silver badge

          Re: They expected Google to keep it after it was deleted?

          So what? Nobody forced the company to use Google and the principle ofcaveat emptor applies, especially when you're potentially betting the house on a service.

          If you feel that a contract is one-sided, you normally also have recourse the challenge the contract in court. But basically this is a SNAFU that the company hopes Google will fix for them.

          The countercase is, of course, you use GSuite for your very important and confidential data and then decide to move to another service. Surely, in such a case you'd expect that Google would make the data irretrievable within a short period of time after the contract was cancelled and that you'd take them to court if that wasn't the case?

  3. Elregouk

    How do you accidentally delete a whole G Suite account? You have to jump through so many hoops it's like being at Crufts

    1. Steve Goodey

      As per wittertainment.

      How do you accidentally delete a whole G Suite account? You just accidentally delete a whole G Suite account.

      1. BebopWeBop

        Re: As per wittertainment.

        Very Kermode

    2. jake Silver badge

      Point of order ...

      ... there is only one hoop in the Crufts agility course. And it's not really a hoop, it's more of a framed lifebuoy.

    3. Anonymous Coward
      Anonymous Coward

      What if... you don't pay the bill? And ignore the notifications?

      Maybe someone who left the business dealt with it, and it wasn't handed over - credit card cancelled, emails ignored?

    4. LewisRage

      Zak : "Hey Bob, I need you to delete the GSUITE account that we set up as a test"

      Bob : "No worries Zak, I was just about to leave for the weekend but I'll get it done before I go"

  4. Mike 16

    Outsourced

    Google support has long been outsourced to the vicinity of Epsilon Eridani,with the inevitable increase in response time. (See long delayed echoes for background)

    1. Terje
      Alien

      Re: Outsourced

      I would actually have guess on the three first planets in the Sirius Tau system.

      1. fedoraman

        Re: Outsourced

        " Share and Enjoy! "

  5. b0llchit Silver badge
    FAIL

    Conflicted who and what to bash

    This is a story, where I get so very much conflicted of who and what to bash.

    - A business having no backups of its own business data; must be SNAFU.

    - A simple click to delete all access to whom exactly?

    - A non-responsive customer service; is there any other?

    - A lawsuit to determine who to blame for stupidity; lawyers must make a living too.

    - A cloudy venture entrusted with your data; its no longer your data.

    At least, they used google. Then they may probably find all of their designs, conversations and sales eventually used in advertising, somewhere. That is actually a good idea, they should write a bot scraping google for their data. There must be something! Maybe in the cookies of their browser and certainly loads of incriminating data in the (google-shared and transparent) cache of Chrome.

    1. Carpet Deal 'em

      Re: Conflicted who and what to bash

      A lawsuit to determine who to blame for stupidity; lawyers must make a living too.

      The lawsuit's pro se; ie, the plaintiff is self-representing. They were definitely being idiotic(the request for subpoena mentions private medical documents as amongst the lost data, for instance), Google's a bit out of line here if they took weeks to inform a paying customer their data was deleted(during which time they probably charged the company at least once while giving the impression access would be restored).

      1. ElReg!comments!Pierre

        Re: Conflicted who and what to bash

        if they took weeks to inform a paying customer their data was deleted

        Well they didn't. The customer terminated the account, and thus became a non-customer. While Google gives you a grace period when you accidentally delete a document, they may not extend the courtesy to the accidental deletion of a paying account.

      2. wcpreston

        Re: Conflicted who and what to bash

        I do agree that Google should have notified them quickly that there was no hope. Google does not have a backup of your entire account. As with other SaaS vendors, backups are your responsibility. Feel free to consult your service agreement.

  6. unredeemed

    Gosh.. If only there were 3rd party backup products that can back up an organization's cloud data... If only.

    "Put it in the cloud," they say. "It's always online," they say... Well it's not backed up apparently!

    Fact, cloud providers make your data available as a best effort. They don't fix stupid.

    1. chuckufarley Silver badge
      FAIL

      If only...

      ...There was a way to do on-site backups. If only there was a way to then make a backup of that backup. If only there was a company that could store that copy for you off-site and return it to your business as needed.

      If only this has been the standard model for backing up data for over 30 years. Then perhaps this news story would never have been written.

      All sarcasm aside: You data is like your child in that you are ultimately responsible for it's well being. Choosing not live up to that responsibility doesn't make it go away.

      1. Doctor Syntax Silver badge

        Re: If only...

        "Choosing not live up to that responsibility doesn't make it go away."

        In this case it did.

      2. Baroda

        Re: If only...

        'if only there was a company that could store it off site and return it to your business as needed'.

        Around 15 years ago I went as part of a group to support a third-party customer who had hired a dedicated DR company.

        The DR company's dedicated DR site was in another country in a site that was entirely surrounded by a grass-covered embankment - bomb proof(?). The 3rd party who used the DR company wanted to test the end-to-end process. They picked a nice simple NT server with a couple of DBs on it as the first step.

        They allocated a whole weekend to this - did they know something we didn't?

        On Saturday morning, the DR folk who *knew* we were coming did not have the required tapes ready.

        They then took around two hours to find them.

        They then realised after more hours of faffing about that they could not provide the contractually required server until late the same day. Despite having dozens and dozens of servers 'ready' on site.

        Long before they found and allocated us one we had left in disgust.

        On Sunday morning we returned and restored the NT server, recovered the DB's and completed local testing to our customer's satisfaction.

        After a short booze-cruise by the driver, the boot fully stocked and the car's front wheels making occasional contact with the ground, we headed back to the UK.

        The driver generously handed over some of his liquid stock to us passengers, so not an entirely wasted trip.

  7. Anonymous Coward
    Anonymous Coward

    Under a cloud

    I have always suspected google has no viable backup and recovery process in place.

    Remember a cloud is just vapour, after it rains there is nothing but blue sky left.

    1. oiseau
      FAIL

      Re: Under a cloud

      ... a cloud is just vapour ...

      Indeed ...

      I've been saying this very same thing for years now and been mocked and criticized for it.

      But it'll be -30º C in hell before I even think about putting anything in the cloud, be it Google's or anyone else's, no matter how convenient it may be.

      Old?

      Yes, that's why I know better.

      O.

      1. inquisitive2014

        Re: Under a cloud

        O, If you choose not to use a cloud solution then I am assuming you would use an on premise solution. If you delete files from an on premise solution there is a strong chance that you won't be able to get them back.

        As someone with a long history in IT you would know that this story indicates two failings which apply equally to on-premise or cloud scenarios:

        1. How did they "accidentally" delete their account? This is like del *.* on a root account for an on premise system - which has been done many times.

        2. Why did they not have a backup that they could restore from? HA and DR principles don't change because you move to the cloud.

        Hope you have a nice day.

        1. DavCrav
          Headmaster

          Re: Under a cloud

          "O, If you choose not to use a cloud solution then I am assuming you would use an on premise solution. If you delete files from an on premise solution there is a strong chance that you won't be able to get them back."

          The only comment I have is that 'premise' is a supposition, 'premises' is a location.

      2. Anonymous Coward
        Anonymous Coward

        Re: Under a cloud

        Me too....and then I dropped my IPhone and discovered that to use ‘Find my phone’ I was forced to sign (for the first time ever) to sign into something called ICloud, and now It tries to steal my data or divert my instructions pretty much all the time. It’s like dealing with ransomware merchants.

    2. Toby Poynder

      Re: Under a cloud

      Erm vapour is transparent. When you look at a nice fluffy white cloud you're actually looking a water droplets.

      1. Anonymous Coward
        Anonymous Coward

        Re: Under a cloud

        All this talk of cloud is giving me the vapours.

      2. Anonymous Coward
        Anonymous Coward

        Re: Under a cloud

        And ice crystals.

        1. whileI'mhere

          Re: FailCEO

          "And ice crystals."

          All hail our cloudy providers.

      3. The First Dave

        Re: Under a cloud

        "Erm vapour is transparent. When you look at a nice fluffy white cloud you're actually looking a water droplets."

        You are thinking of steam - that is transparent, but vapour is not.

        1. the spectacularly refined chap

          Re: Under a cloud

          No, they're both invisible gases. Steam is water directly converted to a gas. Water vapour is water dissolved in air whereupon it assumed a gaseous state.

          If you're thinking of the stuff you see coming out of e.g. your kettle that is microscopic droplets of liquid water as the vapour begins to condense. They are small enough to remain airborne via Brownian motion but the reason you can see it is due to the difference in refractive index between the water drops and the surrounding gas.

    3. phuzz Silver badge

      Re: Under a cloud

      "I have always suspected google has no viable backup and recovery process"

      The surprising thing is that they don't have one and charge extra for it. You'd have thought they'd have had loads of business for a built-in backup system. (Amazon are quite happy to sell their customers a backup service.)

      No, the real fuckup here is assuming your data is being backed up, without actually checking the contract.

    4. ridley

      Re: Under a cloud

      Actually....

      A cloud isn't vapour, water vapour is transparent. The cloud is actually tiny liquid water droplets.

      On a hot day you can see "little fluffy clouds" forming above thermals. You cannot see the water vapour rising but you see the cloud where that vapour has cooled enough to condence back into water.

    5. wcpreston

      Re: Under a cloud

      It's not in the service agreement for them to do so, so why should they? None of the major SaaS vendors (Salesforce, Microsoft, Google) have backup your data in a way that any backup person would recognize as a backup.

      Salesforce has a restore service that generates CSVs of your account. It takes SIX TO EIGHT WEEKS, costs $10K, just to get a bunch of CSVs that you can then use to restore your data. Metadata is gone forever. Even they don't recommend you to use it.

      All of MS and Google's documented data protection features leverage the recycle bin or versioning, and thus are not a backup. MS does have a delayed replicated copy of your entire account they can use in case of THEIR disaster, but they do not make it available in a scenario like the one in this article.

      Please look at your service agreements. Look for the words backup, restore, recovery, data protection, etc. I think you'll be surprised.

  8. jake Silver badge

    Fact is ...

    ... the only person who can control your data is you. And sometimes not even you. Trusting it to a third party is foolhardy at best.

    I'm no fan of the gootards, but this Mosss outfit really needs to re-examine it's priorities if it wants to stay in business.

    1. Anonymous Coward
      Anonymous Coward

      Re: Fact is ...

      Jake,

      "... this Mosss outfit really needs to re-examine it's priorities if it wants to stay in business."

      For clarity, on the basis of the information given in this article ........ There is no ̶S̶p̶o̶o̶n̶ Business !!!

      1. jake Silver badge

        Re: Fact is ...

        That's hyperbole. I've seen a several businesses lose everything due to fire ... paper records, computer records, inventory, tools and tooling, the building, the company car & delivery vehicle, name it ... and still stay in business.

        The businesses? A bakery, a machine shop, a boat dealership/repair shop and a startup making an application specific piece of medical gear. Insurance helped for all four.

        1. werdsmith Silver badge

          Re: Fact is ...

          Because a really good business is about it’s people. The world needs to take care to not forget this,

  9. Anonymous Coward
    Anonymous Coward

    I've never felt comfortable about "The Cloud", for exactly this reason. Who's responsible for backing up and insuring a restore from backup policy and procedure? Trusting that to some outsourced clown, just isn't going to work. There should always be a backup and restore process that is internal and verified to be working. Trusting Google with all of your data? HUMPH....

    1. Yet Another Anonymous coward Silver badge

      Or even external and tested and verified.

      Just because it's internal you don't trust the tape drives you didn't make

      1. Anonymous Coward
        Anonymous Coward

        Bit of a long story from a land far away, but...

        I worked at a place where the CEO wouldn't approve a backup agent for the Oracle *spit* DB the LoB ran on once he knew we could sqldump and restore the DB for free. The business accepted a 24hr recovery point, so every night cron would dump the data to an nfs share in the backup server room where it would be archived to tape. Simple as you like.

        Logs and email confirmation were configured, a restore was tested every 6 months and visually checked on an adhoc basis. Worked fine for a few years, then one day when it needed to recover for real, it didn't work.

        Looking at the SQL everything looked fine, but by about halfway in, hints of file corruption started. By the end, it was just line noise.

        Lots of overtime for hundreds of the guys manually re-entering just shy of 3 months' worth of records, and operating at a reduced capacity for almost a month on the backup paper system. The IT manager got the old 'choose the sword or get the bullet' and the CEO was out within the year. Plus we made the national news, and international industry-specific news.

        Winning bigly!

        1. Doctor Syntax Silver badge

          Re: Bit of a long story from a land far away, but...

          "The IT manager got the old 'choose the sword or get the bullet'"

          Did he have an audit trail of the "no, you can't have it" correspondence? If so it should have been a silver bullet - or even golden.

          1. GFK1

            Re: Bit of a long story from a land far away, but...

            You think it was the CEO who proposed doing an sql dump instead of paying for the Oracle tool?

          2. vtcodger Silver badge

            Re: Bit of a long story from a land far away, but...

            "Did he have an audit trail of the "no, you can't have it" correspondence?"

            Sure. The audit trail info was right there in the data ba ... Oh ...

    2. werdsmith Silver badge

      “The Cloud” is broad definition though. This little start up was using G-Suite which is nothing like an Enterprise cloud service where a business might run their own IaaS infrastructure(s) with their own ring-fenced network(s) containing hundreds of their own VMs or even their own hypervisor to create their own VMs. Along with their own Enterprises applications and BYOL licences. They would also have their own firewalls, routers etc, so “the Cloud” is effectively the new data centre, but without the need to have people racking and patching.

      1. Anonymous Coward
        Anonymous Coward

        G-suite is googles equivalent of office 365, Callander, office suite, booking rooms, document store with versioning etc. So it's where a business is to store their documents and collaborate. This is one part of the 'cloud', and is an enterprise product.

        1. werdsmith Silver badge

          Like I said “the cloud” has a broad definition. And g suite is no ERP.

        2. Jonathan Richards 1
          Facepalm

          Where's that, again?

          Nae Callander, shurely?

          1. Anonymous Coward
            Anonymous Coward

            Re: Where's that, again?

            "Nae Callander, shurely?"

            Typing on the phone, swipe, didn't notice it until after edit period, was waiting for someone to say something :)

    3. MachDiamond Silver badge

      "Who's responsible for backing up and insuring a restore from backup policy and procedure?"

      The questions gets even murkier when an online account is closed. Is any cloud provider responsible after you close your account?

    4. Tim99 Silver badge
      Coat

      The backup bit is easy. It's the restore that's troublesome.

  10. Doctor Syntax Silver badge

    Whilst it was pretty dumb - but for many SOP - not to maintain their own backup a lawsuit seems an effective way of getting the attention of support. It doesn't necessarily mean they'll get their data back but it'll be interesting to see if they do.

    1. Jamie Jones Silver badge
      Happy

      I've written all sorts of scripts and blocks to disable/remove tracking things on my system, but still, every time I get the (almost constant) "You must review and acknowledge our privacy agreement" popup from google, I'm not convinced:

      "Nice try Google, but you still know who I am - don't try and pretend that my simplistic tidyups fool you", I mutter to my screen with a grating sneer.

      Oops, did I write that last paragraph out loud? :-)

  11. Anonymous Coward
    Anonymous Coward

    This is exactly why Amazon use their own machines not a cloud.

  12. Stuart Castle Silver badge

    The company should have had backups. That said, they were likely not technically minded, so probably thought that the data being held on Google's servers should be enough. After all, nothing would be able to take out Google, right? The problem is, it doesn't need something to take Google out. All it needs is for someone to push the wrong button, either inside Google or (as in this case) outside it.

    That said, do Google allow backups of G Suite? I don't see why they wouldn't, but I don't have access to the admin console of a G Suite account, so don't know what they offer..

    1. Anonymous Coward
      Anonymous Coward

      do Google allow backups of G Suite?

      Not sure about the rest of gsuite, but for email they have imap, so you can back up using that. It's pretty easy to set up. And I'd be surprised if their file storage thing didn't have an API that you could use to scrape/backup your stuff (I'm assuming they're not going to allow things like rsync).

      1. Jamie Jones Silver badge

        Not sure about g-suite either, but they do have "google takeout" which lets you backup supposedly *ALL* your google related data...

        My last google-takeout download compressed to a 5.6GB tar file...

        1. AMBxx Silver badge

          I use multcloud to move data between cloud accounts. At the moment, I just move stuff between personal and business OneDrive accounts, plus a little bit to DropBox. Free tier enough for me, but the paid thing is only a few pounds per month to shuttle your data all over the place. You still have to pay for the storage with MS, DropBox or Google, but it's almost as good as backup if you really don't want anything local.

    2. Warm Braw

      thought that the data being held on Google's servers should be enough

      When you start a business, there are lots of "non-core" responsibilities you have to take on: tax, employment law, health & safety, planning, consumer rights, etc. Even if you don't immediately run into them, you have to understand them them to the extent they affect your business plan. Even if you decide to ignore them, you need to fully understand the risk of doing so. If your approach to critical business data is "What could possibly go wrong?", I'm not sure you should be in business at all. Certainly, if I'd been asked to contribute to the $1.5M investment, I'd have wanted to find out a bit more about how the business intended to operate: the role of investors is not just to provide money, but scrutiny and experience.

    3. vtcodger Silver badge

      That said, do Google allow backups of G Suite?

      Good question. And even if they do, how easy are the backups to do? I recall that about the third thing Google did when in bought Blogger many, many years ago was to delete the capability to load Blogger websites by ftp. So much for any possibility of easily maintaining a local copy of your website. (Or of moving it quickly to another host).

    4. solv

      "That said, do Google allow backups of G Suite? I don't see why they wouldn't, but I don't have access to the admin console of a G Suite account, so don't know what they offer.."

      Yes, various options. To automate it, the best way is to use their API.

      There are plenty of services out there that are very reasonably priced that will do it several times a day and let you restore email and drive contents right back where they were with a couple of click. ackupify (which was bought out by Datto) is the one I put people on. There is also SpinBackup which is popluar

    5. ridley

      There is a "vault" that keeps backups of deleted files etc.

    6. Anonymous Coward
      Anonymous Coward

      Yes

      Yes, Google allow you to backup G Suite, there are many such 3rd parties that will do it for you. Most people we speak to have no idea that there aren't any backups for them to access on G Suite. They assume Google will take care of it all for them if they cock something up. Same with Office 365.

  13. Blackjack Silver badge

    So...

    No one ever does offline backups anymore?

    1. midcapwarrior

      Re: So...

      The service provides an option for backups.

      It's not free but it's an option.

      The problem is they deleted their master account which would include whatever back up service they were using.

      Most cloud providers make you step through multiple steps to delete an enterprise account up to and including multi factor validation.

      Not sure how Google would win in this situation, ignore the delete request and be nailed for not deleting.

      Except the delete request and be nailed for accepting the request.

      1. 142

        Re: So...

        > Accept the delete request, lock the account, and put a stay on final execution for a month or two.

        If anything, that covers Google's ass in case they have screwed up authentication at their own end, and accepted an invalid delete request.

        1. Yet Another Anonymous coward Silver badge

          Re: So...

          > Accept the delete request, lock the account, and put a stay on final execution for a month or two.

          And be responsible for the data not being deleted ?

          Suppose I pay Iron mountain to destroy some paperwork and I find that they have been hanging on to it for a couple of months "just in case" I might get a little unhappy

      2. wcpreston

        Re: So...

        Can you provide any links on this extra-cost G Suite backup?

  14. herman

    If their designs were any good, they should be able to buy their schtuff from China any time now.

    1. keith_w

      It has to be a good design to be counterfeited in China?

  15. man_iii

    Tested backups

    All this nonsense about running entire business on cloud with no backups should worry more IT folks given how frequently cloud platforms tend to fail.

    Even multiple cloud providers can experience near simultaneous regional outages and having things in a S3 bucket and github project doesn't cut it as being "backed up".... it is how data exposure and breaches happen....

    Sounds like the olden days grey and white bearded druids were right about managing your own IT inhouse if your business depends on it!

    1. sabroni Silver badge

      Re: inhouse

      Sounds like the olden days grey and white bearded druids were right about managing your own IT if your business depends on it!

      FTFY.

  16. chivo243 Silver badge
    Facepalm

    If the plod wanted their data

    I'm sure Google could easily provide it. I also wonder what the backup plan was(n't). I think there is enough of "blame sandwich" for every party involved to have a bite.

  17. Mike007 Bronze badge

    Google will have a backup, however it will probably be a daily tape archive. They might not have the ability to easily restore a single account, and even if they do it will probably be a lot of effort/cost.

    If it had been a major account they likely would have restored it from the last daily snapshot, possibly sending a $XXX,XXX bill... however they certainly will not want to set a precedent saying that they will do a tape restore of a small account for free.

    Why do they have to go to tape? I am sure that a lot of the time when you delete a file or email it is just marked as deleted then only garbage collected after a certain period of time (hence them saying they might be able to restore), however doing that for an entire domain and all sub-accounts is a lot more complex and prone to error. If they were able to do a partial restore... the lawsuit would be over the fact that they agreed to restore the data but something was missing!

    Of course the next question will be, how long do they keep their daily tape backups for before overwriting them? If they wait a month before responding that there is no Federal jurisdiction and it needs to be referred to a State court then the data problem could disappear by itself.

    1. Anonymous Coward
      Anonymous Coward

      The backup data that they keep on tape is encrypted, and when you want to delete the data, they delete the encryption key.

  18. jason 7

    I call BS!

    Google keeps everything forever. They just don't want to admit it.

    My other halfs Gmail had a brain fart a year ago. Suddenly 10 years of deleted emails all came back, including all the deleted binned spam.

    This was all stuff that was fully deleted as far as she knew. Took ages to clean up.

    1. Hans 1
      Paris Hilton

      Re: I call BS!

      So, we have two options, here, either:

      1. "other half" knows how to delete and expunge email in a mailbox AND Google keeps everything, even deleted stuff - shit, this would mean they have extended their IMAP protocol implementation to "not actually delete" stuff, simply move it elsewhere.

      2. "other half" DOES NOT know how to delete and expunge email in a mailbox.

      I am very sorry for your other half, but option 2. is so much more likely ...

      1. jason 7

        Re: I call BS!

        So she clicks 'Delete' then she go to the Bin and clicks "Delete Forever". Some emphasis on the FOREVER part there.

        Well I never, seems she does know how to delete stuff from Gmail!

        And then 10 years later it all comes back.

        You don't have to be such a 'typical IT guy' about it.

    2. 142

      Re: I call BS!

      This is the kind of thing that can happen with a misconfigured mail client. The mails were most likely being copied elsewhere, like an old forgotten outlook express, but without the deletions on Gmail being properly mirrored, and then suddenly resync'd for some reason. I've seen this kind of thing happen a few times... It's an absolute pain in the ass to tidy up, as you found.

      1. jason 7

        Re: I call BS!

        In this instance no other client or service is or was being used. Just Gmail.

        They keep it all.

  19. Andrew Jones 2

    Hmm deleting data accidentally is one thing, and one could reasonably expect Google could recover that. But the suggestion here, is that someone has deleted the actual Google account in its entirety (accidentally or otherwise). Expecting that an entire account can be recovered - in spite of the warnings that use words like "irreversible" and "permanently" is a bit unreasonable in my opinion.

  20. fredesmite
    Mushroom

    Remember - Cloud computing

    IS NOTHING MORE THAN SOMEONE ELSE"S COMPUTER that other people are using and you think they care enough to keep your data safe

    1. Yet Another Anonymous coward Silver badge

      Re: Remember - Cloud computing

      In this case they did keep the data safe, the client ordered them to delete it and they did

  21. Anonymous Coward
    Anonymous Coward

    What use is Google's statement about backups?

    Google states that it is able to recover data when you get hold of them quick enough, but like all bigger US companies they use self-help and forums to explicitly avoid talking to customers.

    Ergo, their backup promises are worth sweet FA, because by the time you have dug through all the defensive layers to NOT to talk to customers, your data will be gone anyway.

    Cloud computing - for when you want your data to evaporate...

    1. fozzy73

      Re: What use is Google's statement about backups?

      There are backups. If you delete mails or drive files you have 25 days to recover them. if you delete a user account you have 20 days. Drive files which have been edited have some 100 versions with them.

      Deleting an entire account has to be done on purpose (see post below).

    2. wcpreston

      Re: What use is Google's statement about backups?

      That is not what that google statement says. It says they can restore a deleted USER, because a deleted user goes into a recycle bin type area. This is not a deleted user; it's a deleted account. BIG DIFFERENCE.

      This is not a cloud computing problem. This is a "user assuming their SaaS service was backing them up w/o verifying that" problem. I've been preaching this for quite a while. This will now be my test case every time I talk about why you should backup your SaaS data.

    3. herman

      Re: What use is Google's statement about backups?

      Clouds: Sometimes your data evaporates and sometimes your data rains down on other people.

      O'l Mick said it best: Hey! You! Get offa my cloud!

    4. MachDiamond Silver badge

      Re: What use is Google's statement about backups?

      "Cloud computing - for when you want your data to evaporate..."

      And rain that data far and wide.

  22. fozzy73

    This wasn't a "fat-fingered" mistake.

    to delete an entire GSuite account you first need to delete all user accounts, then go to a different page, click on "DELETE MY ACCOUNT", click on two or three boxes which say "ARE YOU REALLY SURE? THIS CAN'T BE UNDONE" and only then everything is gone.

    Single user accounts can be recovered for 20 days after being deleted.

    So no, this wasn't a "mistake".

    Blame the admin, not Google.

  23. Mark Manderson
    Thumb Up

    classic case of oh its now in the cloud, i no longer have to backup or give a flying fox about MY company data.

    company of fuckwitts, good on google.

  24. David Gosnell

    Fat fingers?

    Seems to be a bit of an issue with the typo-ridden company, even down to their chosen name.

  25. Anonymous South African Coward Bronze badge

    Another possibility would be that it was a 3rd party who've managed to hack the account and deleted it.

    Not sure if GSuite account have 2FA on by default, or that you have to turn it on yourself.

    1. Anonymous Coward
      Anonymous Coward

      The company has FA now, so 2FA can't be far off :).

    2. This post has been deleted by its author

  26. Anonymous South African Coward Bronze badge

    If you put your data in cloud, do make a backup - either on premises or at a different cloud provider. (like the other commentards above posted).

    And ensure that you have the option to recover from accidental deletions.

    And should you do backups, do test for recovery, preferably on a different/separate system.

    It all costs money, but compared it to total data loss. I'm sure total data loss will be much more expensive. Unless you're a fly-by-night operator.

    1. Blackjack Silver badge

      For the love of Pi

      First, store data in the local machine. Is already there and the company is paying for it, but check the company rules first just in case.

      Store the data in at least two cloud services from different providers. Again, check company rules.

      And for the love of Pi do offline backups! Even if it just the files you changed on the day, in some external storage device! Just don't take the offline backups home without asking because company rules.

      Offline backups can be the difference between paying millions in ransomware or not doing it and just losing the work that was done that day.

      But of course remember the golden rule, ask tech support and be polite while doing it.

      Zeroth law of working on a company: Don't break the company rules and policies.

  27. Anonymous Coward
    Anonymous Coward

    Contract Terms.....and other things....

    Elsewhere other comments make the point about JUST KNOWING about backups (as opposed to ASSUMING that "the cloud takes care of all that").

    *

    But as someone who has had to set up and manage a regular disaster recovery test from scratch, it's pretty clear that backups, although important, are a long way from the whole risk management story! I wonder if any of the people described in this article have even bothered to read their contract with Google, never mind think about backups or disaster recovery or user involvement in risk management.

    *

    I know....I'm a dinosaur....all that cool cloud stuff is the answer to all business problems.....until it isn't!!!

    1. Anonymous Coward
      Anonymous Coward

      Re: Contract Terms.....and other things....

      No, the dinosaur bit might be more for actually reading a contract.

      I can't begin to describe how inept most tech companies are at reading supplier contracts and terms - make them long enough and you could probably ask for their first born and still get sign-off. TL;DR is well alive, with few realising that the "TL" part of that may not be accidental.

      That said, you can push it too far. If Adobe ever gets into court for issues with terms they may very well lose the case because of how difficult they have made it to read the terms that effectively define the contract - having to wade through two mazes of links to find the relevant data could be construed as deliberately obstructive..

  28. Tlingit

    You're missing the point!

    Nearly everyone here is completely missing the point: the problem is not that the startup's data is gone, the problem is Google doesn't care and is non-responsive.

    Had Google just told them in a timely way that they cannot restore their data, we'd all be talking about something else. But Google initially talked like maybe something could be done, and then did nothing.

    The startup is just hoping the law suit will make the corporate giant care just enough to act — or at least communicate.

  29. Arkhanist

    Gsuite admin here. They do have recovery tools if user data in gmail or google drive etc gets accidental deleted - the user-accessible 'bin' being the simplest! But admins can recover that data from the last 30 days regardless, along with user accounts - presumably it's not actually gone from disk till after that. Beyond the 30 days though, they warn the data is likely gone for good.

    To delete the entire gsuite account though, that's not a simple fat-finger error, that's a full on failure to read what's in front of you, and doing it anyway including clicking through the warnings. I've never actually done that particular step, but the admin interface is clear when what you're doing will cause permanent data loss, including disabling certain services, and has 'are you sure?' confirmations. I'm not surprised that deleting the whole shebang meant google couldn't just turn it back on and have all the data linked back with user accounts etc. there's a huge amount of different services linked together under the 'gsuite' brand, I'm sure the bits were still there somewhere, but it's a bit like deleting the partition table on a disk, or formatting your last AD server, and expecting the restore to be simple without your own backups.

    As a long-time sysadmin, with great power comes great responsibility etc...

    And yes, we back up important data via external backups. There's a bunch of cloud-to-cloud backup services, as well as API access to download things, and on the individual level 'google takeout' lets you back up literally everything - including a bunch of stuff you didn't even know google had, though a lot of the data analytics/advertising stuff is turned off for gsuite business/education customers.

  30. Steve Cooper

    Loving all this cloud bashing folks :)

    People also don't realise how important backups are when you look at the cloud provider's SLA - with 'cold' Azure or Google storage you're only guaranteed a 99% chance that your read/write will actually work - https://azure.microsoft.com/en-in/support/legal/sla/storage/v1_5/ / https://cloud.google.com/storage/sla

    If I had a RAID array or similar only guaranteeing 99% success of a read/write operation I'd be sending it to landfill immediately!

  31. Patched Out

    C.L.O.U.D.

    CLOUD => Customer Loses Own Use of Data

    I get to re-post this every couple months on here. :-)

  32. wcpreston

    Your SaaS vendors aren't backing up your data

    Please check your service agreements. You might be surprised. Neither Salesforce, Office 365, or G Suite have what I would consider a backup for your data. I would define that as something I could use to restore all my data when the caca completely hits the rotary oscillator.

    Salesforce has KIND OF a backup, but it takes 6-8 weeks to get ahold of it and costs $10K. And it doesn't restore everything.

    MS and google has recycle bin type features, but those are all stored w/in your account. They go away if you (or a bad actor) delete your account.

    What I don't like is that they are not open about this. Either they should 100% cover you and support it, or they should very plainly state that backups are your responsibility. Right now they do neither.

  33. SAdams

    ‘The cloud’ is just a different datacentre

    As a few have said, using a cloud service (whether SaaS, PaaS or IaaS) doesn’t take away the need to think about recovery processes etc, although recovery testing can be more complicated.

    The top level admin who had the rights to do this probably would have had similar abilities in an ‘on premise’ world - delete all the VMs/clusters, delete everything in AD, delete all the passwords, delete the encryption keys used for backups etc etc. Only a delegation of rights that separates rights over separate areas prevents this, but its not uncommon for the top admins to have rights over everything.

    I guess maybe the difference is that you could have someone with very little experience as your top level Gsuite administrator. But if it was done deliberately by someone who, say, just got sacked, the fact its a cloud service is not that relevant. The company just has someone else they can try to put the blame on, and possibly less chance of scavenging around for fragments of data...

  34. Adrian Bool

    Shouldn’t be an issue

    Google’s own docs state that deletion of content starts 51 days after the deletion of the G-Suite account. These guys should be fine, something gone wrong in Google’s processes? https://support.google.com/domains/answer/6313602?hl=en-GB

    1. wcpreston

      That timeframe doesn't apply

      I know this is a reply to a really old question, but I was doing research for my podcast. (We're covering this event next month on The Backup Wrap-up podcast.)

      So just in case someone comes across it, your question is about why they don't keep the data for 51 days, as it says in their policy. That policy applies to deleting your subscription. It's similar to when you delete your subscription to, say, Netflix. The actual deletion will be some time later (usually the end of your billing period). They're even saying that if your subscription lapses, they'll keep your data for 51 days to give you a chance to renew it.

      But in this case, the customer deleted the account itself, not the subscription. That is a purposeful act they are then immediately honoring.

  35. Anonymous Coward
    Anonymous Coward

    Your idiocy pays for my holidays.

    - devops - lol.

    - cloud - lol.

    - millennials - lol.

  36. JAPANEUR

    This is silly. They are going to lose the lawsuit before it even happens.

    There is a checkbox in Google Vault called "data retention". They had the option, but didn't use it.

    1. wcpreston

      Google Vault is not a backup. "Data retention" is for e-discovery and archive purposes. It would not have been able to restore this account if it was deleted. In addition, Google Vault is part of your account. Delete your account and it goes with it.

  37. wcpreston

    Story update: Musey dropped their lawsuit two weeks later

    I can't believe I didn't find this out until now. Musey quietly dropped its pro se lawsuit 12 days later.

    https://dockets.justia.com/docket/california/candce/4:2019cv03864/344456

    I knew the lawsuit was without merit because there is nothing in the Google contract that says they provide backup.

    I still wish that any service contract (and service description on the website) with an IaaS/PaaS/SaaS vendor would specifically address the situation of backup. Salesforce is probably the most forthcoming, but most SaaS vendors never address the FACT that they are not providing backup for your account. In fact, none of them (as of this writing) even offer it as an additional service.

    And if they did, I would question its validity due to what happened to OVH's customers who paid for what they thought was an offsite backup. The service description said backups were triple-replicated and the service agreement said backups were "physically isolated." Turns out that meant the server was in a different part of the datacenter. You know... isolated.

    Please make backups of your IaaS/PaaS/SaaS data, and know exactly where those backups are going.

    Disclaimer: I do work for a backup as a service vendor, but that doesn't decrease the validity of what I said above.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like