back to article InfluxData apologizes for deleting cloud regions without performing 'scream test'

InfluxData has lost the data of customers using its services in Australia while users in Belgium are struggling to figure out if they can restore the last 100 days. The vendor behind the InfluxDB time series DBMS has now apologized to customers caught out when it discontinued InfluxDB Cloud service in the two regions: AWS …

  1. cookieMonster
    Facepalm

    Dooh

    Where’s the Homer Simpson icon??

  2. John Robson Silver badge

    It seems like a mistake they won't want to repeat

    ...

    Though better still to learn from the mistakes of others.

    1. Arthur the cat Silver badge

      Re: It seems like a mistake they won't want to repeat

      Though better still to learn from the mistakes of others.

      Sadly that's not a skill many people possess.

    2. Doctor Syntax Silver badge

      Re: It seems like a mistake they won't want to repeat

      Given that the customers affected might well sue them out of existence and whilst that's happening customers in other regions will, presumably, take note they might not have too much chance to repeat it.

      1. Jim Mitchell
        Holmes

        Re: It seems like a mistake they won't want to repeat

        What grounds do the customers have to sue? Did they have a contract that requires notification of service changes by carrier pigeon?

        1. Doctor Syntax Silver badge

          Re: It seems like a mistake they won't want to repeat

          They were paid to look after other people's data. They didn't. If you're paid to do that you make sure the data is preserved even if you withdraw the service.

          DBA's priorities:

          1. Ensure the safety of the data.

          2. Ensure the continuity of the service.

          In that order.

          1. Anonymous Coward
            Anonymous Coward

            DBA's priorities

            It seems like the DBAs at the companies affected have some questions to answer:

            1/ Why didn't they enter contracts that protected against this case?

            2/ Why didn't they have a separate backup of their oh-so-important data?

            3/ Why didn't they pay attention to the emails etc that Influx sent out?

            1. Teal Bee

              Re: DBA's priorities

              Well said. No matter what the contract says, the data owner is always responsible for the continuity of their data.

              Not the database service provider.

              Not the backup service provider.

              The owner.

            2. donk1

              Re: DBA's priorities

              Who said they still have DBAs if it is all outsourced?

              With DevOps where is the 'Admin' part?

              So the experience DBAs or any type of Admin have after 30+ years does not count any more?

              Well then learn this type of lesson over and over again..

        2. tracker1

          Re: It seems like a mistake they won't want to repeat

          Negligence, plain and simple. Damages will be very easy to prove for a lot of their customers. All couple have been avoided with a simple scream test or automated backups.

          1. Crypto Monad Silver badge

            Re: It seems like a mistake they won't want to repeat

            That claim would have to be on some sort of *implied* guarantees of service - which might work in a consumer court, but not in B2B, where parties are expected to read the contracts they sign.

            I haven't seen the contract itself, but I'd expect that it has clauses that explicitly disclaim all consequential damages; that compensation is limited to service credits; that they have the right to cease the service by providing X days notice; and that serving the notice via E-mail is treated as sufficient.

            That doesn't mean that they were not jerks. At minimum they should have had brownouts prior to closure, identified the customers still using the service and helped them to migrate, and even at the end they should have *switched off* the service but not *deleted* anything for at least a month.

            However, you can be sure that there will be no redress in court for those affected. Maybe some private deals will be reached, but only to sweeten those customers to stay with influxdb and not walk to a competitor.

  3. Terje

    I find that even if everyone had received the information when first sent, it's very little time to a find a replacement service, plan and execute a move.

  4. Doctor Syntax Silver badge

    "In hindsight, our assumption that the emails, sales outreach, and web notifications would be sufficient to ensure all users were aware of and acted on the notifications was overly optimistic,"

    They were evidently acting as DBAs for their customers. I've said a number of times that the first requirement of a DBA is paranoia. That comes before any particular skills or product knowledge. Optimism of any degree is not a substitute; it's the exact opposite.

    The other thing I've said here a number of times is something their customers are now realising: cloud is somebody else's computer and you don't control it.

  5. SJA

    There is no "Cloud"

    There is no "Cloud". There is just someone else's computer.

    1. b0llchit Silver badge
      Facepalm

      Re: There is no "Cloud"

      They told me the spoon was wholly real, functional and indestructible. Both Neo and Mr. Anderson told me so!

      1. Anonymous Coward
        Anonymous Coward

        Re: There is no "Cloud"

        "They told me the spoon was wholly real, functional and indestructible"

        The first NFT was a spoon!

  6. Mike 137 Silver badge

    Change of habits needed

    "We have a running use case and are not in the habit of checking the documentation every week just in case our service gets cancelled without prior warning."

    When you store your data on someone else's computer, it's probably necessary to keep checking just in case.

  7. xyz Silver badge

    Ifuxdata

    Seems to be a better company name. I had this happen to me on a much, much, much smaller scale and I'm still pissed. Corporate suicide at its best.

  8. Jeff 11

    When decommissioning *your own* production data where you haven’t had absolute confirmation that it isn’t in use, you at least snapshot or backup first prior to removing it from your active set. That way you can quickly revert a potential mistake, which is ALWAYS a possibility when performing any non standard operation with a data source.

    It’s frankly imbecilic not to do this with customers’ data.

  9. Anonymous Coward
    Boffin

    "Your number one expectation as a cloud database provider is to keep data safe and recoverable."

    "Your number one expectation as a cloud database provider is to keep data safe and recoverable."

    What guarantees do the Cloud providers give for data recoverability and integrity.

    1. botfap

      Re: "Your number one expectation as a cloud database provider is to keep data safe and recoverable."

      >What guarantees do the Cloud providers give for data recoverability and integrity.

      None whatsoever. Ive been through this with clients who have lost data and servers from cloud providers (Azure & AWS) and VPS providers (Vultr, IONOS, HeartInternet and others). Even when you replicate data to multiple zones a stray deletion command in one zone replicates to the others before you can finish your brew and investigate. The contracts are always water tight on the providers side, there is no comeback or compensation when it happens

      As an earlier poster remarked: There is no cloud, its a mental abstraction, only other peoples servers. If you are not doing your own, locally stored backups and replicas then you are asking for a disaster. The cloud isnt some magic place where data is secure. Its just a marketing abstraction to make you believe thats the case. Its also fucking expensive compared to running your own, even on a small scale, unless you are in a location where appropriate bandwidth isnt available. In which case you probably chose the wrong location to base your data intensive business

      1. Piro

        Re: "Your number one expectation as a cloud database provider is to keep data safe and recoverable."

        The "cloud" is the new "external hard drive", in that sense that both are places people leave data, expecting it to be entirely safe, contrary to all reason.

  10. Joe 59

    I've worked with this company and their products in the past and have found them to be useful. I've also worked with "cloud" providers and universally found them to not give a rat's ass about my data, making data integrity and disaster recovery wholly my responsibility. Yes, a scream test is always warranted, just did this myself on around a thousand servers. No one responds after multiple attempts at contact? OK, power off, but wait to destroy for a month or more. Saved my bacon more than once, and I'm pretty surprised this particular company let themselves forget that lesson.

    Unfortunate bad press, certainly, but corporate suicide it ain't.

    1. axbu89

      Thousand servers at once, good God, I imagine there were a lot of screams unless it was 99% unused legacy VMs.

      But yeah, even if I'm removing a server that I'm 99% sure is part of an already decommed system and just got left behind I'll still shut it down to do a scream test for peace of mind before I nuke it.

      Oh and following the change process is a pretty essential solution for anything that is or has been production. Just not worth getting chewed out or fired for being a cowboy.

  11. axbu89

    This is why we scream test before deleting servers and data

    Absolute morons deleting customer data.

    You know that if they treated their own production servers and DBs this way then they'd be fired but it's ok to do it to customers apparently.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like