back to article Google Cloud (over)Run: How a free trial experiment ended with a $72,000 bill overnight

Sudeep Chauhan, founder of startup Milkie Way, suffered a bad case of bill shock when a test with a $7.00 billing budget and a free database plan on Google Cloud platform (GCP) generated a $72,000 invoice overnight. "I jumped out of the bed, logged into Google Cloud Billing, and saw a bill for ~$5,000," Chauhan wrote on his …

  1. Ragarath

    There is a happy ending. "After going through our lengthy doc on this incident sharing our side of the story, various consults, talks, and internal discussions, Google let go of our bill as a one-time gesture," said Chauhan.

    And how many people did he know inside to get this attention? If I did the same I'm pretty sure I'd be told where to file the check.

    1. Anonymous Coward
      Anonymous Coward

      And this is why you always set up your cloud tests with a shell corporation.

      They can't collect from a corp with no assets.

      1. Roland6 Silver badge

        It is also probably why you should manage all your cloud accounts through a shell company...

        The cloud providers are going to have to provide better controls to the user and have meaningful spend caps that the user can set and that are adhered to by the cloud provider. It's not rocket science, its how bureau and timesharing systems worked back in the 1970's and it also how mobile operators provide contract and PAYG plans with a variety of add-on options, with hard spend caps.

        This should also be a warning to hobby dev's....

        1. Martin M

          Idiots

          Not sure I buy the "not rocket science". Compared to an old-school centralised mainframe bureau or even telco caps, it would likely be a significant challenge to implement hard spend caps given the highly distributed nature of hyperscale cloud infrastructure, and the complexity and granularity of billing. Particularly without potentially impacting availability and latency in the normal case of credit being available - are you going to have each function execution or database write check a central service before executing? Or distribute information on available credit information to every server in GCP. I'm not saying there aren't solutions (probably based on streaming predictive analytics etc.) but it's certainly not trivial.

          Regardless of the technical question, this is basically an incentive alignment problem though. The clouds should really be forced (through competitive pressure, regulation, court decisions etc.) to forgive any spend above the spend cap. They'd then manage to find an appropriate balance between stopping this kind of thing occurring and the technical costs of doing so.

        2. the hatter

          > The cloud providers are going to have to provide better controls to the user

          They do, google calls them "quotas" and I discovered them several minutes after I discovered budgets and it warned me that a budget did not limit spending. Quite why someone who actually worked previously in this sphere didn't have that deeply internalised is at best puzzling.

          1. Tomato Krill

            It just isn’t as straight as that though - so say you I hit your monthly spend cap on day 5 of the month - what happens to your data, VMs then? They have a cost associated with storage them so the only option then is to delete them immediately?

            Even if your budget / quota is sophisticated enough to allow you to earmark some for that storage, generated user data etc still grows and has an ongoing cost requirement once the quota is reached

            1. doublelayer Silver badge

              In many cases, they don't. Storage is often charged by the month, not by the hour. Operation is charged by the hour. Therefore, the storage is already paid for and the operation is cut off. If the cap is also a monthly one, the user could in fact continue to store the VM without running it perpetually without exceeding their cap. Retrieval requires certain other charges like bandwidth, but that only happens after the user has deactivated the services creating the unexpected expenses, after which they may increase the cap to run other things during the month.

      2. General Purpose

        Do check the local law on limited liability first. Reasonably enough, the limitation's limited. If you set up a company to defraud someone, take their IP, etc then the law may not let you hide behind your company and you may be personally liable and sued, even if you're not a director.

      3. bombastic bob Silver badge
        Meh

        They can't collect from a corp with no assets.

        They sorta tried using a credit card with a $100 limit. Didn't help.

        If Google wanted to, they could set up budgeting PROPERLY so this would never happen. But they didn't. And if they hadn't "forgiven" the debt, chances are the courts would have decided NOT in Google's favor during subsequent legal actions to collect the $72k

        in any case, corporate officers are registered for all corporations with appropriate gummint authorities. if they wanted to, they could sue the CEO directly (for example), or go after stockholders. So a shell corporation would help a little, but not a lot.

        A lot of lawsuits have been won against manufacturers because product documentation/labeling didn't have an explicit enough warning. Not sure if in this case such a thing would apply, but who knows what can happen with high paid l[aw]yers in a court room with an advocate-type judge and/or jury pool.

        [this also makes the case of having a private cloud server to test things like this on yourself, before submitting to a provider like Google]

        1. Anonymous Coward
          Anonymous Coward

          Companies House

          > in any case, corporate officers are registered for all corporations with appropriate gummint authorities.

          In the UK, Companies House doesn't verify the information provided. So dodgy types have been gaming it for years.

          Further info: https://www.globalwitness.org/en/campaigns/corruption-and-money-laundering/anonymous-company-owners/getting-uks-house-order/

          They also legally go after whistleblowers who try fixing the problem: https://www.accountingweb.co.uk/business/finance-strategy/false-filings-prosecution-a-pyrrhic-victory-for-companies-house

          Clearly, that's not a problem that's going to be fixed.

      4. ragnar

        That's probably illegal, if you live anywhere sane. People have wised up to the possibility of someone doing this and if you do it with the intention of avoiding paying your bills, piercing the corporate veil is a thing that absolutely does happen.

        1. PhilipN Silver badge

          intention of avoiding paying your bills

          Spade-calling : intention to defraud*. Many people do it all the time in trivial ways. Happily the enforcement authorities have more pressing matters to handle, and they do not wish to become debt collectors for commercial organisations anyway. *Free advice : commit nothing in writing (including 1's and 0's) setting out the game plan.

    2. Steve Channell
      Mushroom

      kamikaze testing

      They write a program that has been written thousands of times before, don't do any effective analysis, design or testing before deployment to thousands of nodes... And the good news is they escaped the bill?

      Public cloud is expensive because "we" pay a tax to subsidies these "free trial" services for stupid people who think they can use billing caps as a substitute for testing.

      The risk that holds back cloud computing is that your critical workload might end up sharing infrastructure with these stupid people and not migrated away before iops overload the hypervisors.

      I know we're supposed to feel bad for them, but bankruptcy is an effective way to weed out the stupid fools that use kamikaze testing

    3. EricB123 Bronze badge

      It could happen to anyone

      I had a free account, and hitting the wrong key generated a $120 bill overnight.

      Actually, Google let it pass with a short phone call. But that was long ago, maybe different now?

  2. Anonymous Coward
    Anonymous Coward

    VISA gift cards

    Always sign up for these sort of offers with those top-up visa cards

    Ideally with no link to your real name / email address or bank details

    1. TonyJ

      Re: VISA gift cards

      "... VISA gift cards

      Always sign up for these sort of offers with those top-up visa cards

      Ideally with no link to your real name / email address or bank details..."

      I am not sure how this works in other countries but here in the UK, I don't believe that it has been possible to get even a top-up card without passing ID checks for anti-money laundering for some time now.

      1. Charlie Clark Silver badge

        Re: VISA gift cards

        The important thing is the credit limit because you hand over the risk to the card operator who also licences service provider to comply with such limits. Might be some clauses in the T&Cs (on both sides) to work around this, and checks by the service provider to such cards for precisely this reason. But still a good place to start.

        But, basically, this is the business model of all cloud ("as a service provider"): get the user to pay more than they expected to. Once they've uploaded their data it's not as is they're likely to leave any time soon.

        And, also, what were they thinking not to check for recursion / duplicates in the first place?

        1. Doctor Syntax Silver badge

          Re: VISA gift cards

          "And, also, what were they thinking not to check for recursion / duplicates in the first place?"

          Even more fundamentally, what were they thinking of to just let it run unattended without being sure they knew what it was doing?

          1. Lord Elpuss Silver badge

            Re: VISA gift cards

            "Even more fundamentally, what were they thinking of to just let it run unattended without being sure they knew what it was doing?"

            Wasn't one of the problems that the dashboard wasn't updating in real time, so it looked like it was within daily limits until it was far too late? In which case an observer might not have spotted anything amiss...

            1. Charlie Clark Silver badge

              Re: VISA gift cards

              Not really, they deployed untested code on a massively scalable system, which was what they wanted, and went to bed. While I think we've all learnt from leaving something running unattended, doing this with untested code is always asking for trouble.

        2. bombastic bob Silver badge
          Meh

          Re: VISA gift cards

          And, also, what were they thinking not to check for recursion / duplicates in the first place?

          weren't they testing at the time? I'll never see a programmer (with any significant experience, working on non-trivial projects) that hasn't had an occasional recursion problem or infinite loop in the testing phase. This is even MORE true when going from testing on a limited set of test data into the real world, where 'the unexpected' is the norm.

    2. Claptrap314 Silver badge

      Re: VISA gift cards

      You have a contract with the provider, not your credit card, and not your back. If the charge is refused, you get a bill. If you don't pay it, you get a notice from a lawyer (with a larger amount to pay).

      That's where the LLC mentioned above comes. The cost of setting up one of those can vary a LOT by state, however.

    3. J27

      Re: VISA gift cards

      I question that, as most recurring charges reject all prepaid cards.

  3. Doctor Syntax Silver badge

    Isn't being able to manage costs one of the benefits of Cloud?

    1. johnfbw

      Google managed the costs just fine

      Their invoice went out as expected

    2. doublelayer Silver badge

      Supposedly, and sometimes, but when it's not, it's really not. They can often manage to add so many possible billable things that it's hard to figure out what you will pay. Worse, it can be mind-numbing to attempt to compare different providers for their prices, as prices are never clearly displayed together and some providers (well, one in particular) go to extreme lengths to hide the price lists and suggest you use a calculator instead. For example, I recently attempted to compare prices for bandwidth egress from various clouds and various cloud CDN-type features as an exercise to see how much it would cost to use them to handle a spike in demand for static files. The results of my survey can best be summarized as follows: what on earth do cloud companies do to set their prices.

      Dedicated VM's egress charges are usually easy to understand, but they vary quite a bit between providers because I don't know why. The big three are in the same range (approximately 20% difference between minimum and maximum) and each include the first 5 GB egress per month with the VM. Fine, they're relatively similar and could be compared. Then, I looked at Oracle cloud, which costs a tenth of what the others cost per gigabyte and provides two thousand times as much free bandwidth. I don't get it. Either Oracle has a much cheaper system, is much worse, or is very desperate to get new customers. Still, I'd have expected that Oracle wouldn't be eager to make bandwidth a loss leader, and that other providers would compete that price downwards. But then comes the CDN options. Every single one manages to bill for cache hits, cache misses, bandwidth (completely different prices than VMs), and reading from wherever the CDN fetches data. Some of them also charge different prices based on the CDN endpoint location to the extent that it would end up being cheaper to set up VMs on their service for some regions and use their CDN for other regions to minimize bandwidth costs for the same activity. Before you ask, they usually don't let you restrict which regions you use.

      This complexity means that, although cloud can offer price benefits for specific tasks, it can only really do so if you've paid close attention to all the things that can get billed. As pointed out by this article, don't necessarily trust that the limiters on an account will necessarily work like you think they will. The answer you seek is in the documentation somewhere. It may take you days to find it, but it will end up being better for you to spend the time.

      1. Claptrap314 Silver badge

        Look up "confusopoly". The car companies pioneered this.

        1. Julz

          But the utility companies perfected it.

          1. trist

            I never quite understood this privatisation thing....

            I don't get how it can be more efficient to have a wholesaler producing the power and a a number of "billing" companies whose sole purpose is to charge the consumer whatever they think that that they can get away with for the same product.

            1. fajensen
              Coat

              Re: I never quite understood this privatisation thing....

              The extraction of ressources is more efficient when more process steps are used - basic thermodynamics really - changes in enthalpy should be contionous in a perfect design.

              The clowns who designed this stopped at: "Yay! One limited company for each "gateway"! Many, Many, Board positions for Our Kind of People And Markets to Game for our spiv- and chancer- Offspring!!!

            2. FatalR

              Re: I never quite understood this privatisation thing....

              Competition drives the prices down, but of course never below their source cost, so....

              There has been companies trying this in the UK, but apparently more common in Denmark or Belgium, who charge source pricing for power, but you pay a monthly "membership" fee, which covers the power companies costs etc - and its in their interest to keep costs/admin down etc and compete via their service charge.

      2. walterp

        Oracle is desperate for new customers. As a small player in the cloud business, they need to grow the business. The business model they have been using for decades is to make the big money after you are customer on the renewal fees. Once in their cloud, they hope to do the same thing that they did with databases.

    3. Claptrap314 Silver badge

      As I mentioned before, in order to actually deliver four nines, you need someone awake & on call & not "occupied" 24/7 with a mature runbook for every alarm with a regionally diverse set of datacenters which have maintenance schedules such that you are never below N+1 by plan. If your business needs four nines, they WILL save you money. If not, you're paying for paying for it anyway.

      Almost no SMB actually will make money going from three nines to four. There will be exceptions for companies supporting 911 operations and the like. For them five nines is more like a (barely) MVP. It appears from my (limited) talking to them, however, that most don't understand just how massive an effort it takes to deliver continuous service through natural disasters...

      1. IGotOut Silver badge

        Four Nines?

        Telecoms engineers laugh at your uptimes.

    4. Zippy´s Sausage Factory
      Devil

      Yes, it is.

      One of the benefits to the cloud services provider, naturally. Not to the users.

  4. tin 2

    "Unfortunately, a billing budget "does not automatically cap Google Cloud or Google Maps Platform usage/spending," according to the docs."

    and that... is a crappy thing.

    1. Yet Another Anonymous coward Silver badge

      Wait till we have contract manufacturing in the cloud.

      Get a sign wrong and wake up to a queue of trucks delivering 2^32-1 prototypes

      1. David 132 Silver badge
        Happy

        Unfortunately, right now it's more likely to be "...wake up to a call from Felixstowe / Long Beach / whichever port, complaining that they're already full to the brim and you've just dropped another 2^32-1 shipping containers on them"!

      2. MatthewSt

        Why wait for manufacturing when you can just have shipping - https://thedailywtf.com/articles/Special-Delivery

        1. David 132 Silver badge
          Thumb Up

          That is absolutely hilarious, albeit almost certainly true-in-the-Internet-sense, i.e. "not true". Thanks for the chuckle!

          1. Lord Elpuss Silver badge

            Who the f$ck downvoted that!!

            1. David 132 Silver badge
              Pint

              I don't know. I appear to have my very own anti-fanclub, someone here who fills their life with meaning, of a sort, by downvoting even my most inoffensive posts. In a way it's rather sweet, I suppose.

              1. Korev Silver badge
                Pint

                Because it's Friday I'm giving you an upvote and a pint... Next week I'll find your post about puppies and downvote :)

      3. fajensen
        Boffin

        ABB did something like that in the 1980's.

        Someone ordered a 6 MW induction motor via their totally new, fully automated robot factory, in Västerås, they just forgot a detail: That the number of poles in the rotor and the stator must be different or it will not run. The robots and automated assembly lines built it exactly as specified, it got to the test area and it sat there, humming.

        It is still there on a pedestal outside the factory cantina, today, most people at ABB does not know *why* a motor the size of a big garden shed is placed there, other than 'advertising'.

        1. Anonymous Coward
          Anonymous Coward

          "ABB did something like that in the 1980's"

          I'm not sure an induction motor would fail to work with an incorrect number of rotor poles, although the torque, velocity and efficiency will be affected by the relative number of rotor and stator poles.

          As I understand it, the motor rotates because the different AC phases in the stator coils create a rotating magnetic field. So long as the rotor is *not* spinning at the same rotational velocity as the magnetic field, a current will be induced in the rotor, and a force is generated by the electric current flowing in the stator at right angles to the magnetic field from the stator, and the force turns the motor.

          I can't visualise how this wouldn't work with a "bad" number of stator poles, although it might mean the motor doesn't generate much torque.... Any thoughts from anyone else?

          1. Loyal Commenter Silver badge

            Presumably, the induced rotating field in the rotor and in the stator have the same rotational period, so never change relative to each other, so a force is never effected in the rotor, as the filed lines are all nicely aligned. If anything, it would magnetically "glue" the rotor in place as the aligned fields would have a lower energy configuration from "snapping" into each other.

    2. Jimmy2Cows Silver badge
      Devil

      Sounds more like a cynically deliberate feature than a bug.

    3. yoganmahew

      Hmmm, when you sign up for a free trial, G a number of times reassures that it will close down the project and look for confirmation before proceeding past the free $300 credit. I have experienced this to be so, though with a spanner cluster server.

      So there's a bit not being spoken, maybe it was a company account, not a free-sign-up?

      1. Roland6 Silver badge

        >Hmmm, when you sign up for a free trial, G a number of times reassures that it will close down the project and look for confirmation before proceeding past the free $300 credit.

        And after the free credit is used up?

        Can you simply open another free trial account with the same email address etc. or are you forced to convert the free account into an unlimited debit potential account?

        Basically, at the minimum, I should be able to specify my own total £spend limit for my account and for the cloud provider to honour that limit.

        1. yoganmahew

          Email addresses are free... there's no cross verification that I can see.

      2. Loyal Commenter Silver badge

        Of course, as with a "free" account, they can't bill you. As soon as they have your billing details, that promotional feature disappears.

    4. fajensen

      Same with AWS!

      The customer basically faces unlimited liabilities if their code is faulty or their root keys stolen.

      It is really shitty that a 'billing budget' settings does not actually control anything other than some dashboard colourings and emails (and that the AWS documentation takes care to obfuscate this, without being outright fradulent)!

      The advice from here is to wear a business condom before any interactions: Set up a limited liability company to be ones 'cloud service provider' holding the legal minimum of assets. That company owns the contracts with the cloud providers, then 'the mothership' procures it's cloud services via that.

    5. hoola Silver badge

      And this is what I find so bonkers, what the hell is the point of a cap if it can be exceeded. Not just by a small amount either, 10,000 times.

      One would hope that something on this scale is an exception but possible no. The problem is that everyone is a minnow compared to Google and has not the slightest hope in hell of being able to get out of paying the invoice. Presumable people just pay or disappear.

  5. Brian Miller
    WTF?

    Not so free after all

    free Firebase plan had been "upgraded due to activity in Google Cloud" and that this "initiated billing"

    Wow! Instead of an expected shut-off of services, Google's real policy is to very unexpectedly put the customer on the butcher's hook.

    1. Phil O'Sophical Silver badge

      Re: Not so free after all

      Just like credit card companies, who set a credit limit, and then "generously" increase it because you've almost reached it & will clearly need more.

      1. Qumefox

        Re: Not so free after all

        There are more variables in that analogy that make it not very fitting. While it's true CC companies will do everything they can to fleece their customers for as much as they're worth, they're also very risk adverse as well for the most part. If you're barely making minimums and are constantly up against your credit limit, they're actually going to be less inclined to expand the credit limit because generally people in that situation have few assets to go after if they end up defaulting, which becomes more likely if limits are raised and minimums payments get even higher. The only times i've gotten my credit limit upped without me requesting it, it has occurred after I carried a balance for a while, then suddenly paid it all off and kept it paid off for more than a month.

      2. keith_w

        Re: Not so free after all

        I have never 'almost reached' my credit limit before a credit card company decided to increase my limit.

        1. Anonymous Coward
          Anonymous Coward

          Re: Not so free after all

          I reserved a credit card for online transactions - with a deliberately moderate limit. The card company periodically increased the limit unnecessarily.

          I explained and asked them to reduce the limit back to my original. They then pointed out that reducing someone's credit limit is taken as a sign of financial stress - and the credit rating agencies will consequently downgrade you to everyone.

    2. Shadow Systems

      Re: Not so free after all

      Which is why you *always* sign up for "free" services with a disposable, Pay As You Go, generic credit card that is in no way linked to your real financial accounts.

      If the "free" service suddenly pulls a Google & tries to bill you umpteen zillion dollars "in error", they get yanked up short when the ~100 bucks you've given it goes poof & the card stops working. Sure you're out the hundred, but not the life destroying, financial devestating, "OMFG! I'm bankrupt!" apocalypse they've tried to ram up your arse.

      You can calmly, cooly, dispassionately discuss the proper billing for the services with them confident in the knowledge that, if they refuse to behave, you can just give them TheFinger & walk away. They can scream & gnash their teeth all they want, you don't have to pay them another bloody red cent.

      This lesson brought to you by personal experience with the likes of BMG/Collumbia CD music club trying to rape me for thousands of dollars in "free" music I never ordered. I had to send them a certified, registered letter from a lawyer informing them of the U.S.Postal rules that anything not ordered by the recipient is legally a *free gift* and the recipient is under no legal obligation to pay for any of it. Even AFTER that little smack upside the head they STILL kept sending me "free" CD's for YEARS. I had over a dozen 400 CD wallet binders full of music I didn't have to pay a cent for because the company couldn't get it through their head that "I used a disposable CC to start this ''free'' trial. I completed the contract, canceled my account, & you idiots kept sending me stuff... At Your Own Expense. I'm under no legal requirement to pay for ANY of it. So if you want to keep it up, go for it, I enjoy free music!"

      1. doublelayer Silver badge

        Re: Not so free after all

        I'm not sure it works that way. If they issue you a bill and have your name, they can send a legal threat at you until you pay it. In your situation, they had to back down since you canceled the account, but if you didn't, they could sue you and win. For a place like a cloud provider, they can argue that you intended to operate the services and that you know the costs, which is probably not exactly true but they can likely get someone to accept it. Just because they don't have a payment method that they can bill automatically doesn't mean they're out of options for ways to make you miserable.

    3. N2

      Re: Not so free after all

      What took you so long to work that out, its policy the world over now.

  6. Natalie Gritpants Jr

    No sympathy from me

    It's idiots like this that cause parts of the web to fall over for the rest of us. The idiot may have been able to get away with not paying a bill to Google but 19 billion requests went somewhere and even if it didn't crash a web-site it will have wasted energy.

    1. Roland6 Silver badge

      Re: No sympathy from me

      Totally agree Google are idiots in allowing this situation to arise.

      All chargeable processing should have been stopped (VM's paused) once the $7 limit had been hit and the owner of the account informed. The account owner can then either permit a further $7 or so use of resources or dig into the logs (I assume Google does provide run logs) to determine what went wrong.

    2. Michael Wojcik Silver badge

      Re: No sympathy from me

      The explanation for the overrun is bizarre. Who starts a web-crawling project and thinks "oh, yeah, the web definitely an acyclic graph"? Making a mistake like that is just wildly technically incompetent.

      If someone came to me with some web-link-traversal project for any purpose, my first question would be how they're handling loops, because that's important for performance and scalability. And if the response was "oh, we hadn't thought of that", it would be a long time before anything got deployed in any sort of environment that might incur liability.

      1. Anonymous Coward
        Anonymous Coward

        Re: No sympathy from me

        And how do _you_ know about loops?

        Like the rest of us - we tend to learn from experience.

  7. Anonymous Custard
    Trollface

    Let's do the timewarp?

    OK, so how many people thought this was a "Who, Me?" a few days early (or late)?

    Would have made a good one too...

    1. Androgynous Cupboard Silver badge

      Re: Let's do the timewarp?

      My first thought was "I bet he failed to check for loops" - reminded me exactly of the RTM Sendmail worm in 1988. Nice to see how little our industry has learned in the intervening 30 years.

      1. Anonymous Coward
        Anonymous Coward

        Re: Let's do the timewarp?

        In 1970 a programmer decided to prank the online operations centre. A message appeared on the main console saying "I am a ghost and I have changed your master password" - which his program really had done.

        In those days recovering from such a situation was relatively easy - except he had inadvertently put his program in a loop. The main console was inundated with the message being repeated - and each time it had generated a different password.

        He was fired next day.

  8. Sgt_Oddball
    Holmes

    Surely though...

    On though there should be a way to force, come hell or high water a hard spend limit. If the service goes past that... Down she goes.

    Something like the clowdy version of an electric meter. Once the credit's all gone, out goes the lights.

    Is it really so hard to manage?

    1. Chris G

      Re: Surely though...

      "Is it really so hard to manage?"

      No! But Google don't give a crap whether you live or die so long as the money is coming in, which means they will bill you and hope you pay even if it means selling your granny for cat food.

    2. Jimmy2Cows Silver badge

      Re: Is it really so hard to manage?

      Ah but you're thinking like a reasonable, decent human being, not a greedy executive of a greedy multinational corporation.

      Thinking more like Google, it's pretty clear this is entirely by design. Another dark pattern.

    3. David 132 Silver badge
      Thumb Up

      Re: Surely though...

      Something like the clowdy version of an electric meter

      It's an intriguing idea, but I'm not sure what a group of cats have to do with it.

      Sorry, I'm feline somewhat picky this morning :)

      1. Sgt_Oddball

        Re: Surely though...

        (fur) balls... Spell check fails me again (I'd swear SwiftKey was getting dumber).

        And yes, cloudy electric meter was what I was thinking of...

        Though not a cloud-driven meter, as those don't seem to work either (if the smart meter in my basement is anything to go by).

    4. doublelayer Silver badge

      Re: Surely though...

      Google suggestions has received your suggestion. We will not implement this suggestion because cloud users would be impacted negatively by any abrupt termination of their services. A terminated virtual machine may have been running important tasks, so we can't do that until the user says so. Similarly, if we blocked reads of a database, the customer wouldn't be able to get their content out of it. If we just blocked writes, then the user's system could [PRBot error 1004: could not think of convincing-sounding argument, please assist]. An abrupt termination of any service could cause a business customer to lose revenue for each second that clients are unable to make use of the services, and inconsistent termination, where some services are blocked but others which don't incur charges, could cause chaos when [PRBot error 1093: attempting to rephrase message "user could decide they didn't need it after all and stop paying us money" to sound diplomatic, couldn't manage it, please assist]. A user would never accept us pulling the rug out from under a service which they rely on for their livelihood, unless it's the Play Store, in which case we'll shut them down without a second thought [PRBot error 1015: sentence appears to contain data that should not be referenced, but module do_not_outright_lie requires it, please assist]. Also, adding the feature would be expensive for our developer resources for a very small number of users and hence is not an economical decision for us PRBot warning 1093: believe previous sentence is a suitable translation of "we're not a cloud monopoly player, so we don't have to do anything for our customers. Ha ha ha." However, a translation error has already occurred in this message, so please check anyway].

      1. Anonymous Coward
        Anonymous Coward

        Re: Surely though...

        @doublelayer

        Didn't read your post. Have you any notion of paragraphs or spaces?

        Chers… Ishy

        1. Loyal Commenter Silver badge
          Facepalm

          Re: Surely though...

          If you had read it, you would have realised that this was probably an intentional part of the joke.

    5. FatalR

      Re: Surely though...

      "It's too complicated" is the excuse.

      But they added the complications into billing for very finite events, so its their problem.

  9. mark l 2 Silver badge

    What is the point of a billing budget if Google are free to ignore it if your service uses more resources than your budget allowed?

    They don't do that for Google ads, if you set a budget of $10 per day then after $10 of clicks your ads stop showing until the next day, so why doesn't that apple to Google cloud?

    1. Roland6 Silver badge

      Also what is the point of Google's Dashboard if it takes 24 hours before it reflects the current situation, I would expect the latency to be sub 1 hour and potentially sub 10 minutes.

      Also, why couldn't the account user not pause/stop the VM's from the dashboard - I thought that was one of the core functions of a dashboard...

    2. Anonymous Coward
      Anonymous Coward

      AWS don't do it either

      It seems to be common practice in cloud land. All of the providers have a configurable limit where they send you a bill, but they only suspend service at the point that the credit card stops working for a months or two...

  10. Pascal Monett Silver badge
    FAIL

    He should have checked the docs and settings before releasing the code

    It seems to me that he did all the necessary checking after the fact.

    I'm sure a $75K bill prompts you to do some deep verification, but it appears that he could have avoided the whole kerfluffle by, <gasp>, actually reading the documentation of what he was about to use.

    It's quite obvious, as mentioned by a previous poster, that his contacts in Google is what got him out of the mess he put himself in. Anyone else could just go crying to the bank for a loan.

    1. DavCrav

      Re: He should have checked the docs and settings before releasing the code

      "It seems to me that he did all the necessary checking after the fact."

      Apart from setting a billing budget. I would expect 'billing budget' to be a billing budget, not a billing suggestion.

      1. Pascal Monett Silver badge

        From the article

        "The GCP Cloud Run defaults also played their part. "The max-instances is preset to 1,000, and concurrency set to 80," he said. If he had corrected this to small values like 2 and 1, the bill shock would not have occurred. "

        So he did find the setting and understand its importance. Only he checked after the bill, not before.

        1. doublelayer Silver badge

          Re: From the article

          They could have gotten their system to not result in the bill, but they did set up that billing budget thing. Theoretically, such a feature would mean that you don't have to change all the other settings to avoid a massive bill. You might reach your limit quickly, and if you set the settings right you wouldn't have, but you should be fine. In a case where that feature worked as people expect it to, the user without special access would have to pay a bill for a service that only ran for half an hour, but they wouldn't have to pay a bill four orders of magnitude over what they were expecting. Imagine how it would have gone if other such limits were sometimes considered optional. You could end up in situations like this:

          1. You set a caching server to keep copies of your files which expire every ten minutes, but it decided that the ten minutes was optional and instead used the value infinity. All your customers are getting days-old versions of everything. If you had only set the server to erase itself through a hidden task, it would have done what the TTL value is there for.

          2. You used a programming language's thread pool and set the maximum number of worker threads to equal the number of processor cores because your task is compute-intensive. It decided your maximum was unimportant, so it spawned a bunch of threads which slowed you down immensely before eventually swamping the OS requiring a forced reboot. If only you had also made the OS restrict the number of threads, the defect in the thread pool library wouldn't have caused a problem.

          3. You were filling a car with fuel, and you requested the pump to continue filling until the fuel tank was full. It decided to just keep going, so now your car is at the bottom of a flammable pond and you have a fuel bill more often associated with aircraft. If only you just measured the empty volume and specified the exact amount of the fuel, you wouldn't have had this problem. On second thought, that's also a number so you would be in the same situation. Too bad for you.

        2. DavCrav

          Re: From the article

          "So he did find the setting and understand its importance. Only he checked after the bill, not before."

          OK, that's another option that would have stopped it. But still, 'billing budget' should be something that would override that. I mean, if I write $10 in my budget, and someone says "I'm sure he meant $100k, you can go ahead and spend it" I think that would be considered unreasonable, no?

      2. fajensen

        Re: He should have checked the docs and settings before releasing the code

        They worked extra hard on making you expect that

    2. Doctor Syntax Silver badge

      Re: He should have checked the docs and settings before releasing the code

      "Anyone else could just go crying to the bank for a loan."

      Or declare insolvency.

      1. StephenH

        Re: Or declare insolvency.

        There is a reason why I own nothing & everything is in my wife's name. I trust her far more than Google

    3. Anonymous Coward
      Anonymous Coward

      Re: He should have checked the docs and settings before releasing the code

      @Pascal Monett

      I so totally agree with you. Sudeep Chauhan totally forgot the rules that if it is free, you are the product, and if it so totally cheap you are stupid enough to have not read the contract…

      Cheers… Ishy

  11. Anonymous Coward
    Anonymous Coward

    Cloud.

    Lol.

    Never quicker, never cheaper, never more reliable.

  12. David Pearce

    Blank cheque

    How many businesses are comfortable placing open ended purchase orders?

    I have had bad experiences with a Telco when my office line got spurred to make expensive international phone calls and we ran up a bill way beyond our deposit

    I also no of a VIP whose child started surfing the Internet heavily on one of the early airliner satellite connection services and run up a lottery win sized bill.

  13. giin

    Stupid user error

    It is quite obvious he did not even glance at the documentation before setting out with his big brain scheme. The fact that billing budgets act as triggers for alerts and things like automatic spend capping is quite obviously highlighted.

    1. Roland6 Silver badge

      Re: Stupid user error

      >The fact that billing budgets act as triggers for alerts and things like automatic spend capping is quite obviously highlighted.

      Is it "obviously highlighted" that such alerts will be sent out hours after the event and not actually prevent you incurring rapidly escalating charges and that the dashboard provides no simple way of stopping charges being incurred...

      Whilst the user is not blameless, Google chose to set the system up in a way that would rip users off...

      I expect business users will start to put pressure on Google (and the other cloud vendors) to massively improve the controls and their effectiveness, as currently it seems some junior programmer in the IT department can spend a $100+K over a weekend when they to get a managers signature to buy some stationary...

  14. ChrisBedford

    Google sent an automated email informing him that his free Firebase plan had been "upgraded due to activity in Google Cloud"

    Yeah imagine if your phone provider did the same thing. Or your credit card company. "We have limits, but if you exceed them we just increase them." What a stupid, utterly anti-ethical default that is.

    1. giin

      That's the whole thing, there aren't actual limits. The budgets are for triggering alerts and any kind of automatic capping you prefer to do, they aren't meant for spend capping which is immediately obvious with even a cursory glance at documentation. This is also stated directly in the console, which states quite clearly that budgets are meant for monitoring. This is a case of dumb people doing dumb things and then acting surprised.

      1. fajensen
        Angel

        You almost sound like the lead designer of the RyanAir Ticket Purchasing Experience:

        The path to that elusive 35 GBP ticket to Warsaw being festooned with more digital booby traps than the Cambodian border, but this is OK beacuse: "Dumb people deserve to get a 129 GBP travel insurance, a 23 GBP lottery ticket and a FREE (current month only, after that 99 GBP weekly) cable TV subscription added onto their 35 GBP Warshaw ticket purchase!"

        1. The Griff

          You forgot 50 quid for printing the ticket at the airport, plus another 100 quid for the pressurised cabin option.

      2. Anonymous Coward
        Anonymous Coward

        If you call a feature something, and a sufficient number of people believe it to function in a way other than the way it functions, then you gave it a misleading name. No amount of calling people idiots and telling them to read the documentation (be it clear or not) will change the fact that it was given a misleading name. This is very basic tennet user interaction design.

        1. Roland6 Silver badge

          >If you call a feature something, and a sufficient number of people believe it to function in a way other than the way it functions, then you gave it a misleading name.

          Advise taking a look at your mobile phone's "SpendCap", you know the one advertised as giving you peace of mind and freedom from unexpected bills. Whilst the small print does say what it includes, it doesn't tell what is excluded, which can be a quite surprising once it is brought to your attention.

          I wonder how many (normal people) have set a spend cap, as advised in various articles and in their phone providers literature, only to discover their teenager has still been able to purchase massive amounts of additional data, paid out of the phone account not their pocket and sign up to subscriptions from NetFlix, BT Sport, Spotify etc. and buy stuff in the Play store - billed to the phone account...

      3. UK DM

        Not limit issue, forced upgrade issue

        It is a FREE product he ordered (firebase) it got upgraded without his explicit consent. It has nothing to do with limits, the quantity of resources Google gives away in a free product is upto them.

        But the obvious contract he agreed to and his purchase order was for the FREE product.

        I assume the real issue is the small print of the contract, what would allow Google to getaway with billing like that.

        My recent experiences with Azure and Oracle Cloud Infrastructure is that the services are added to a free tier plan and upgrade is explicit, maybe to prevent adding a non free service accidently, you have to remake the profile on top of a normal profile that is chargeable.

  15. RyokuMas
    Mushroom

    The biter bitten...

    "The idea was to build a system that scraped web pages and stored the results in a database."

    So it wasn't so much an invoice as a fine from Google for trying to copy one of their key business model components???

    Bwahahahahahaha....

  16. Anonymous Coward
    Anonymous Coward

    Normal IT Process

    Poor developers create buggy code and then blame infrastructure when it all goes wrong.

    1. Roland6 Silver badge

      Re: Normal IT Process

      I don't know any developer who wasn't a "poor developer" to start off with...

      Basically, Google has ruled out their cloud offering as being suitable for education. The message is loud and clear: if you aren't a competent cloud developer who already knows the in's and out of our cloud then don't use our cloud...

      Looks like no college/university will touch Google cloud unless students sign up and accept fully liability, naturally, bank of mum&dad will complain, loudly - just as they did with Apple's poor credit control when they didn't prevent children running up massive bills through in-game purchasing...

      The upside is that if your company is toying around with cloud, you'll only need to mention to the FD the unlimited liability aspect of the service, for them to be a convert to in-house IT...

  17. SecretSonOfHG

    I'd likely would have (wrongly) believed that "budget" term meant...

    I've checked the linked docs and it is clearly stated in a highlighted section that the term "budget" does not mean "expense limit" but only "a number above which we'll notify you at some point in time in the future" Not sure if the documentation was so clear when this happened or Google updated it as a consequence of this incident.

    But if anyone thinks that everyone is reading all EULAs and service agreements of every service they purchase, sorry, it is no happening. True, one should be careful and read all the terms and conditions, but in this case I'd assumed that "budget" meant something more close to "expense limit" than its actual meaning.

  18. Loyal Commenter Silver badge

    It is eminently possible to have a flag in the user account database set to "do not exceed billing limit", and to stop services / virtual machines / databases when that limit is about to be exceeded, or is exceeded by only a small amount. It's also eminently possible to have that set to 1 by default, and to have obvious and intuitive user controls for it, along with appropriate warnings.

    The fact that no cloud provider does this means that "surprise billing" is very much their business model. Until the industry insists that they provide these controls, and they listen, or more likely, legislatures regulate them so that it is a requirement, they will continue to fleece their victims customers.

    1. MOH

      This x 1000.

      Based on the Firebase performance of 1bn reads per minute at 0.06 per 100K, that's $600 per minute.

      If your "budget limit" is only an advisory limit that triggers an email, that's $600 of charges for every minute before that mail is read and actioned.

      That's just ludicrous if there's no option for an automatic hard cutoff after a certain limit.

  19. TeeCee Gold badge
    WTF?

    Who could possibly have seen that coming?

    Rather surprisingly, Shakespeare.

    For 'tis the sport to have the engineer hoist with his own petard.

  20. Vegemite Sandwich

    Disney...

    So, a Sorcerer's Apprentice kind of tale...

  21. Anonymous Coward
    Anonymous Coward

    Hilarious

    Page scraping is not something that fills me with sympathy, as a rule (there are a few good reasons to scrape, and an infinite number of bad ones).

    But at the same time, there are a few lessons to be learned:

    * If someone who used to work at Google managed to misunderstand how the billing works, what hope do the rest of us have?

    * Never enter into an agreement where you are not absolutely certain of what the final maximum cost is going to be.

    * Never trust an automated process. Much less a whole chain of them. What we have here is a chain of automated process, some of which are running for the very first time and some of which you have absolutely no control over. What could possibly go wrong etc.

    * "Cloud" computing is, as a rule, more expensive than other alternatives. Its marketing point is the supposed convenience and flexibility, etc., not price.

    * Do not let things run unmonitored on a first test. Especially when they have access to virtually unlimited resources.

    1. bazza Silver badge

      Re: Hilarious

      Yes, page scraping; there does seem to be an element of reaping what was sown in this article.

      Does make me wonder what the hell is going on really. An easy way to avoid all this is to do development and test on premises, get it right and then explore the benefits of using cloud. That way there's low cost of mistakes, unless one's IT department takes an unprecedented view of response times to infrastructure overload.

      But AFAIK with things like AWS lambda and presumably all the Google stuff mentioned in this article you can't do development testing on-prem, you go straight to cloud and that's it. It's pretty off-putting really. Worse these days is that one seems to end up with expensive to run software that cannot be run on any other infrastructure. That sounds highly inadvisable from a business point of view.

      It sounds to me that the idea of having rentable VMs is still the long term best way of doing things, allowing mixing of cloud, on prem, and other clouds. If the major cloud providers are pricing that model to be expensive then I think that ought to be considered as a indication that their "cheaper" offering is not going to be in one's long term interests.

  22. martinusher Silver badge

    Same old...

    This reminds me of the paid phone service bait and switch scams where you'd have to call a premium number to get some information, the trick being to get you to call a really expensive premium number. (The North American system of area codes covered places in the Carrabbean so users would be unaware they were calling internationally.) The phone companies could have easily put a stop to those scams because they were necessary for the scammers to get paid. They never did because they were getting a cut.

    Then there's the colleague who showed one of his kids how he ordered stuff with his iPad. That cost him about $700 in 'in game purchases' before he got the bill and closed the door. Nobody, but nobody, runs up that kind of bill but everyone's happy to let them fully knowing that they're getting a cut of the loot (they don't call the things 'loot boxes' for nothing.....).

  23. heyrick Silver badge
    FAIL

    Fail Google

    If the service user sets up a spending limit, then that limit is to be respected. Automatic upgrades and email notifications of such should only be possible by explicit opt-in, because there's a hell of a difference between seven and seventy thousand.

    The way it is currently set up (heavily benefiting the service provider, note) just sounds like a scam, like the sort of crap mobile operators used to pull (here's your free data allocation, going over will cost you €10 per megabyte and we will notify you by SMS at some random time afterwards). Or the infamous roaming charges. The law put a stop to that nonsense. As it should automatic service upgrading as demonstrated by this article

  24. jonfr

    The cloud is a scam

    There isn't much more to it than the cloud is a scam. Just get a dedicated server with a monthly fixed price and never have to deal with this type of problem again.

    1. SecretSonOfHG

      Re: The cloud is a scam

      Are you aware that "a dedicated server with a monthly fixed price" plus bandwidth charges is one of the basic offerings of all cloud providers, right?

      1. Anonymous Coward
        Anonymous Coward

        Re: The cloud is a scam

        "The cloud" being an ill-defined marketing term, it understandably means different things to different people. In this context, I take the gentleman above is referring to what is also known in marketing as "serverless", meaning use of pre-provisioned services such as databases or scripting engines, billed by the second / minute / hour.

        He is correct that, in many common cases, an alternative offering such as a virtual private server or dedicated hosting is a much better and economical approach.

      2. jonfr

        Re: The cloud is a scam

        @SecretSonOfHG Depending on the hosting provider you either have unlimited traffic or you have traffic limit in the TB's of data. Last dedicated server I was hosting on had 25TB data limit.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like