back to article AWS customer faces staggering charges over S3 bucket misfire

AWS looks set to intervene after a customer highlighted a flaw that allows S3 bucket owners to be stung with potentially massive charges for attempted accesses they have no control over. Amazon's Simple Storage Service (S3) was the first and one of the most widely used of the cloudy giant's online services, and also regularly …

  1. abend0c4 Silver badge

    This is just one example

    There are so many ways in which cloud service customers can rack up unexpected charges and the whole problem needs a comprehensive solution.

    As far as I'm aware none of the big players allow you to set an expenditure limit that they will actually adhere to. They may agree to warn you at some point when you've exceeded some threshold, but quite possibly not until the threshold is dust in the rear view mirror and you're facing eye-watering bills.

    The fact that you (usually) need to hand over you card details even for the "free tier" should be enough to send prospective customers fleeing for the hills, but clearly isn't.

    1. elsergiovolador Silver badge

      Re: This is just one example

      I'd say most businesses don't even need AWS. Computers are so powerful now, most could get away with a single dedicated server (and an extra for redundancy) with an unmetered connection. This cloud nonsense is so unnecessary.

      1. Anonymous Coward
        Anonymous Coward

        Re: This is just one example

        Tell me you've never worked at a large scale without telling me you've never worked at a large scale :D

        1. unimaginative

          Re: This is just one example

          A single servers will not scale hugely, and lots of people needs more, but the word used in that comment was "most".

          Most organisations are not Google or FB. Most businesses are SMEs.

          1. Tim 11

            Re: This is just one example

            yes but for smaller customers there are probably more gains to be had. A number of my customers host web apps on azure for less than $15 per month, and they don't have to employ an admin to configure firewalls or do windows updates

            1. phuzz Silver badge

              Re: This is just one example

              This is another good reason people use the cloud (aka 'someone else's computer'). When a RAID card goes bad at 3am, someone else gets paged in the middle of the night to go fix it, while your website keeps on running.

              1. Anonymous Coward
                Anonymous Coward

                Re: This is just one example

                But your are also then relying on someone else to fix your problem. Your on-call techie is accountable to you, the cloud techie will place you below all their bigger customers

            2. Nuff Said

              Re: This is just one example

              "for smaller customers there are probably more gains to be had" - did you read the article?

              There might be gains to be had, but there are also massive potential downsides.

            3. Anonymous Coward
              Anonymous Coward

              Re: This is just one example

              yes but for smaller customers there are probably more gains to be had.

              By whom? From the article it sounds like there's more to be gained by Amazon from smashing smaller customers with unwarranted charges.

        2. abend0c4 Silver badge

          Re: This is just one example

          TBF, the majority of businesses probably are not working at a large scale. In this particular case, it was a proof of concept.

          And unless cloud providers are going to exist solely for large scale operations, they need to fix this.

          Indeed, if they do exist solely for large scale operations they need to fix it because the current evidence is that even those customers are over-provisioning and being overcharged as a result. That can't persist if there is genuine competition between providers. And they're going to run out of large scale customers if they don't provide a conducive environment for today's small scale customers who may happen to have a growth trajectory.

          1. Mike Pellatt

            Re: This is just one example

            "enterprise" customers massively over-provisioning is a key element of the cloud business model.

            1. Cris E

              Re: This is just one example

              Real, unquoted, Enterprise customers have been over-provisioning all over the data center for ages because it's easier to spend once than try to add scale later. It's been a key element of large system sales forever.

              1. Mike Pellatt

                Re: This is just one example

                But.....

                But.....

                "Cloud" promised to actually, you know, reduce those costs by offering capacity-on-demand.

                Didn't workout like that, though, so here we are.

                1. doublelayer Silver badge

                  Re: This is just one example

                  You can save money with cloud resources depending on what the alternative is, but it requires work. The sales pitch of it always being cheaper and you hardly have to do anything was a lie, but so is the assumption that it's always more expensive than anything you could do. It is heavily dependent on what resources you want and what you're comparing them to.

          2. Martin M

            Re: This is just one example

            It was a long time ago (about 2016 I think) I was dealing with AWS on a genuinely enterprise scale, but back then they were very good at coming in and helping find and fix stupid overprovisioning. I was pretty impressed with their "what's good for the customer is long term good for us" attitude. It may have changed since.

            Regardless, I've little sympathy for enterprises. Cloud providers give you tons of MI, policy tooling and so on to enable you to keep control of your cloud estate. If you want, there are even better third party tools. They just doesn't get used very much/well. The point around hard spending limits is valid at the small scale, but for most enterprises monitoring rather than blocking is fine.

            I'm cynical, but the way many enterprise IT organisations seem to do on prem capacity management is this: 1/ Make it generally slow and hard to provision new services 2/ suddenly realise that the datacentre/big expensive SAN/core network switch etc. is almost full. 3/ only provision new things on an emergency exception basis while the capacity issue is fixed (which takes ages). At no point does a serious attempt at decommissioning, consolidating or optiimising old services take place.

            That approach makes stupid capacity management decisions slowly, at the expense of capacity not being available when needed. Moving to Cloud lifts the constraints and in the absence of proper policies, stupid capacity management decisions can now be made really fast, via API call. On the other hand, the capacity is there when you need it.

            But it doesn't have to be this way, either on prem or on the cloud.

            1. Doctor Syntax Silver badge

              Re: This is just one example

              The common key element missing from your descriptions of those alternatives is planning.

              You'll probably find there are two types of business, or departments within a business. Those which being IT into their project plans early because they expect IT will be helpful and efficient and those who don't because they think it's a waste of time as they don't expect IT to be able to do what they need. Oddly enough, both types find their expectations are usually met.

              This applies not to IT but to a wide variety of service provision.

              1. Martin M

                Re: This is just one example

                I'm sure there's sometimes (often?) bad behaviour and laziness by delivery projects, and those projects really deserve what they get (or don't get). I also suspect in some cases infrastructure organisations know perfectly well they're going to run out of space but their budgets get squeezed, which is not really their fault.

                However - I do also remember a quite a number of cases where I have given infra orgs notice well ahead their stated (long) SLAs, and somehow they still missed them - not by a little but a lot. This is across a fair spread of enterprises.

                In comparison, unless you have huge workloads, you never give a cloud any notice at all and what you need appears magically in minutes. There are rare exceptions - if you've failed to reserve instances and a major AZ outage happens, you're going to find yourself at the back of a very long queue for spot instances. And I think I've read new instances in Ireland(?) are currently problematic because access to power for new DCs is rationed. But in general, the massive scale of clouds just makes the problem go away from a consumer perspective.

        3. Terry 6 Silver badge

          Re: This is just one example

          Poster wrote most businesses

          That does some heavy lifting here. But so does worked at a large scale

          Both statements need some defining.

          1. Anonymous Coward
            Anonymous Coward

            Re: This is just one example

            >Poster wrote most businesses

            In UK at least, most businesses employ 0 people other than the owner - ~74% or 4.1 million in 2023. Another ~1.4m had between 1 and 49 employees and ~90,000 employed more than 50.

            https://www.gov.uk/government/statistics/business-population-estimates-2023/business-population-estimates-for-the-uk-and-regions-2023-statistical-release

        4. Max Pyat

          Re: This is just one example

          Tell me you flunked reading comprehension without telling me you flunked reading comprehension...

      2. Martin M

        Re: This is just one example

        Most normal non-tech companies with that scale of IT requirement don’t run servers themselves at all - they either started SaaS native or moved to SaaS years ago. It’s generally not economical to pay two FTEs (to cover leave and illness) to look after one server. Instead they buy apps on a £/user/month basis on someone else’s multi tenanted service, and tie everything together with nasty spreadsheets.

        They won’t use AWS either - still needs IT admins / “devops engineers” etc.. They don’t have or want them.

        1. Doctor Syntax Silver badge

          Re: This is just one example

          Once upon a time it was perfectly feasible for a small business to have someone freelance set up a server with a simple backup arrangement - put in a tape, run a script at end of day and store off-site (i,e, the business owner took the tape home). As needed for tweaks the freelancer could come in for a day or so as required to make changes.

          But then, once upon a time the government of the day decided to kill as much of the IT freelance segment as it could.

          1. Martin M

            Re: This is just one example

            The increased patch frequency and security monitoring required to prevent your business being overrun by ransomware may have changed some of the economics on that.

            SaaS wasn't really a viable alternative back then and really should be more cost effective due to automation and scale, at least for anything of moderate complexity.

            Finally, "contractor under bus" was very much a risk with that arrangement, whereas hopefully with a decent SaaS product, they will have more than one person who knows how everything works.

      3. bigtimehustler

        Re: This is just one example

        I'm not entirely sure why you think an SME can afford to be offline for any amount of time. Particularly if they are a Web based SME snd it's not just an ad on website. That is their whole business offline. Also, you assume a simple switch over to a cold server, what about DB replication, how does that server have an up to date copy of the data being written into the hot server? All of this stuff can be sorted, sure, but not by someone running a solution on a couple of PCs

        1. Roland6 Silver badge

          Re: This is just one example

          The trouble is IT (especially Microsoft) is far too reliable…

          So whilst people understand the concern, because the tech works they turn a blind eye to their exposure until something happens and they go off line…

          Currently I have a client who, due to recent change in management don’t understand why their site with the on-prem server, which the business naturally depends upon, has two different Internet services connected to it (during lockdown one service went offline for 3 weeks due to a fibre outage). Obviously, they still only have one server, but that is due to go as they move to a hybrid cloud arranged around 365.

      4. Anonymous Coward
        Anonymous Coward

        If Adam Smith had known about lambdas, APIs, and the digital economy instead of the needle maker

        "I'd say most businesses don't even need AWS. Computers are so powerful now, most could get away with a single dedicated server (and an extra for redundancy) with an unmetered connection. This cloud nonsense is so unnecessary."

        This farming concept is so unnecessary. We can all grow our own vegetables, raise our own chickens, till our land, and milk our cows.

        We can all be network engineers, system engineers, software engineers, cybersec engineers... in addition to catering to our non IT business.

      5. jonsg

        Re: This is just one example

        And when that one server has a disk outage or RAM failure?

        Or power fails, and the UPS doesn't pick up in time?

        Or the network goes down?

        Or the machine room air conditioning breaks down? (Been there, done that.)

        Or server load goes 100x without warning after a favourable article in a major publication?

        ...and so on, and so on. Running your back office in the cloud means negligible capital expenditure, operational expenditure that scales with the business, and machine room reliability issues pretty-much going away. It's the cheapest insurance you can buy, to cover both unexpected success and unexpected failure.

        1. Doctor Syntax Silver badge

          Re: This is just one example

          Most businesses don't have any of these things. The owner/user's PC sits on the desk, not in a server room. If the power goes off the whole business shuts down anyway, or at least reverts to whatever can be done without any powered machinery or even lighting. It seems to be an assumption in these parts that a business automatically has a server, a number of clients and an online presence.

          1. jonsg

            Re: This is just one example

            We're talking about the kinds of businesses for which having their back office and web presence in the cloud is a sensible option.

            The discussion's irrelevant for the sole-trader firms and self-employed, where that PC is on the desk. They're never going to learn how to use AWS. It's the multi-person startups and high-growth SMEs where that starts to become a consideration.

          2. doublelayer Silver badge

            Re: This is just one example

            Because we're debating about one or two servers versus cloud. A company that needs neither of those because the only employee's personal mobile phone and company email (companyname@gmail.com) is all the communication they use isn't relevant.

      6. claude191

        Re: This is just one example

        Here is an argument for somewhat the reverse; small=cloud, big=DIY

        https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/

        My personal view is that (either way) it matters far less than ensuring there is a plan/strategy, and the right ppl and processes to execute & manage. You can make a mess either way.

    2. Fat Guy In A Little Coat

      Re: This is just one example

      I thought Azure let you set a hard limit - is this not the case anymore?

  2. elsergiovolador Silver badge

    18 years

    Is this a reheated old story?

    Using S3 on AWS is like leaving your stuffed wallet up on the street ripe for easy pickings.

    It's meant for use by VC backed happy clappy businesses with more money than brain cells, not actual, real businesses.

    1. M man

      Re: 18 years

      We use cloud and we have 1 guys taking an axe to usage full time.

      A lawyer and finance guys, 3 guys in upper management to fight the supplier on contracts and not forcing to buy service and resources we don't need every year.

      We fly the guy with axe out to the US every year, cost us a fortune, mainly because he has a new axe when he's gets of the plane.

      Works out cheaper then just "letting thing be"

      Probably because of the axe.

    2. MyffyW Silver badge

      Re: 18 years

      Oh sweet reason at last!

      The "at scale" argument seems to get trotted out all the time in support of Cloud, but actually it doesn't scale well in cost terms. It's generally linear. Having costs like that is not how really successful businesses grow.

  3. Jason Bloomberg Silver badge

    An immoral path to profit

    I think Amazon will need to stop charging for unsolicited PUT requests or come up with some other solution before the script kiddies start hammering random buckets, rogues start hitting specific targets.

    It seems a more effective modern day equivalent of sending postcards to businesses who have a Freepost address so I am sure it will be exploited now it's known.

    Though why anyone would sign-up to any service which makes you pay for people trying to kick your door in, intentional or not, is beyond me.

    1. Anonymous Coward
      Anonymous Coward

      Re: An immoral path to profit

      DDOS is so old school

      Anonymous LEOPRD tool incoming

      (Low Earth Orbit Put Request Deployment)

      Posting Anonymously obviously

    2. fajensen

      Re: An immoral path to profit

      I think Amazon will need to stop charging for unsolicited PUT requests

      Right there I got a vivid picture of an AWS-excecutive vomiting blood right onto the conference desk, he is surrounded by his accolytes rending their clothes and throwing used cat-litter over themselves.

  4. Henry Wertz 1 Gold badge

    There's this thing called passwords

    There's this thing called passwords. I know Google, and Amazon, and basically all these services have fancy OpenID based authetnication methods (or probably some non-standard Active Directory based thing for Windows), two factor authetnication in some cases, and so on. But I'm surprised S3 can be set up at all without a mandatory password. Obviously, having a hard-coded bucket name and password in your program is not terribly secure; but it's infinitely more secure than just having a bucket name then any rando on the planet can access your bucket.

    Good on Amazon for deciding they'd refund for this. But, to be honest, I don't know if I could even call these accesses unauthorized insofar as the bucket was set to not require any authorization to access.

    I will note, when I was in like junior high, I had words with the local authorities about this -- I got knicked for unauthorized access to a computer system. I pointed out I dialed this telephone number, it answered and did not ask for a password so I assumed no authorization was required. This was a terminal server, so I used it for some internet access, not poking around somebody's files. It made the police REALLY nervous when they handed me a sheet of paper to sign away my rights to an attorney and I said "I'll talk to you but I'm not signing this." They had my parents sign something to the effect that I'd refused to sign (I don't think they read it, so they probably signed away rights they were unaware of. No matter though.) They got even more nervous when I asked what I was being charged with, I pointed out I had definitely not met the requirements for the Computer Fraud and Abuse Act of 1986 (and at that point there were no newer laws on the books yet). As a bonus, their "computer guy" was out that day! When I described how a web browser worked, they said with a straight face I should get prior permission from each and every web site owner before I load a web page from their site. LOL. From their followup questions it was clear they had bigger problems, someone was in their systems trying to crack into military targets. I pointed out "I didn't do that". One of their computers off in the corner spontaneously blue screened, so I pointed over there and said "I didn't do that either". LOL. This was in fact around the time "The Cuckoo's Nest" covered so maybe it was the same guy poking through their systems. Since they couldn't figure out what to do they ended up deciding to fine me $50 (I'm not sure what specifically for, but OK I guess).

    1. Jason Bloomberg Silver badge

      Re: There's this thing called passwords

      This doesn't seem to be about passwords or lack of to me, more like Amazon charging you every time someone tries to login as you with the wrong password.

    2. katrinab Silver badge
      Facepalm

      Re: There's this thing called passwords

      Did you actually read the article before writing your screed?

      Every time someone attempts to log in with an incorrect password, you get charged for it. They had millions of failed login attempts and therefore got a bill for thousands of dollars.

    3. Crypto Monad Silver badge

      Re: There's this thing called passwords

      Sorry, but you have appears to have missed the point of this story.

      * The bucket *was* set up to require authorization

      * The requests did not have valid credentials, and therefore were refused (correctly)

      * However, the bucket owner was still charged for the failing requests - tens of millions of them.

      1. Roland6 Silver badge
        Joke

        Re: There's this thing called passwords

        But security costs and is an integral part of the service. Hence the charge for processing credentials is incurred everytime credentials are presented to AWS, regardless of whether they are authorised, correct, unauthorised, incorrect. Thus that $1,300 (per day) is the charge for S3 security service usage…

        /joke

        What is troubling is apps usage of cloud services; does the (offending) app make clear to users the implications of its cloud usage ie. The app might be free and have no in app purchase, buts usage of cloud services is such the user is going to be hit with an unpleasant surprise.

        Pocwierz really should identify the offending open source tool, particularly given it has been fixed, so people can look and better understand the cloud usage problem. I anticipate a need for app developers to list the (chargeable) cloud services they use and the billing rate for its usage; in this case the number of PUT requests per hour/day successful/unsuccessful and the amount of data written.

  5. JohnSheeran

    "To demonstrate the security implications of this, Pocwierz said that he opened up his S3 bucket for public writes, and in less than 30 seconds it amassed over 10 GB of data from numerous sources."

    So, you open your bucket to the public and you expected what exactly? You use unnamed open source tools and they rack up writes? While I might agree with the injustice of AWS charges, the incompetence of their users is also a significant point of concern.

    1. doublelayer Silver badge

      I don't think that's the point in the statement. This bucket demonstrates two things:

      1. If you don't allow public accesses, you can still be charged for unsuccessful attempts.

      2. If you do allow public accesses and an open source library uses your bucket name and people don't change it, they end up handing you gigabytes of their data. They probably don't want you to have gigabytes of their data. They have a security problem. I think this is what they were trying to say.

    2. Raphael

      my accidentally naming my AWS bucket the same as you name your AWS bucket shouldn't mean I can do writes to your one.

      (it should be going to my bucket alone)

      1. doublelayer Silver badge

        Bucket names are globally unique. [name].s3.amazonaws.com. The first to register it is the one who gets to use it. There is no naming your bucket the same as mine. The first one of us to set it up gets the name. In this case, their name was evidently used as a placeholder somewhere and people didn't update it with a real address.

        1. Doctor Syntax Silver badge

          This is a bit puzzling in the article. How did he come to have the same bucket name as that of the unnamed tool? Alternatively how did the tool work before there was an actual bucket set up with the name it uses? It looks as if obfuscation of the original report has succeeded to the point of incomprehensibility.

          1. Anonymous Coward Silver badge
            Facepalm

            There wasn't already a bucket set up with that name.

            The unnamed tool (I imagine some form of backup utility which can send to S3 amongst other places) probably defaulted to something like "changeme.s3.amazonaws.com" - no need to register something which is clearly a placeholder. Especially if doing so means that you'll get millions of probes and rack up an unexpected large bill!

            (Yes, I know you shouldn't use as a placeholder something which is possible to register)

    3. Max Pyat

      You've failed to understand the story

      Please read it again more slowly

  6. vogon00

    Let's have an AWS singalong...

    There's a hole fault in my bucket, dear Liza, dear Liza,

    There's a hole fault in my bucket, dear Liza, a hole fault.

    Then mend it, dear Henry Jeffrey, dear Henry Jeffrey, dear Henry Jeffrey,

    Then mend it, dear Henry Jeffrey, dear Henry Jeffrey, mend it.

    1. Androgynous Cupboard Silver badge

      Re: Let's have an AWS singalong...

      Username checks out

      1. vogon00

        Re: Let's have an AWS singalong...

        "Username checks out"? I'm famous for bad poetry, not bad singing! :-)

    2. Doctor Syntax Silver badge

      Re: Let's have an AWS singalong...

      Well played, sir.

  7. Bebu Silver badge
    Windows

    " to alleviate this risk by avoiding short or common names for S3 buckets"

    "attempt to alleviate this risk by avoiding short or common names for S3 "

    Hash your human friendly name with a pinch of salt to get a random string of valid S3 Bucket name characters and of the S3 max bucket name length. Or an equally long but random string and use a directory service to publish the friendly name to the random actual name mapping.

    As for giving Bezos my CC details ...

    1. fajensen

      Re: " to alleviate this risk by avoiding short or common names for S3 buckets"

      Well, yes, but, even if sounds like it's kinda hard and it might take a while, wouldn't someone attempt to write a tool that does a brute-force or maybe even distributed search for S3 buckets - just to deliberately mess with this "feature"? And once your bucket is on an "existing" list, then those l33t script-kiddies are going to stomp it.

    2. Anonymous Coward
      Anonymous Coward

      Re: " to alleviate this risk by avoiding short or common names for S3 buckets"

      Straying into security by obscurity there ....

    3. Roland6 Silver badge

      Re: " to alleviate this risk by avoiding short or common names for S3 buckets"

      I thought alleviating this (accidental use of common names) was one of the reasons Microsoft automatically uses names of the form:

      Xyz.<your domain name>.on Microsoft.com

      Rather than

      Xyz.onmicrosoft.com

      Along with defaults to user local paths, so my usage of “MyStuff” defaults to MyStuff.Roland6.onmicrosoft.com, whereas some else would default to MyStuff.ANOther.on Microsoft.com

      So part of the problem here was down to poor coding in that popular open source tool.

  8. Zippy´s Sausage Factory
    Devil

    "The takeaway is that anyone who knows the name of an S3 bucket can send it PUT requests, and potentially rack up massive charges for the AWS account that owns it."

    The new wave of incoming denial of service attacks is going to be interesting. I wonder if other cloud providers are subject to similar issues?

    1. Roland6 Silver badge

      Just need some way of making money out of this…

      Then just need to send out an email with a “click me” button to cause it to generate tons of (failed) PUT requests, resulting in transaction commission being credited to my account…

      1. Alan Brown Silver badge

        Or large charges being accrued to a company you don't like - "For the Lolz"

        Economic DoS attacks have been around virtually forever

  9. Nick Ryan Silver badge

    Gaming the system

    I strongly suspect that the charging of non-authorised put requests in intentional. AWS charge for everything.

    A moderately creative developer could communicate with their services without paying for it. For example, through using deliberately non-authorised comms. Such comms would be rejected, and therefore free, but each would still be able to upload some information in the headers. End result is free transfer of data.

    1. doublelayer Silver badge

      Re: Gaming the system

      Given that inbound bandwidth is one of the few things they don't charge for, this doesn't seem very useful. I am charged for each put request to S3, but if I put an object there and it is a gigabyte in size, I'm not charged any more for that. I would be charged to store it, but if I simply read it and deleted it, that is very cheap. Therefore, building a protocol around free unauthorized put requests wouldn't save you much at all.

  10. jonsg

    Use the tools that are there to protect you!

    Within the Billing and Cost Management console, you can set budget limits, and configure alerts at various percentages of the upper limit. Better still, the comparison is based on forecasted cost, not spent cost, so you can catch a misconfiguration or runaway progress before it actually costs you the money it might do.

    If you don't put in these guard rails, you're a fool to yourself. It's saved my company from idiot mistakes before now.

    1. Doctor Syntax Silver badge

      Re: Use the tools that are there to protect you!

      Yes, but this was just a test. After all texts are trivial. You can even test on the live system...

  11. Alan Brown Silver badge

    21st cemtury Slashdot Effect

    Who remembers those?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like