AWS customer faces staggering charges over S3 bucket misfire AWS looks set to intervene after a customer highlighted a flaw that allows S3 bucket owners to be stung with potentially massive charges for attempted accesses they have no control over. Amazon's Simple Storage Service (S3) was the first and one of the most widely used of the cloudy giant's online services, and also regularly …
There are so many ways in which cloud service customers can rack up unexpected charges and the whole problem needs a comprehensive solution.
As far as I'm aware none of the big players allow you to set an expenditure limit that they will actually adhere to. They may agree to warn you at some point when you've exceeded some threshold, but quite possibly not until the threshold is dust in the rear view mirror and you're facing eye-watering bills.
The fact that you (usually) need to hand over you card details even for the "free tier" should be enough to send prospective customers fleeing for the hills, but clearly isn't.
TBF, the majority of businesses probably are not working at a large scale. In this particular case, it was a proof of concept.
And unless cloud providers are going to exist solely for large scale operations, they need to fix this.
Indeed, if they do exist solely for large scale operations they need to fix it because the current evidence is that even those customers are over-provisioning and being overcharged as a result. That can't persist if there is genuine competition between providers. And they're going to run out of large scale customers if they don't provide a conducive environment for today's small scale customers who may happen to have a growth trajectory.
You can save money with cloud resources depending on what the alternative is, but it requires work. The sales pitch of it always being cheaper and you hardly have to do anything was a lie, but so is the assumption that it's always more expensive than anything you could do. It is heavily dependent on what resources you want and what you're comparing them to.
It was a long time ago (about 2016 I think) I was dealing with AWS on a genuinely enterprise scale, but back then they were very good at coming in and helping find and fix stupid overprovisioning. I was pretty impressed with their "what's good for the customer is long term good for us" attitude. It may have changed since.
Regardless, I've little sympathy for enterprises. Cloud providers give you tons of MI, policy tooling and so on to enable you to keep control of your cloud estate. If you want, there are even better third party tools. They just doesn't get used very much/well. The point around hard spending limits is valid at the small scale, but for most enterprises monitoring rather than blocking is fine.
I'm cynical, but the way many enterprise IT organisations seem to do on prem capacity management is this: 1/ Make it generally slow and hard to provision new services 2/ suddenly realise that the datacentre/big expensive SAN/core network switch etc. is almost full. 3/ only provision new things on an emergency exception basis while the capacity issue is fixed (which takes ages). At no point does a serious attempt at decommissioning, consolidating or optiimising old services take place.
That approach makes stupid capacity management decisions slowly, at the expense of capacity not being available when needed. Moving to Cloud lifts the constraints and in the absence of proper policies, stupid capacity management decisions can now be made really fast, via API call. On the other hand, the capacity is there when you need it.
But it doesn't have to be this way, either on prem or on the cloud.
The common key element missing from your descriptions of those alternatives is planning.
You'll probably find there are two types of business, or departments within a business. Those which being IT into their project plans early because they expect IT will be helpful and efficient and those who don't because they think it's a waste of time as they don't expect IT to be able to do what they need. Oddly enough, both types find their expectations are usually met.
This applies not to IT but to a wide variety of service provision.
I'm sure there's sometimes (often?) bad behaviour and laziness by delivery projects, and those projects really deserve what they get (or don't get). I also suspect in some cases infrastructure organisations know perfectly well they're going to run out of space but their budgets get squeezed, which is not really their fault.
However - I do also remember a quite a number of cases where I have given infra orgs notice well ahead their stated (long) SLAs, and somehow they still missed them - not by a little but a lot. This is across a fair spread of enterprises.
In comparison, unless you have huge workloads, you never give a cloud any notice at all and what you need appears magically in minutes. There are rare exceptions - if you've failed to reserve instances and a major AZ outage happens, you're going to find yourself at the back of a very long queue for spot instances. And I think I've read new instances in Ireland(?) are currently problematic because access to power for new DCs is rationed. But in general, the massive scale of clouds just makes the problem go away from a consumer perspective.
>Poster wrote most businesses
In UK at least, most businesses employ 0 people other than the owner - ~74% or 4.1 million in 2023. Another ~1.4m had between 1 and 49 employees and ~90,000 employed more than 50.
https://www.gov.uk/government/statistics/business-population-estimates-2023/business-population-estimates-for-the-uk-and-regions-2023-statistical-release
Most normal non-tech companies with that scale of IT requirement don’t run servers themselves at all - they either started SaaS native or moved to SaaS years ago. It’s generally not economical to pay two FTEs (to cover leave and illness) to look after one server. Instead they buy apps on a £/user/month basis on someone else’s multi tenanted service, and tie everything together with nasty spreadsheets.
They won’t use AWS either - still needs IT admins / “devops engineers” etc.. They don’t have or want them.
Once upon a time it was perfectly feasible for a small business to have someone freelance set up a server with a simple backup arrangement - put in a tape, run a script at end of day and store off-site (i,e, the business owner took the tape home). As needed for tweaks the freelancer could come in for a day or so as required to make changes.
But then, once upon a time the government of the day decided to kill as much of the IT freelance segment as it could.
The increased patch frequency and security monitoring required to prevent your business being overrun by ransomware may have changed some of the economics on that.
SaaS wasn't really a viable alternative back then and really should be more cost effective due to automation and scale, at least for anything of moderate complexity.
Finally, "contractor under bus" was very much a risk with that arrangement, whereas hopefully with a decent SaaS product, they will have more than one person who knows how everything works.
I'm not entirely sure why you think an SME can afford to be offline for any amount of time. Particularly if they are a Web based SME snd it's not just an ad on website. That is their whole business offline. Also, you assume a simple switch over to a cold server, what about DB replication, how does that server have an up to date copy of the data being written into the hot server? All of this stuff can be sorted, sure, but not by someone running a solution on a couple of PCs
The trouble is IT (especially Microsoft) is far too reliable…
So whilst people understand the concern, because the tech works they turn a blind eye to their exposure until something happens and they go off line…
Currently I have a client who, due to recent change in management don’t understand why their site with the on-prem server, which the business naturally depends upon, has two different Internet services connected to it (during lockdown one service went offline for 3 weeks due to a fibre outage). Obviously, they still only have one server, but that is due to go as they move to a hybrid cloud arranged around 365.
"I'd say most businesses don't even need AWS. Computers are so powerful now, most could get away with a single dedicated server (and an extra for redundancy) with an unmetered connection. This cloud nonsense is so unnecessary."
This farming concept is so unnecessary. We can all grow our own vegetables, raise our own chickens, till our land, and milk our cows.
We can all be network engineers, system engineers, software engineers, cybersec engineers... in addition to catering to our non IT business.
And when that one server has a disk outage or RAM failure?
Or power fails, and the UPS doesn't pick up in time?
Or the network goes down?
Or the machine room air conditioning breaks down? (Been there, done that.)
Or server load goes 100x without warning after a favourable article in a major publication?
...and so on, and so on. Running your back office in the cloud means negligible capital expenditure, operational expenditure that scales with the business, and machine room reliability issues pretty-much going away. It's the cheapest insurance you can buy, to cover both unexpected success and unexpected failure.
Most businesses don't have any of these things. The owner/user's PC sits on the desk, not in a server room. If the power goes off the whole business shuts down anyway, or at least reverts to whatever can be done without any powered machinery or even lighting. It seems to be an assumption in these parts that a business automatically has a server, a number of clients and an online presence.
We're talking about the kinds of businesses for which having their back office and web presence in the cloud is a sensible option.
The discussion's irrelevant for the sole-trader firms and self-employed, where that PC is on the desk. They're never going to learn how to use AWS. It's the multi-person startups and high-growth SMEs where that starts to become a consideration.
Here is an argument for somewhat the reverse; small=cloud, big=DIY
https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/
My personal view is that (either way) it matters far less than ensuring there is a plan/strategy, and the right ppl and processes to execute & manage. You can make a mess either way.
We use cloud and we have 1 guys taking an axe to usage full time.
A lawyer and finance guys, 3 guys in upper management to fight the supplier on contracts and not forcing to buy service and resources we don't need every year.
We fly the guy with axe out to the US every year, cost us a fortune, mainly because he has a new axe when he's gets of the plane.
Works out cheaper then just "letting thing be"
Probably because of the axe.
I think Amazon will need to stop charging for unsolicited PUT requests or come up with some other solution before the script kiddies start hammering random buckets, rogues start hitting specific targets.
It seems a more effective modern day equivalent of sending postcards to businesses who have a Freepost address so I am sure it will be exploited now it's known.
Though why anyone would sign-up to any service which makes you pay for people trying to kick your door in, intentional or not, is beyond me.
I think Amazon will need to stop charging for unsolicited PUT requests
Right there I got a vivid picture of an AWS-excecutive vomiting blood right onto the conference desk, he is surrounded by his accolytes rending their clothes and throwing used cat-litter over themselves.
There's this thing called passwords. I know Google, and Amazon, and basically all these services have fancy OpenID based authetnication methods (or probably some non-standard Active Directory based thing for Windows), two factor authetnication in some cases, and so on. But I'm surprised S3 can be set up at all without a mandatory password. Obviously, having a hard-coded bucket name and password in your program is not terribly secure; but it's infinitely more secure than just having a bucket name then any rando on the planet can access your bucket.
Good on Amazon for deciding they'd refund for this. But, to be honest, I don't know if I could even call these accesses unauthorized insofar as the bucket was set to not require any authorization to access.
I will note, when I was in like junior high, I had words with the local authorities about this -- I got knicked for unauthorized access to a computer system. I pointed out I dialed this telephone number, it answered and did not ask for a password so I assumed no authorization was required. This was a terminal server, so I used it for some internet access, not poking around somebody's files. It made the police REALLY nervous when they handed me a sheet of paper to sign away my rights to an attorney and I said "I'll talk to you but I'm not signing this." They had my parents sign something to the effect that I'd refused to sign (I don't think they read it, so they probably signed away rights they were unaware of. No matter though.) They got even more nervous when I asked what I was being charged with, I pointed out I had definitely not met the requirements for the Computer Fraud and Abuse Act of 1986 (and at that point there were no newer laws on the books yet). As a bonus, their "computer guy" was out that day! When I described how a web browser worked, they said with a straight face I should get prior permission from each and every web site owner before I load a web page from their site. LOL. From their followup questions it was clear they had bigger problems, someone was in their systems trying to crack into military targets. I pointed out "I didn't do that". One of their computers off in the corner spontaneously blue screened, so I pointed over there and said "I didn't do that either". LOL. This was in fact around the time "The Cuckoo's Nest" covered so maybe it was the same guy poking through their systems. Since they couldn't figure out what to do they ended up deciding to fine me $50 (I'm not sure what specifically for, but OK I guess).
Sorry, but you have appears to have missed the point of this story.
* The bucket *was* set up to require authorization
* The requests did not have valid credentials, and therefore were refused (correctly)
* However, the bucket owner was still charged for the failing requests - tens of millions of them.
But security costs and is an integral part of the service. Hence the charge for processing credentials is incurred everytime credentials are presented to AWS, regardless of whether they are authorised, correct, unauthorised, incorrect. Thus that $1,300 (per day) is the charge for S3 security service usage…
/joke
What is troubling is apps usage of cloud services; does the (offending) app make clear to users the implications of its cloud usage ie. The app might be free and have no in app purchase, buts usage of cloud services is such the user is going to be hit with an unpleasant surprise.
Pocwierz really should identify the offending open source tool, particularly given it has been fixed, so people can look and better understand the cloud usage problem. I anticipate a need for app developers to list the (chargeable) cloud services they use and the billing rate for its usage; in this case the number of PUT requests per hour/day successful/unsuccessful and the amount of data written.
"To demonstrate the security implications of this, Pocwierz said that he opened up his S3 bucket for public writes, and in less than 30 seconds it amassed over 10 GB of data from numerous sources."
So, you open your bucket to the public and you expected what exactly? You use unnamed open source tools and they rack up writes? While I might agree with the injustice of AWS charges, the incompetence of their users is also a significant point of concern.
I don't think that's the point in the statement. This bucket demonstrates two things:
1. If you don't allow public accesses, you can still be charged for unsuccessful attempts.
2. If you do allow public accesses and an open source library uses your bucket name and people don't change it, they end up handing you gigabytes of their data. They probably don't want you to have gigabytes of their data. They have a security problem. I think this is what they were trying to say.
Bucket names are globally unique. [name].s3.amazonaws.com. The first to register it is the one who gets to use it. There is no naming your bucket the same as mine. The first one of us to set it up gets the name. In this case, their name was evidently used as a placeholder somewhere and people didn't update it with a real address.
This is a bit puzzling in the article. How did he come to have the same bucket name as that of the unnamed tool? Alternatively how did the tool work before there was an actual bucket set up with the name it uses? It looks as if obfuscation of the original report has succeeded to the point of incomprehensibility.
There wasn't already a bucket set up with that name.
The unnamed tool (I imagine some form of backup utility which can send to S3 amongst other places) probably defaulted to something like "changeme.s3.amazonaws.com" - no need to register something which is clearly a placeholder. Especially if doing so means that you'll get millions of probes and rack up an unexpected large bill!
(Yes, I know you shouldn't use as a placeholder something which is possible to register)
"attempt to alleviate this risk by avoiding short or common names for S3 "
Hash your human friendly name with a pinch of salt to get a random string of valid S3 Bucket name characters and of the S3 max bucket name length. Or an equally long but random string and use a directory service to publish the friendly name to the random actual name mapping.
As for giving Bezos my CC details ...
Well, yes, but, even if sounds like it's kinda hard and it might take a while, wouldn't someone attempt to write a tool that does a brute-force or maybe even distributed search for S3 buckets - just to deliberately mess with this "feature"? And once your bucket is on an "existing" list, then those l33t script-kiddies are going to stomp it.
I thought alleviating this (accidental use of common names) was one of the reasons Microsoft automatically uses names of the form:
Xyz.<your domain name>.on Microsoft.com
Rather than
Xyz.onmicrosoft.com
Along with defaults to user local paths, so my usage of “MyStuff” defaults to MyStuff.Roland6.onmicrosoft.com, whereas some else would default to MyStuff.ANOther.on Microsoft.com
So part of the problem here was down to poor coding in that popular open source tool.
"The takeaway is that anyone who knows the name of an S3 bucket can send it PUT requests, and potentially rack up massive charges for the AWS account that owns it."
The new wave of incoming denial of service attacks is going to be interesting. I wonder if other cloud providers are subject to similar issues?
I strongly suspect that the charging of non-authorised put requests in intentional. AWS charge for everything.
A moderately creative developer could communicate with their services without paying for it. For example, through using deliberately non-authorised comms. Such comms would be rejected, and therefore free, but each would still be able to upload some information in the headers. End result is free transfer of data.
Given that inbound bandwidth is one of the few things they don't charge for, this doesn't seem very useful. I am charged for each put request to S3, but if I put an object there and it is a gigabyte in size, I'm not charged any more for that. I would be charged to store it, but if I simply read it and deleted it, that is very cheap. Therefore, building a protocol around free unauthorized put requests wouldn't save you much at all.
Within the Billing and Cost Management console, you can set budget limits, and configure alerts at various percentages of the upper limit. Better still, the comparison is based on forecasted cost, not spent cost, so you can catch a misconfiguration or runaway progress before it actually costs you the money it might do.
If you don't put in these guard rails, you're a fool to yourself. It's saved my company from idiot mistakes before now.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
