Dev put AWS keys on Github. Then BAD THINGS happened Bots are crawling all over GitHub seeking secret keys, a developer served with a $2,375 Bitcoin mining bill found. DevFactor founder Andrew Hoffman said he used Figaro to secure Rails apps which published his Amazon S3 keys to his GitHub account. He noticed the blunder and pulled the keys within five minutes, but that was …
Aside from the obvious issue of storing credentials securely, why isn't there a pre-payment model for these services so that you can limit the financial damage to the amount of your outstanding credits?
This has always struck me as the achilles heel of cloud services - DDoS someone's trivial and seldom-visited website and you can stiff the owner for huge bandwidth and compute charges.
I wouldn't hand over my credit card to anyone on the understanding that unlimited amounts could be charged to it.
"Sorry, this site has been suspended because they didn't have enough credit in their account."
Except they don't have to put up a message of that kind. All they have to do is report the unavailability, not the reason why.
Even if pre-payment isn't desirable it would be reasonable enough to put a cap on things, trigger an authorisation request from the payer to keep the service running and incurring costs. That could be opt-out for those who want to take the risk.
Customers are going to demand things like that if Amazon stops refunding money for carelessness and mistakes.
"Even if pre-payment isn't desirable it would be reasonable enough to put a cap on things, trigger an authorisation request from the payer to keep the service running and incurring costs. That could be opt-out for those who want to take the risk."
FWIW Microsoft allow caps and warnings on Azure.
Aside from the obvious issue of storing credentials securely
And, as appears to be relevant here - actually bothering to set up non-privileged keys. If they were spinning up EC2 instances (and the dev seems surprised by it) then either he was using a key with permission to do so (i.e. it's been configured in IAM) or more likely was using his root keys, granting the attacker unlimited access.
Wonder whether he remembered to purge the keys from his commit history, a 5 minute window is pretty short...
Yup hadn't purged them fully (though he's now revoked) - https://github.com/andhof-mt/shriek/commit/799a62ed075954eac673322b9f69963ad815c4d0
@bigtimehustler
Looking at his post, I'm not sure they were just S3 keys, though it's hard to say for sure. Certainly can't find any reference (based on a _very_ quick google) to being able to fire up EC2 instances through the S3 API - though if it is true, that's some spectacularly bad design by Amazon.
But yes, either way, they definitely had too many privileges. Mind you, if you look at the average S3 tutorial online, the various authors all seem to think that creating limited privileges in IAM is too complex and skip over it.
You get buy pre-paid "credit cards" with a fixed amount of money on them. They are intended for people who need a credit card, but can't get one for credit history reasons, but need the use of one. I don't see why a cloud service wouldn't accept one.
Alternatively, if you use a genuine credit card, as opposed to a debit card, they usually have a credit limit built-in. The maximum financial damage might be higher than you would like, but it won't be unlimited.
To be fair, if you're that determined to 'Pre-Pay' your Amazon account, it's fairly straight forward.
Buy £50 of Amazon gift cards
Apply them to your account
Don't register a card against the account
Whenever your bill comes out they'll subtract from the credit, though you can probably expect some shitty emails if you do go over.
Pre-payment credit cards are not accepted by organisations that would want to apply debits to your card without your explicit permission for each transaction (or the card issuer will not allow them to be registered for this). I know this is true for Paypal since I tried to register one This is to prevent you from running up a large bill before they try to debit a card that only has a small amount of credit on it.
I routinely use with PayPal a "virtual" credit card (it's a service from my bank, it gives me a valid credit card number different than my true one) set to a max amount, and expiring after n months.
I never had issue registering and using it with PayPal. After all, whatever card you use, I guess those services checks you can pay before accepting the transaction.
> "I only had S3 keys on my GitHub and they were gone within five minutes!"
Assuming they got comitted to a git repo, I wonder if there were still in the version history. Seen that happen a few times on internal repos - people delete the file from HEAD, but nuking the history totally is much harder.
I wonder if a better strategy than spinning up the maximum number of instances would be to run them with much lower load, perhaps gradually increasing it over time. That way you may be more likely to slip under the radar and get more out of it before anyone notices.
1. Don't store credentials alongside code, ever. Change the code so that it picks up the credentials elsewhere, outside of what you commit to public (or any) repositories.
That's best practice, if not common sense.
Failing that, at least learn the ropes...
2. Use .gitignore to avoid pushing sensivie files.
Blaming it on the used framework is lame. But hey, everybody with little to no clue can call themselves "developer" these days and share the result of that for free with world and dog.
It is easily done, I accidentally committed a signing key to my local repo early last year before I realised that my .gitignore wasn't quite right.
It's the downside of automatically staging new files - though personally I think it's better than the alternative.
The first key difference is that I checked my history and purged it from my local repo before I pushed it.
(The second, possibly more important detail being that the repo I push to is also private.)
It is going to be the case that keys are going to be posted. The question is, why does AWS allow the default to be that someone who compromises an account is able to start 20 monster E3 instances in all 8 regions?
This happened to me (no, my keys have never been public and AWS staff were unable to find any) and AWS did remove the credit. However it took a lot of correspondence to have them set the number of available instances in all regions of my account except 1 to zero and in the region I use, set it to 6 instances (3 running, 3 spare).
In my correspondence I likened AWS to a credit provider who is delinquent in their responsibilities by letting creditors run up massive bills without even trying to limit the scope of their credit.
I recommend to any other AWS users that in addition to following the advice to cycle keys regularly they also contact AWS support and ask them to prevent instances from running especially in regions they are unlikely to use.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
