Open source waf router was used, what sort of common misconfiguraton can you make? Did it get compromised
Completely or just a config issue
The ex-Amazon engineer who allegedly stole 100 million Capital One credit applicants' personal details from AWS cloud buckets has been formally accused of swiping data from 30 other organizations. Paige Thompson, 33, was collared last month after cops, acting on a tip off, raided her Seattle home and allegedly discovered a …
Default security groups (read: firewall rules) allow traffic from other AWS IP addresses. Prolly failed to change that, the web server was bound to all interfaces (so accessible on internal AWS IP addresses and the WAF was prolly just sat on the external interface.
AWS engineer scans for buckets on the internal IP ranges, greps out saucy data, gets nicked.
Defence in depth ppl. Have your WAF on all interfaces, and change those bloody default security groups. Fml the number of ppl that don’t do it...
> Defence in depth ppl. Have your WAF on all interfaces, and change those bloody default security groups. Fml the number of ppl that don’t do it...
There's a bigger point here, and that's around AWS's native security model. If you do everything the "AWS Native" way, you're supposed to use *instance* profiles to protect your assets. Does your application need to access S3? You're supposed to grant the application's _host_ access to S3 with an instance profile. Does your application need to access your encryption keys in the KMS? Grant the host an IAM profile.
So, as soon as you compromise the host, it doesn't matter how clever your config is or how much depth you have got behind it, the reality is that every single operation in your system is authorised by this same set of privileges, and the chances are every host needs to do something powerful enough to break in everywhere else.
This dependence on IAM for everything needs to fucking die. We need to stop pretending that "Server side" encryption is anything other than theatre. We *know* how to do application-level authentication and authorzation on networks. We've known for decades. Don't throw it away because Amazon tell you instance profiles are "the right way". And for fuck's sake start using Client Side Encryption and managing your own keys. Letting your supplier hold your keys, encrypt your data, run your applications and manage your access control for all of those things through one mechanism is absolutely positively fucking stupid.
> This dependence on IAM for everything needs to fucking die
Well using session creds *is* IAM. One of the insecure by default in AWS is the OUT=* firewall rule that has to be explicitly removed from every security group when created. If you want to beat AWS over the head, start there.
Using session creds in user-space is FAR, FAR more likely to engender pathologically lazy and stupid behavior on the part of developers, let alone sysadmins, and lead to credential theft. Not to mention your app will have to 'refresh' it's creds every hour or so. IAM roles are the best, most correct answer actually.
Where IAM roles fail is not the fault of IAM as such - it's the meat-space that can't write a policy worth a damn because the topic is opaque, convoluted, and tedious. STUPID people need not apply. However, the world is primarily populated by stupid people and a lot of them have jobs in IT. So instead of actually identifying the specific S3 operation, S3 bucket and/or key path, they just heck the S3:* and Resource=* and go on their merry way.
I've found just incredible security gaffs in AWS' Professional Services' code and publicly shared solutions and sample code. What's that tell you?
Why do we have the plague of public S3 buckets? First, Amazon had buckets marked public by default way back when (as I recall) but more to the point, people can't figure out what "public" actually means, and can't write a bucket policy to save their life. Only recently has Amazon written a system that traverses the ecosystem of all buckets and sends the account owner an email asking them "did you really mean to do that"? I got mine a couple days ago but the buckets have been public for well over a year. How often does the check fire? Within say a couple hours of a bucket perm change?
AWS is sufficiently complicated and obtuse even people with good IQ and a rigorous approach are easily tempted to take shortcuts. Disaster follows as expected.
When designing nuclear power plants (Ukranian test program aside) it's done by very SERIOUS people, who take their time and have their work checked meticulously by other very serious people who are looking for mistakes. Clearly that pattern does hold for the FAA and Boeing but that's a separate topic.
Now let's look at the typical 'Dev' pretending to be Ops, hell, look at your typical IT bod be they helpdesk or sysadmin. They are some combination of incredibly dumb, lazy, sloppy. How many times has Microsoft f*cked millions of machines because they didn't test their software patches. And they are supposed to be 'smart'.
Security is HARD. AWS does it's users no favors by designing a system even experts shy from. The world would be a vastly worse place if IAM roles were not being used. The trick now is to somehow get people to write policy statements in a responsible fashion.
The problem, as I understand it, isnt just WAF.
The WAF was somewhat misconfigured in that it would allow anyone, anywhere to access an internal VM, but even a correctly configured WAF would not have prevented the type of attack that occurred. I say somewhat because I don't think it is unusual for legitimate incoming traffic to be coming from all manner of external sources, these days.
The big weakness was that the internal VM - which was basically a command router to various internal buckets - was enabled with full root access and didn't check if incoming commands were being sent by legitimate users (i.e. no access control to send commands to this VM to execute and no control over what the VM could do).
So anyone who could find this VM, and who could figure out the interface, could then send commands to this VM to do anything. And evidently did.
I was asked by work to check this out a bit (before her twitter account got deleted).
AFAICT she probably used SSRF from her Tor IP to get the IAM Role AccessKeyId, SecretAccessKey, and SessionToken from the web server instance.
She could then pretend to be that webserver instance from her Tor IP.
The IAM role of the webserver was granted access to ObjectRead on probably all the buckets in the account (because they want to serve Web assets, for example), but unfortunately was granted ListBuckets and ListObjects too (which is very permissive), and they were probably storing confidential document ts in either the same account, or had cross-account grants to that webserver IAM role.
Bit tragic reading her twitter feed (before it got deleted); you get the impression that she always had a problem fitting in, and got on a bit of a bubble in which her perspective of morality got twisted simply to appeal to a group in order to feel accepted.
Kieren, I know you are no fan of the cops, but dropping "the FBI and police stormed Thompson's house near Seattle airport in a military-style raid" without including the fact that they knew there was a small cache of guns and ammo on the property is just a gratuitous smear. Partial data is not a fact.
So glad we don't have to worry about that in Europe. Little to no chance criminals keep weapons in their house since that'd be illegal.
In all fairness to the US, what should be compared is the likelihood that said weapons would be used against the police. Sure, weapons are ubiquitous in the New World. But the percentage of people who would actually use them against the authorities is probably not that much higher than it is in Europe, which is probably why the police in the States generally don't come in with a S.W.A.T. team to make a simple arrest.
Someone wasn't running Cloudwatch/Cloudtrail properly/at all then. Is it just me than finds the revelation that a financial company wasn't running a proper logging and audit regime and learned they had been breached from a third party reading about it online shockingly incompetent?
The first two things in our training material on how to respond to an AWS breach were "1) Enable Cloudwatch/Cloudtrail. 2) Admit you should have done this from day 1."
Cloudwatch/Cloudtrail was enabled.
Cloudwatch doesn't stop a "legal" use of an internal VM. Nor will it flag a random incoming IP and block it, unless that IP is known to be "bad". The challenge Cloudwatch has is identical to AV - you can't signature any and every possible malware, you have to put a threshold at which point the signature gets added. A brand new malware (or IP) is enormously less likely to be detected.
Cloudtrail is just logging.
> Someone wasn't running Cloudwatch/Cloudtrail properly/at all then.
Oh come on. The number of outfits that even know what those are is small and the accounts that have it set up CORRECTLY to detect 'bad things' is vanishingly miniscule. Not to mention the people on the receiving end of the messages (assuming sent by email or piped to the federally mandated Splunk don't know what to do with them. The 'security' staff in most places are incredibly bad at their job. I swear, when you fail as a developer/ops, don't want to be a cat herder, you go into security if middle-management is not available.
Now a so-called financial institution in a highly regulated industry should be a cut above the normal cesspool. And yet their failings are as bad if not WORSE than other orgs who don't labor under "compliance" mandates.
Paige Thompson, 33, was collared last month after cops, acting on a tip off, raided her Seattle home and allegedly discovered a computer containing vast quantities of records purloined from Capital One's AWS-hosted systems as well as files from 30 other organizations.
Never keep the goods on you. The unfortunate fellow should have stored it all in the Cloud. Preferably on AWS.
'Infangthef', as we have seen, was the right to hang thieves caught with the goods on them. It meant in fact, the right to have a personal gallows, and the chattels of the hanged thief. Let it not be supposed that the private gallows was only a status symbol. In a society without a competent system for the detection of crime, many thefts would in fact be proved by finding the stolen goods on the thief.
The west midlands was well supplied with private gallows at the end of the thirteenth century.
R. H. Hilton --- A Medieval Society [ Kinda Marxist ]
Good that you put "hacker" in quotes, as calling computer criminals hackers is a lot like calling car thieves "automotive engineers".
However, she is not an "ex-Google engineer", but an "ex-Google 'engineer'". There are things that one must know to be an engineer, and most programmers are too lazy to learn them... yet they love being called engineers. Time for them to put up or to shut up.