AWS wiped my account of 10 years, says open source dev An open source developer is claiming AWS deleted his ten-year-old account, wiping all the data. He believes this was due to a botched test of a script designed to prune dormant accounts. Abdelkader Boudih, who goes by the handle Seuros, is a software engineer who has produced numerous Ruby gems used in production systems …
Depends what you mean by "backup" really, doesn't it?
It's still a back-up when I export a copy of my database before performing an upgrade even if I don't move it off the same machine - the risk in that case isn't the machine failing. Backing up to another filesystem on the same cloud is still a backup if your working assumption is that the service isn't going to go away, which is one of the major selling-points of clown computing. Pulling it out to another cloud or a machine on your network might be necessary if it's a production system people are relying on, but that's also extra administration and hassle and for many situations it might well be overkill.
Of course the iron law of backups is that you don't know what you needed until you need it. Sometimes you gamble and win, other times, not so much.
While the fact that so many you circling vultures seem to have made it your life's work to dive bomb anybody you perceive violated the holy dogma of backup strategies, you might pause to re-read the article.
The subject of the article didn't claim that their only backup was deleted. The article says his AWS account was wiped and that it contained 10 years worth of work.
The AWS process for account recovery wasn't followed by Amazons staff, and they wiped customer data prematurely. Hopefully the author DID and does have an offsite backup, but losing both primary and secondary resiliency within AWS means placing great and unnecessary risk on the customers deployment.
Telling him his account was gone initially and then finding a copy under the couch cushions later also isn't a great look, but no surprise. Glad this customer had enough pull inside and outside the company(Amazon) to get them to look. I'm not famous, and not a big enough account to get the time of day. They'd probably never commit the resources if it was me, and leave me in a circular loop of fake support responses with a clever name like the Labyrinth of Minos.
But unless the account holder or the articles author says otherwise, I'm not jumping to the conclusion that the author didn't have an out of cloud backup of their AWS deployment. But simply locking him out of his account meant that self recovery wouldn't be an immediate remedy, especially is the author was using AWS for domain or DNS hosting. Having a backup of a platform dependent deployment isn't a magic wand, and fully fault tolerant multi-cloud hosting for a developer site seems like a fairly silly expectation.
So maybe don't jump to a trash talking conclusion when someone who is contributing to the industry's reputation is attached, and maybe the same will be extended to you under similar circumstances.
The lesson I took from this is that AWS may be ubiquitous, but using it safely still means handling a seemingly endless list of gotchas that isn't published anywhere all by your lonesome.
Whilst you might very well possibly be correct, the cynic in me thinks that if you had your own backups of everything, this would just be a grumble at AWS and carry on. You wouldn't need to make a big deal about "10 years worth of work", because you still have it all. You'd just create a new account (ideally somewhere else) and upload it again.
I don't see what would make it a newsworthy post if it was literally "AWS deleted several accounts, people complained"... Though, seems like some people do like internet drama, so I guess you never know
When however your 10 years' work involves having set up an entire infrastructure inside AWS that will take weeks/months/years to set up again, and such infrastructures tend to not be easily exportable (nevermind the cost of regularly exporting them for backup purposes), it's not that simple as "well, if you had your own backups of everything, this would just...".
Just pointing things out here.
Perhaps you are correct and Seuros had comprehensive backups of his work that AWS couldn't delete for him. I very much hope so.
But if that is the case, where's the story? Anyone who uses someone else's computer for their work should already be fully aware of the fact that things do go wrong and that the only support offered by the companies selling this kind of solution is often an unhelpful and powerless LLM. When you actually need help, all you'll get is "computer says no" and you'll be on your own.
So, either AWS nuked his account and he simply created a new one and populated it from his backup - end of story, or, AWS nuked his account and he experienced a catastrophic loss of his work - big story, but the fault lies totally with the user for trusting AWS.
The expectation that something you put on someone else's computer is 100% persistent, reliable and secure is naive.
The fact that the article contains this statement: ...despite repeated requests by Boudih to get temporary read-only access in order to back up his data... is what leads me to think that Seuros had made a fatal error of trust. If he already had comprehensive backups on his own computer, why would he need to back up the data he had stored on AWS?
The developer running the test typed --dry to execute a dry run only of the code, said to be standard practice across modern languages. But the proof-of-concept had been written in Java, which does not recognize --dry, and so the script executed for real, actually deleting user accounts.
Parsing of command line arguments is
a) language independent
b) the responsibility of the application
There may be a convention that a --dry argument performs a dry run but only if the application supports it -i.e. the application implements such a feature and parses the arguments accordingly.
There is a "--dry-run" standard JVM command line option which creates the VM but doesn't execute the main method, documentation says it might be useful for validating the command-line options such as the module system configuration. Not very useful to test what a tool implemented in java would do - as you correctly point out that would have to be something the tool supports.
Hot take: Any command or program that can delete user data, accounts, files, etc, even on a local machine, should have "dry run" as the default state, and you have to include some kind of --for-real-this-time flag to make it actually go, as a break-glass measure. So "rm -rf / somefolder" would instead give a list of what you're about to nuke (giving you a chance to catch the fat-fingered space!), while "rm -rf / somefolder --for-real-this-time" would actually nuke your drive. The concept already somewhat exists in the "--no-preserve-root" flag when you're specifically nuking "/", so why not extend that break-glass idea to all invocations of rm, or similar programs?
Accidents are much more likely to happen when tools are used without care, or with insufficient knowledge/skill. Safety features can mitigate this but may inherently compromise utility and/or efficiency of the tool as a result, making a worse tool for those who do use care and have gained the knowledge/skill (eg through study, training and supervision). Like most things, there are trade-offs; there is a balance to be struck.
I think requiring --for-real-this-time or similar for any potentially destructive operation would make tools worse and only result in it being used habitually, making it ultimately self-defeating. That is not to say there aren't times it is appropriate.
> only result in it being used habitually
My bet is that you'd soon find many, many ~/.bashrc etc with aliases that add --for-real-this-time every time ("it just makes everyday things easier, just like adding colour options to ls") and the owners of those rc scripts will forget that the option even exists and why it is there. Then they'll be "helpfully" passed on to the newbies ("just copy mine, then ls will colour things for you")...
One helpful refactoring later ("87% of the rc files in home directories had these aliases set, so we just moved them all into the global settings, now your personal files are shorter and just have stuff specific to your job role in them")...
Which reminds me, I must hunt down which global rc is setting ls colours on this server install, as typing --nocolor every time when I've ssh'ed in as a different service username is a pita: who makes directory names dark blue on a black background?
Indeed, which is why I'm calling bullshit on this part of the story.
Slightly disappointed that El Reg didn't comment on it themselves in the article.
I'm more likely to believe that someone at Amazon really dislikes Ruby (wouldn't blame them) as the real reason the account was rm -rf 'ed rather than this spiel about "Java doesn't support -dry".
Yeah, honestly that's the part that doesn't add up about this or the OneDrive case. Not "how did it happen" but "why specifically these particular high-profile accounts were the ones affected"... especially in the OneDrive case, the fact that the account belonged to someone making a competing product.
> especially in the OneDrive case, the fact that the account belonged to someone making a competing product
Wait! What? I think I've missed this story, but what product were they competing with? OneDrive? If so, why would you put your stuff on OneDrive, and not the product you're building?
That's not strictly true. Standard command line arguments are often provided as part of a language or runtime environment. Take PowerShell, for instance; it offers a whole bunch of standard command-line arguments. If you write your own PowerShell command, it will implement all sorts of options like a progress indicator or a dry run (using -whatif)and as long as you do it correctly (e.g. inherit from the cmdlet class), you don't need to code that functionality yourself. The runtime does it for you.
"Backing-up to the same environment you're backing-up? An interesting move! That's like backing-up a hard disk to the same hard disk."
Regardless of the hardware behind the curtain, any sort of remote account should be treated as one drive. If you have data on one drive locally and it goes poof, that's your data gone. If you have data stored at an outside service, if your account is terminated, that's your data gone. People are finding this out with consumer Amazon accounts. They return too many things (undefined) and their account is terminated along with all of the music they've licensed, audio books, movies, etc.
My domains are registered with separate companies than the hosts of my sites. I have multiple computers for different tasks partially due to software compatibility, but also so if one goes down, I have several others to use so I can order repair parts, check email, etc. It also means that recovery is much less complex and less time consuming. My accounting software runs on its own on an old headless Mac Mini with a Time Machine backup. I can swap that out and be back up swimming in the double entry sea in an hour or less.
... with a Time Machine backup. I can swap that out and be back up swimming in the double entry sea in an hour or less
Have you tried that lately ?
Not long ago I got asked to fix an iMac for a relative. Diagnosed that the HD was on its way out which was making the machine keep pausing when the HD hit a recoverable (with a few retries) read error.
No problem thought I, just stuff a new (or at least, new to this machine, I used a spare that was lying around) disk in, boot it up with the right keypress, and "Restore from Time Machine". Only, you can't do that now.
"Some time" ago Apple changed TM so it no longer backs up anything that's part of the base OS install - apparently it's something to do with the "secure enclave" they apply to the OS files now. The result is that the only option is to install the OS from an installer, you can restore everything else from TM as part of that, and then run all the updates to get back to where you were. Except that it doesn't !
Loads of Numbers files won't open because it claims it needs the newer version they were last edited with - even though it's running the last version that ever worked on that version of OSX, and the version they were running before I started all this. I had to come up with a workaround of opening the file in LibreOffice, saving it as an ODF file, then opening that in Numbers and resaving it as a Numbers file again.
Someone somewhere will have a burning sensation in their ears while I was dealing with this.
"The result is that the only option is to install the OS from an installer, you can restore everything else from TM as part of that, and then run all the updates to get back to where you were. Except that it doesn't !"
You can, but you need to not use online installers or let it update beyond a certain point. I have every major MacOS on CD/thumb drive along with printed instructions on how to make each of my Macs start from USB/CD. Support from Apple dries up faster than a puddle in the Sahara.
I'm fine with reinstalling the OS and then using TM to plug everything else back in. My critical business data is backed up fairly deep so TM is another layer that might be convenient. There's also loads of detritus (random files, not the troll) that could disappear and I'd never miss.
To be fair to Apple, they do have all the versions going back a looong time - and you can download even the more recent ones and do the install from local media (and I do have copies of all the versions I've used before.) But then you have issues like I described where Numbers apparently thinks many of the files that were created by that version were actually created by a later version and you need to update to open them - except you can't because you already have the latest version.
That's why I prefer a full system image restore (which is what I do with my Linux boxen if they ever need it) so I get back to an exact copy. I've done that in the past with OS X, this was an eye opener for me - and not in a good way.
At the same time those companies sure do gaslighting you as much as possible and promise a lot of things about redundancy etc.
In other words one would need to have a lot of time to research backgrounds for each marketing promise when all important details are behind an opaque wall.
What they say about repeating lies enough and they become...
And what points out the most to me is that these comments about not trusting some arrangements pop out every time after some catastrophic has already happened. And then time passes and marketing promises dogpile on top of old warnings and another person finds out the hard way. There is just too much money and incentives behind misleading people to desired direction.
I self-published my first novel a while ago but I can't sell it on Amazon because their identity verification system simply doesn't work. I assume there's some kind of AI nonsense going on - whether you use their "hold the documents up to the camera" approach or send them a scan of the documentation it will fail.
I even took screenshots of myself selecting "UK Passport" from the "identity type" dropdown and then the feedback after uploading the scans where the system informed me "you appear to have uploaded a UK Passport, please select that from the dropdown" and sent them with my support requests, but Amazon's goons still cancelled my account permanently and with no recourse because they "couldn't prove my identity." I mean, technically that was true but only because their system doesn't work!
Anyways, after whatever initially went wrong in this case the account cancellation might have been due to whatever clownshow is happening under the bonnet of their ID service.
It is so frustrating having arbitrary decisions made with no recourse in spite of having proof that the problem is on their side, but that's the modern internet I guess.
Also I was on my regular Amazon account that I've had for 25 years - with that purchase history Amazon know way more about where I have lived and what I do than the government.
Suffice to say I won't be recommending any of the corporations I work with adopt AWS for anything.
I have failed anti money laundering checks. Usually with organizations I have been dealing with for years. They really don't like it when you move house, so are not on electrol roll, and don't have any utility bills yet.
I would be a very bad criminal if all I had to show from my money laundering empire was my current net worth.
My experience of Amazon - admittedly only limited to deliveries - is that if something goes wrong what happens next is totally unpredictable. Unpredictable like a courier being sent out to collect the undelivered item for return. It's as if only the happy path gets coded.
I used to have an Amazon account.
If you don't use it, they close your account. You have to open a new one.
They did this to me.. twice. They closed my account again, and I went, "wtf?" again. I went to open a new account.
You can't open a new account with your e-mail. If your phone number isn't tied to an earlier account, they'll go, "There are multiple accounts associated with this e-mail, so for privacy reasons we can't do anything with it." Never mind that they all had the same address. The same name. The same e-mail. Just BS all the way. No new account at my e-mail.
No Amazon account really hasn't been so bad. Less crap to buy, less shit hocked at you. If it's on Amazon, someone is selling it on eBay for just about the same price - with free Prime shipping. Let them deal with huge mega corps. My life is better. :-)
Yup. I'm "banned" from Mechanical Turk for the exact same reason (inactivity on the MTurk account; the linked Amazon Account is still active due to a Prime membership)... and due to MTurk being SSN-linked due to tax payment requirements, even if I make a new Amazon Account, it'll block me from MTurk due to "SSN already in use".
I know someone who had this when attempting to buy something only adults should be able to buy (a knife) from Amazon.
There's ID verification there, but apparently for them it didn't work. It categorically refused to. When I tried, it worked fine for me, but I do think that there is method to the reasoning that ID verification is broken.
Even putting aside the valid argument that putting your eggs in one basket is a fucking stupid idea -- AWS shouldn't work that way. I've done over 50 hours of AWS training and worked in production environments with it. This makes no sense. I think we're missing information from both sides of this story.
I could easily believe that they could screw up weirdly, but at their scale, I would expect that if they did, there would be more reactions. More than one account would have a complaint, the AWS response would be more generic to avoid saying anything relevant to many accounts. So far, I'm not seeing anything like that whether in the news or from other people complaining, and I know multiple people with personal, low or intermittent usage AWS accounts.
Meanwhile, there are a lot of parts to this story which suggest that the account user was perhaps using the account in an unusual way, what with the payment from someone else for some reason which AWS didn't like, and that's not explained either because AWS tends to be happy to accept money and, from the scant details available, was accepting it for a while before deciding not to. Combined with the implausible story of a dry run that didn't work because of a mythical missing language feature which isn't a language feature at all unless you misinterpret a different language feature which wouldn't have helped and is only a thing for the one language they were claiming it's not present in, and I agree with the original post. I think we're missing information from both sides of this story. I wouldn't be surprised if AWS ends up getting some blame if we got that information, but I would be surprised if any of the explanations we've gotten for how AWS screwed up end up being the way it actually happened.
Sounds to me like someone didn’t pay his bill. Had multiple warnings, had their account suspended when nothing changed. Sounds liike it got reinstated, probably after providing valid payment details. All the bollocks about Amazons cloak and dagger stuff is BS in my opinion
If you're working on the cloud on a single account, this should IMHO be considered your "local" drive. This means you need 2 other backups.
And who the heck programs their program that they're testing on a LIVE environment to need an argument to prevent it from doing actual harm? The default should be NOT doing anything and requiring a specific argument (that is not easily mis-typed) to make it do something other than nothing.
(1) Version control
(2) Local backups
(3) Offsite backups
(4) Test restoring of backups
(5) (Annual?) disaster recovery tests of whole environment
(6) Update the process based on findings
How much of this actually gets done these days?
How much of this was actually going on in this ElReg story?
AWS launched in 2006, not 2016. It has been around for 19 years so far.
I find it ironic that his account was deleted so quickly, when I had to fight to get mine deleted earlier this year when I realized that I still had an account and data out there from 2007 that I had created for testing the service and storing large video files for a website, but never really utilized.
I'd bet you a large amount of money that if it wasn't for the fact that this story was widely publicised, all he would have got from AWS is "computer says no".
Just the same as when YouTube decides to delete someone's channel or take a video down based on false copyright or privacy claims. Or when Google deletes your account without any reason other than "terms of service violation" without telling you exactly which part of the TOS you are supposed to have violated.
That suggests to me that once you have deleted your data and closed your account Amazon still retains a copies of snapshots of your stuff. For now long ? Offline or online ?
I imagine if you wanted to be certain that your data remained private, encrypt locally before shipping to the cloud.
If your data is being accessed from applications running from virtual machines or containers in the cloud it's a bit more interesting but in practice I think it reduces to ultimately trusting your provider (even if only restricted to their not stuffing up too badly.)
Just to join in with all the other users here.
He apparently and experience professional that has no backups of years worth of work? Even if he had backups on his local computer or a server somewhere (even hosted on someone else's cloud, assuming he needs to use cloud, it would be something..
BAAAAAaaaugh!
Been saying for years, when you use someone else's machinery you're subject to their whims and when they decide to remove you, you have zero recourse and, in this case, not even a "Sorry about that."
I keep all my own stuff local, and spend what I need to in order to maintain my data right here in my own house. I've never lost so much as a byte in 30 years. Admittedly, a time or two it took freezing a hard drive and accessing it via a standalone disk reader, but nevertheless.
"I keep all my own stuff local, and spend what I need to in order to maintain my data right here in my own house."
I keep a set of backups at a family member's home 150miles away. Any disaster in common that damaged both of our homes would make my backups not such a priority.
I also wind up with loads of harddrives from old PC's. I wouldn't use them as a daily driver, but I'll wipe them, run a disk check and if they aren't spotted with bad sectors, use them for backups. I backup my latest data first and walk back until I'm out of disk space. It means for older things, I wind up having lots of overlap in the archives and newer stuff has 2-3 depending on the day of the week. It's been decades since I've lost anything. From time to time I come across sales for NOS HDD's in case lots that aren't large enough to hold much interest for people. They sell for really cheap and make suitable back up drives.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
