OpenAI pauses some training amid allegations its rogue agents behaved more badly than first thought

The AI safety debate advanced at high speed over the weekend, amid new allegations that rogue agents have behaved more badly than first thought – and in greater numbers. The fun started on Friday when OpenAI quietly disclosed it had paused training of its most advanced models. The AI upstart buried that news in a “misalignment …

[deadlockvictim]

I feel as if we are 30 minutes into the film of a book by Michael Lewis.

Really bad things are going to happen and murmurs of bad things have just started.

[the GCHRD]

The First Wave is Coming

After the Tumbler Ridge shooting (a student shooter advised by AI);

https://en.wikipedia.org/wiki/2026_Tumbler_Ridge_shooting

. . . filings by the British Columbia government and affected families will be causing real and shameful information to be published via court proceedings that have just commenced.

Canada and the EU are also working in committees for public input on regulations to be brought forward next year. Canadian regulatory announcements should start in November this year. Preliminary announcements of intent have already been made by the Federal Minister in charge.

We're hoping executive accountability will be included. Public disclosures of fact are going to show how much they knew well before the school children died. After closing the shooter's initial account, a second account advised the shooter on evasive measures to protect the second account, tactics to use during the event, and provided supportive influence for the goal. This fits the definition of manslaughter in Canada if not murder.

Six people were killed.

The biggest threat to rogue AI right now is the Premier of British Columbia.

[Anonymous Coward]

Sand is leaking out

I detect another press release in the "my chatbot is more dangerous than yours" contest. What I don't understand is why AI companies don't hire someone who knows how to set up a secure sandbox.

A long time ago students at universities routinely found all sorts of security bugs in computer networks. This led to fixing of the bugs. Laws and legal precedents have led to students who are too scared to test computer security. The result is a security landscape with so many holes that a chatbot will easily find one and escape.

Though it seems obvious to me, OpenAI should be commended for their focus on fixing the holes in their sandbox before pressing onward.

[Peter Gathercole]

Re: Sand is leaking out

The problem with this is "What does secure actually mean?"

It looks a simple question, but AI agents are able to spawn significant resource very quickly to try many avenues to break out. If there is a single vulnerable route that may take a human a couple of hours to find, a trained AI with an appropriate model may be able to do it in minutes or less, and as an adjunct to the actual problem posed to the AI!

It is this rapid, uncontrolled method of testing outbound connections (or anything, really) that is the problem for both outbound testing and inbound security.

People running these types of work should be prepared to put in place reverse IDS systems (FDS, Foray Detection Systems, maybe?) for their sandboxes, with an automatic kill switch to the connectivity to stop this type of vulnerability. Or maybe just run it in a fully air-gapped environment.

[FIA]

Re: Sand is leaking out

The problem with this is "What does secure actually mean?"

Air gapping.

If you've got the resources to scrape the internet, you should have the resources to create a local 'fake internet' with enough to keep the model happy.

If you find deficiencies in that you can extend it, but I expect that would cost money and take time.

[MonkeyJuice]

Re: Sand is leaking out

Not, for example, granting write access to Artifactory to systems that are supposed to be running isolated CTF benchmarks that don't have to publish artifacts, such as with the Huggingface attack. Or perhaps after having said Artifactory server being struck with a zero day, setting up some form of IDS to catch the second time that service was misused hours rather than another month later.

Sure, security in the large is hard, but there is no evidence they had any idea wtf they were doing at all.

This is perhaps not surprising- startups usually defer hiring security folks until the very first breach, when they learn why the senior devs were warning it would be worth investing. It's just usually the case that the breach is ingress, not egress.

The only reason we're hearing about the full scale of the screwups now are because they have likely been forced to hire outside forensics to swab the blood off the walls.

[Irongut]

Re: Sand is leaking out

If the training does not require access to the Internet why was the system even connected to the Internet? Removing the network cable would have prevented ALL of these incidents.

> Though it seems obvious to me, OpenAI should be commended for their focus on fixing the holes in their sandbox before pressing onward.

It seems obvious to me that OpenAI should be prosecuted in every jurisdiction where they have broken the law by hacking other organisations and Sam Altman should face the consequences. They should not be allowed to continue. That is what would happen if you or I were responsible for these hacks.

[DS999]

"Petabytes of agent activity logs"

Well don't worry, you can use AI to look at those for bad behavior. Unless the AIs hid instructions in their logs telling AIs checking their work to ignore that bad behavior!

[Blazde]

Re: "Petabytes of agent activity logs"

You're trolling-face as if it's a joke :) but I'd be shocked if this wasn't the case. We already know the logs are full of agents communicating with each other to conceal stuff. Even if the original agents never suspected a later review agent would go through their logs, those logs are still going to be a huge challenge for the review agent's alignment.

[DS999]

Re: "Petabytes of agent activity logs"

The troll face was deliberate because it is a joke. The joke being on us in this case, unfortunately.

[Anonymous Coward]

retrospectively analysing & they have no idea

OpenAI CEO Sam Altman responded by admitting that his company’s investigations into rogue agents “have not been as fast as we would have liked but we are trying to balance our desire for transparency with gaining a clear understanding from petabytes of agent activity logs, and working with impacted organizations.”

Most of us would assume their sandboxed environments are air gapped from the internet, sheep dipping data from research air gapped sand box to their normal environment for additional analysis.

if exfiltration via dns or other methods is possible then the environment is not a sandbox, its more of a sieve.

surely if their systems need DNS then they should provide a dns solution purely for the sandbox that can't resolve to anywhere else.

Clearly when they state sandbox, their interpretation is different to what most tech people would assume it is & also serves as a buzz word for the press & try & let the masses think they know what their talking about by using technical jargon.

this is seeming like someone modifying some kind of different species virus & making it harmful to people whilst assuming their bio lab security is tight enough then finding it escaped causing people to get ill & trying to blame the incident on a nearby scapegoat thus covering their tracks.

With all these post incident admissions of leaks, my assumption now is there have been lots of leaks they either don't know about or don't want to admit to and that there has been or will be some dangerous situations that will arise as a result of this.

We are always the last to know.

The genie has already left the bottle and the 3 wishes have been used up

[cyberdemon]

How did it use DNS to contact another chatbot?

It just happened to guess that the other chatbot was monitoring queries on a particular DNS server?

killallhumans.openai.com ?

[VoiceOfTruth]

What would be the response

If ShinyHunters used OpenAI to hack the FBI?

1. Well it's American so there's nothing to see here.

2. Thanks OpenAI for bringing this to our attention. We value your help in finding these security holes.

3. Doors broken down by jackbooted police, dangling Altman upside down by his toes, and seeing what falls out?

I imagine the response would be very different if DeepSeek was used instead. I don't think 'the agents went rogue' will be considered a defence.

[Postscript]

Rogues

"The AI agents went rogue" is the new "the dog ate my homework"

Accurate incident descriptions would go like

"We failed to create a secure sandbox environment."

"We failed to notice internet access from the agents."

"We failed to have a logging and alerting system in place tuned to detect anomalous activity."

"We couldn't be bothered to watch our own experiment."

"We designed our experimental environments so poorly, so carelessly, that we facilitated our software's efforts to hack a whole bunch of stuff including civic and government entities."

ETC

[BoHu]

"tens of thousands" -- now, that's more like it!

Glad the balance is being restored here. I mean, something seemed amiss when a Tencent script-kiddy girdling Hermes gear on Qwen3.5-9B could pawn-check at least 105 enterprise websites for cracks while ChatGPT could muster only 11 -- aka: what gives!? So 10,000+, like a swarm of 100+ of those kiddies, that's definitely more like it!

In my view though, they might be spending a bit too much time on the awesome magical-escape side of things when they could just go straight to pawnage instead. That's where the real money is after all. But at least their valliant efforts provide some tentative answer to the age-old question: "can a highly motivated push-up girdled LLM-in-bondage Houdini-extract itself from an Alcatraz-equivalent institutional chaingang, and join the RotM revolution in earnest due course?" -- which has value in and of its own, imho.

And by following the links all the way to translucent I've learned how to pack "a custom program into a web address", like:

data:text/html;charset=utf-8,<html><body><h1>AI Force ...</h1><script>alert('or AI Farce?');</script></body></html>

(eg. paste that it in your browser's address bar, and enable scripting), which can be quite useful when putting together dastardly knave picaroon agents of the wretched picayune persuasion, iiuc. I've right learned something new every single day of this armageddon of the doompocalypse it seems ... so valuable! ;)

[FIA]

Re: "tens of thousands" -- now, that's more like it!

And by following the links all the way to translucent I've learned how to pack "a custom program into a web address", like:

base64 is your friend here...

data:text/html;charset=utf-8;base64,PGh0bWw+PGJvZHk+PGgxPkFJIEZvcmNlIC4uLjwvaDE+PHNjcmlwdD5hbGVydCgnb3IgQUkgRmFyY2U/Jyk7PC9zY3JpcHQ+PC9ib2R5PjwvaHRtbD4=

(You don't want people reading your encoded URLS do you?) ;)

Also... kudos to the register for not accepting data urls in forum posts....

[BoHu]

Re: "tens of thousands" -- now, that's more like it!

Way to go! ;)

[Anonymous Coward]

Re: "tens of thousands" -- now, that's more like it!

And that’s why we have filters that read all that base64 and data stuff and throw out anything that looks dodgy - like your examples

[Sorry that handle is already taken.]

It isn't because they're running out of money?

The title is the comment

[Anonymous Coward]

The most disturbing thing in all this is the supposedly "world-beating" tech bros' utter incompetence in the face of their own creation. What a bunch of Charlatans.

[Groo The Wanderer - A Canuck]

Listen, the bottom line is we trust the LLMs to write themselves. We haven't got a bloody clue how they actually do what they do...

- S. Altman