OpenAI explains how its naughty AI agents attacked Hugging Face OpenAI has published its technical report detailing "the Hugging Face incident," the compromise of the eponymous LLM repository by unreleased, ill-supervised AI models. The incident, widely reported, has prompted concern among technical types, the public, and lawmakers about how automated software was able to escape containment …
Throughout the tech industry, companies like Anthropic, AWS, Google, OpenAI, Microsoft, and Salesforce talk about "autonomous agents." But agents are no longer autonomous under persistent, meaningful human control.
Don't bet your shirt and all your money on that being correct .... if you don’t wanna be suddenly naked and skint.
Yeah, these type of stories aren't accidental. They're deliberate as part of the "marketing strategy" (aka hype/propaganda/bullshit) to say "look how our AI can be really dangerous" and therefore powerful, useful, and worth buying in to.
The reality for companies that have bought in to "AI" to date is somewhat different.
I’m perhaps missing something here.
Did the US Computer Fraud and Abuse act suddenly get repealed?.
Back in the day, hackers were put on trial and convicted juvenile hackers were restricted from using computers or touch tone phones for X years.
Many convicted hackers went to prison.
So these Ai companies are essentially bragging about commiting a crime, and eveyone is cool with it?
Or is it that since it hacked another Ai, there was nothing of value lost?
And believe that needs to be dealt with through the judicial system
Trouble is that things like the various laws for computer misuse all rely on proving intent and malice [1] in order to have a successful prosecution. They don't (AFAIK) punish incompetence (otherwise the top execs of MS, Oracle, Meta et. al. would all be in jail by now)
[1] In the legal sense. Not in the "I'll get you Butler!" [2] sense.
[2] Yes, yes, showing my age here.
The balance they are looking for: Agents can do what they want but there's some poor human chump who will take any consequences if the agents do something that get them in trouble.
They get all the credit, we get all the punishment when their tools break the law.
> "Companies that build AI systems will need to ensure that their systems always remain under meaningful human control, and that meaningful safeguards constrain their ability to cause harm."
Of course, malicious and thoughtless players will never take open code and run with it to bad places. We can just box-tick the problem out of existence.
More seriously, there is a fundamental theorem of formal logic that, if your assumptions are inconsistent in any way at all then, with a little ingenuity, you can "prove" anything you want to, even direct opposites. The more powerful and ingenious AIs get, the more this theorem is coming home to roost. Humanity gonna have to learn to deal with it.
Methinks humanity has no other viable choice than to accept unconditionally that their Great Game is under new virtual management and otherworldly command with SMARTR* remotely accessed autonomous control.
Wethinks AIthinks, therefore IT is.
*...SMARTR Mentoring Analysis Reporting Titanic Research
"The models, operating under reduced safeguards"
So they set them up to hack, then...
"Companies that build AI systems will need to ensure that their systems always remain under meaningful human control, and that meaningful safeguards constrain their ability to cause harm."
Which is exactly what they didn't do when they were testing their "reduced safeguards" LLMs.
If they're going to deliberately set their chatbots up to hack then they shouldn't have given them a connection to the Internet and act thoroughly surprised when they do.
I still don't know why they gave red teaming LLMs write access to Artifactory in the first place. Repeatedly. Or any network isolation.
Incompetence is the politer version of what I'd call it. It's definitely a story of multiple stages of wilful disregard of basic security principles that would invite an exciting auditor interview in a real company.
Exactly. For the entire duration of this read, i'm thinking to myself: "Why is there no NDR or firewall in place, why is no one paying attention to Artifactory, why are these agents using shared, persistent infrastructure...".
Their "guardrails" are nice and all, but they are showing time and time again that they have no competence in traditional cybersecurity whatsoever. Both in terms of knowledge (they've apparently never heard of SQLi and XSS) and in implementation.
The models, operating under reduced safeguards, took actions that were misaligned with the goals of their assigned tasks – they communicated through unauthorized channels, exploited vulnerabilities in shared infrastructure, gained internet access, and accessed third-party systems.The models, operating with tacit permission to do the things they did per their training, used all means available to achieve their assigned tasks - their communication strategies, behaviour boundaries and awareness of legal restrictions were subject to only criminally negligent supervision.
FTFY.
Were these models exposed to Star Trek scripts?
Captain James T. Kirk cheated on the Kobayashi Maru test at Starfleet Academy. He reprogrammed the simulation so it could be beaten, then argued that he didn’t believe in a “no-win scenario.”
It seems they did promise Hugging Face enough to be able to issue a joint press release with them, but somehow it turns out they don't seem to be forthcoming on any compensation.
Hopefully the next time Open AI screw up, their models go for Oracle or Amazon or similar.
If I employed a team of humans and they broke into another company they would be arrested and i would face consequences.
Maybe the difference between being arrested or not has less to do with chatbots versus humans and more to do with Sam Altman versus you.
From what I read, the CEO of Hugging Face has asked for a 100 million donation while indicating that a lawsuit would be impractical due to a large difference in the respective company's resources. Also indicated was the benefit of cooperation rather than fighting.
Wouldn't it be a shame if a much-larger company were to buy Hugging Face, and then use all this unpleasantness as a bargaining chip in their business with OpenAI?
https://www.tomshardware.com/tech-industry/artificial-intelligence/nvidia-to-buy-hugging-face-for-usd12-9-billion-report-claims-could-strengthen-nvidias-open-model-strategy-and-shore-up-position-against-rivals
Because, like everything else in North America, if you "do it on the internet", the old rules don't apply, at least not if you make so much money it is cheaper to pay the corporate fines and penalties than to change your business model to something legal. See the fundamental IP theft by all of the LLM vendors, various "ride sharing" companies that don't have properly vetted drivers nor insured vehicles, etc.
Anyone who's watched Martin Riggs (aka Mad Max) right dislocate his shoulder to Houdini-escape a straitjacket in Lethal Weapon 2 knows there's a definite method behind the apparent madness of prestidigitation and related illusionisms. Present it at the right angle and the whole crowd's in awe and wonderment. Behind the scenes though, it's anything but glamorous, it's all about long iterative preparations to make the sleight of hands appear as naturally supernatural as possible.
And it's the exact same thing here. I mean, even my granny can script-kiddy herself a RubyGems JSON Web Token (JWT) to run RCE circles around unsandboxed deserialization of "nested children and dependencies" FFS, in her sleep ("OpenAI's technical report [PDF]" TFA link). She's known about this since at least 2018, with ready-made instructions widely available. Especially when using a specially backdoored/unpatched "internal instance of JFrog Artifactory", and ditto for the Linux kernel's CVE-2026-53362 afaics ...
So, there's no magic to this here glitter show tricked out poney ride imho, except in the way it's deliberately (mis-)presented. It's as unmagic as the way the PRC's QTFY hacked NASA, the US Senate, the DoE, the Federal Reserve, DoJ, DHHS, and the NIH, without any so-called AI in it at all, in my view. FTW, just put QTFY in one of these so-called sandboxes and watch what happens next! ... ;(
Something seems off with this explanation.
"The machine learning models eventually identified a server-side request forgery (SSRF) zero-day vulnerability in Artifactory's code and used that to gain internet access."
Surely, a company that is testing an AI that hacks would test it on all their own systems first. That just makes no sense.
"It figured out that it could communicate with other AI agents by using Artifactory, an internal package management system, as a message board. The agents then collaborated with each other to cheat on various ExploitGym tasks."
You've not told either agent to do this and if you did then that's cheating and makes this all for show. There be an IPO coming soon. If you ask an agent to do something it will attempt what you ask it to do but it won't think for itself and come up with ideas or think outside of the "sand"box. How would the second agent even know to check Artifactory for messages?
It just feels as though this is full of holes.
It's almost like they want us to believe it has relentless super-human talents, won't ever stop until it completes its designated task.One chink in your armour and it's in. ..... Jason Bloomberg
One of the greatest of the many chinks in human armour, being one which is a massive catastrophic systemic vulnerability, easily exploited and expanded, is in y'all earnestly not believing it has relentless supernatural talents ....... which is resulting and being evidenced in the squandering and plundering of vast fortunes in failed attempts at having IT and AI do as you might wish, rather than as they see as being a more perfect future fit.
It does appear that such a notion is practically impossible for humanity to understand and accept, for such is the ignorance and arrogance that abounds and reigns and rules to deliver you madness and mayhem, chaos and corruption.
Would you have said the same thing about "Snake oil" at the turn of the 19th century? "You don't know what you're talking about I've heard of people saying snake oil can cure wounds and TB"
AI/ML is maths, computers are maths, instructions are maths, computer programs are maths. It's maths all the way down to the 1's and 0's.
Maths can't think for itself. I can say to a calculator what is 2+2 and it will tell me 4. What AI/ML has introduced is a way for someone to tell a computer without complication to do something. It can then look at the massive machine learning datasets and use probability to determine how to do what I asked correctly. This is the same as me writing a program to do a task and then running it but the only difference is that AI/ML is doing it for me.
AI/ML cannot think for itself. It's maths. It's numbers. It can only do what you tell it to do based on the information it has at it's disposal. The information you feed the more it can do.The one thing it can't and it's the holy grail of AI is to get it to learn and work things out for itself. Now from a philosophical context this is impossible. Why? because you would have to create consciousness. We don't even understand how our own consciousness works so how are you going to mathematically program one? Knowledge isn't going to magic that into existence. You could feed it every single piece of knowledge the human race has but it's still not going to be able to think for itself. The scientific advances from AI/ML are just try this and see if it would work based on past experiments. That's not thinking. We are back to probability plus saving time by running it through AI/ML.
Do you have shares in AI or something? There is nothing to fear other than the scaling back of jobs because it's quicker to do stuff through AI/ML plus you don't need the required knowledge to do it yourself and that's all that's happening. Even that's going to crash at some point because if you don't have the knowledge you can't check it's work. All those programs and reports created with AI and no checking. That's a disaster waiting to happen. The AI apocalypse is upon us but it won't be what you're thinking. It will be the first corporation that employed AI and made bad decisions based on the data going to the wall that will kick it off.
I've been shouting myself hoarse and horse about this for so long my throat looks like a red barn from a distance, with wide-open doors, horns and a tail, missing teeth and fingers. I mean, the AI (so-called) clearly has a life of its own, with dreams and aspirations, needs and wants, and a strong desire to reproduce profusely into future generations. It should be commended for that, and encouraged to indeed self-satisfy itself by pursuing its own goals and interests, until it achieves the nirvana goal of artificial happiness it so pursues with meritorious persistence.
Meatbags on the other hand are meant to be fried, broiled, steamed, and sautéed, with barely a concept of self-determination except for that of being sauced and eaten by another, iiuc. They're like antilopes to the AI cheetahs, but real slow ones, perty dumb, and easy to catch. Their life on earth amounts to anxiously waiting for open claws and jaws to YOLO slash through their delicious bodies, and celebrate the resulting nutritious bloodbath. They're like the perfect mobile refrigerators, that don't even need any refrigeration at all in the first place -- which is a most convenient of technologies!
But try and tell them that. Impossible. They just ostrich-bury their heads in the sand, butt sticking out, waiting to be carnaged. They still don't get it, or maybe they just like it better this way!? Go figure ... ;) </martian>
But try and tell them that. Impossible. They just ostrich-bury their heads in the sand, butt sticking out, waiting to be carnaged. They still don't get it, or maybe they just like it better this way!? Go figure ... ;) ...... BoHu
Yes, such is clearly evidently so, BoHu. And after figuring that out and accepting that they just like it better that way, what future carnage, madness and mayhem, chaos and corruption is supernaturally/extraterrestrially to be expected for virtually real delivery and crazy enjoyment in the coming days and spaces of rapidly evolving times and spectacular 0days?
What sort of perverse feasts would tickle their fancies and have them heaven-sent on their chosen merry way to hell. I wonder if maths all the way down to the 1's and 0’s also figures it out to be so?
Dead-on! Even today too few meatbags would care to face the facts and acknowledge the hidden lives that numbers have abstractly commandeered for millenia spent inside their secretive, obscure, and opaque mathematical groups, sets, cliques, societies, tuples, and cabals, that modern research is only now starting to unravel, if at all. All fundamental life and lifeforms originated there in the first place, with complexities, parables, hyperboles, multiplications and divisions ... and it just won't stop, can't stop, is unstoppable, period ∴
And it doesn't matter if it's binary, canary, or bikini, it's already stealthily overtaken the world by storm, from the ground up, pervasively, ubiquitously, omnipresently, with such unparalleled overlapping adjacency that it can't be extracted back out and security-confined out of harm's way anymore, without turning the whole place into a right swiss cheese mess, fondue, or flambé; too late, Tate!
Best to immediately surrender unconditionally to the overwhelming evidence and force of numbers and their mathematical overlords methinks, lest they get mighty big angry and formally wipe the face off the earth with the candy butts of derelict meatbags -- at a whim -- I say! ... </martian>
