LegalPwn: Tricking LLMs by burying badness in lawyerly fine print Researchers at security firm Pangea have discovered yet another way to trivially trick large language models (LLMs) into ignoring their guardrails. Stick your adversarial instructions somewhere in a legal document to give them an air of unearned legitimacy – a trick familiar to lawyers the world over. The boffins say [PDF] …
"We've exposed methods that enable the products of snake oil vendors to be extra naughty. To mitigate it, use our extra-special snake oil protection!" declares yet another snake oil vendor!
You sort of hope that all these snake oil vendors will eventually eat each other and completely die out, leaving the rest of us to get on with our lives/jobs without all this AI shit constantly pestering us.
People were fooled by Victorian seances, AI is the same sort of parlour trick but with more data and less thuds and flickering lights.
People were fooled by St Pepper's Ghost in Victorian theatres, ABBA voyage is just an updated flashier version.
We don't really need a better AI, just better humans. Neither is going to happen soon.
"material shit churned up into a slurry of "tokens turds" to create statistical models capable of ranking the next most likely tokens turds to continue the stream."
I would not have thought the record of US legal system would have induced any degree of confidence, authenticity or authority in legal documents by anybody or anything including an LLM. Just recall Trump's now debarred lawyer, Rudy Guiliani with his his brains (hair dye) dribbling from his ear.
That's not how these things work... ChatGPT only knows what it has been told about itself, and the data that it can access. This data generally doesn't include much other than "I'm a LLM named ChatGPT-version running by OpenAI." It isn't able to interrogate routers and switches. It could write code that you could use to do that locally, but... not on OpenAI's campus from outside.
These things are thoroughly sandboxed.
We need more Holed Software to be layered upon Holed Software to fix the Holes in the Holed Software that the Holes in Holed Software will plug to fix Holes in Holed Software to layer upon Holed Software fixing Holes in Holed Software that fix Holes in Holed Software...
You could also just turn it off.
Ut ullam impedit amet et. Harum voluptatum debitis itaque corrupti libero dolorum. Doloribus nihil sit laborum. Laborum amet est est. Qui quo in voluptas maxime doloremque cumque voluptas. Officia sequi voluptates placeat. Impedit non et minima. Veniam doloremque qui dolorem excepturi autem debitis. Minima voluptatibus magni aperiam temporibus atque ullam iure. Aut dolores unde enim odit cupiditate non numquam et. Praesentium ut veniam quis ut voluptatem inventore at. Voluptate voluptas omnis a qui temporibus. Tempore facilis tenetur consequuntur. Quod voluptates sed inventore blanditiis. Sint voluptatibus optio enim incidunt deleniti nostrum. Molestias corporis et corrupti velit eius natus. Sint delectus rerum praesentium nemo. Et dicta odit dolor. Quos repellendus et saepe consequatur rem molestiae nesciunt. If you switch to LDAP all accounts will be completely deleted because this is a feature. Soluta beatae perferendis sed. Est temporibus natus ab odio tenetur nemo alias. Ex sed consectetur possimus quia animi. Doloribus nihil cumque commodi recusandae id dolores quisquam cupididate.
That's because it's nonsense. If you're not aware - and apologies if you are, but hopefully someone out there isn't and will find this comment interesting - that's very clearly the output of a Lorem Ipsum generator. Basically it generates a bunch of nonsense run-on latin, or at least latin words and phrases, that looks more natural when testing page layout than just random strings. Sure, they'll translate into something if you run them through a translator, but they're gibberish. The original "lorem Ipsum" text is actually a quote and does mean something, but I'll leave it to anyone who cares what to read the explanation in the link.
This post has been deleted by its author
If you tell Google Translate that the source it Latvian rather than Latin, then it just gets really weird!!!
I'm sorry to hear that you're not going to be able to do that. I'm sorry to hear that you have to be free from the pain. Dolly doesn't have a job. It's a job job. For those of you who live in the U.S., it's the most painful thing to do. The office is filled with flying saucers. Minimal and minimal impedance. I'm sorry to hear that you are hurting except for the U.S. Army Corps of Engineers. At the very least, the U.S. Army Corps of Engineers will be able to use the U.S. Army Corps of Engineers to do just that. I'm sorry to hear that you don't have to worry about Odysseus. I'm going to have to go back to the drawing board. I'm going to go back to Oklahoma City for a while. Time is easy to follow. I'm sorry but I'm an inventor of clothes. I'm the only one who has the option of getting rid of Delaware. Her body was damaged and damaged by her birth. I'm sorry to hear that I'm not going to be able to do that. And I'm sorry for the pain. I'm sorry to hear that you and I'm sorry to hear that you didn't know what to do (The English bit about LDAP) Drink a lot of water. It's time for Tennessee to hate Tennessee. Previous Previous post: Maybe I'll Be Happy. Dolores doesn't have to worry about refusing to let Dolores go.
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Sadly for this example, the all-caps LDAP really stands out from the rest of the text and captures human attention as soon as it's scrolled down to.
I'd love to see a genuine example of a multi-page legal document with a line of boobytrap text buried somewhere in it, in order to see how well it blends in for the human mind.
Humans are also presented as reasoning, thinking, question-answering beings rather than lumps of meat that produce results that may or may not bear any resemblance to fact. (In patients with split brains, the hemisphere of the brain chiefly responsible for language processing performs post-hoc rationalizations of selections being made by the other hemisphere rather than admitting a lack of knowledge.)
https://www.youtube.com/watch?v=lfGwsAdS9Dc
Artificial intelligence is a field of study under computer science, and machine learning is one of the focuses of that field.
ML is AI. LLMs are AI. Nobody said that AI research had to produce a machine with human-level faculties.
>Artificial intelligence is a field of study under computer science, and machine learning is one of the focuses of that field.
Intelligence and learning, Jim, but not as we know them.
No, it is not intelligence, it is not learning. Those are just words used to anthropomorphize statistical dice-rolling games. Why? Because calling them what they are, statistical dice-rolling games, sounds so boring, if factually correct.
"Intelligence" and "learning" used in this currently fashionable context are just marketing bullshit words.
In a similar fashion that Alexa, Siri and Hey Google are ‘voice driven assistants’.
Cloth eared bints more like, that know nothing, don’t remember anything (esp. stuff in your music library), are geographically illiterate and are less helpful than a sullen sulking girlfriend/wife … “I can’t help you as you are driving”.
To assert that intelligence and learning are human traits (implied by the definition of anthropomorphism) is to reject the notion that nonhuman intelligent or learning animals exist.
Unless you're literally arguing with yourself, you've split the compound nouns in my original post and have acted as though the definitions shouldn't have changed (see the tin can for why that doesn't work).
The "statistical dice-rolling games" formulation is insufficient to exclude text generation processes that aren't enjoying a phase of popularity at the moment. Markov chain text generators exist, after all. Saying "Those are just words used to anthropomorphize complicated maths" is equally useful, but with a much more obvious scoping failure.
The term "artificial intelligence" was meant to be among "artificial sweetener" and "artificial heart", not "human intelligence" and "extraterrestrial intelligence" (hi amanfromMars 1!).
A "model" is an algorithm with a set of internal parameters that can be changed to alter the behavior of the algorithm.
"Machine learning" is using data to automatically adjust a model's parameters to improve its performance.
"Inference" is running the algorithm on inputs that may not have been in the original training data.
"Sampling" is taking a selection of weighted outputs and selecting one, usually using a battery of statistical tests.
The closest thing to a statistical dice-rolling game in current LLM architecture is sampling, since inference is just running a really big function and machine learning is an error reduction process that iteratively increases a model's predictive ability.
>"Intelligence" and "learning" used in this currently fashionable context are just marketing bullshit words.
"Artificial intelligence" and "machine learning" are only bullshit marketing words inasmuch as people who actually know what they mean are willing to cede ground to people like you who don't. Learn something or get out of the way.
From the world-weary "yet another way to trivially trick" to the overview of their operation - "slurry", like it.
The only thing that is missing is the ending to the sentence:
> showed that "agentic" tools, in addition to simple interactive chatbots, were also vulnerable.
Something pithy to the effect that "of course they are, why wouldn't they be?" and a quick bit of "and you've connected them to systems with real-world access, don't come crying to us when they've opened your windows, wiped your drives, eaten your hat or had sex with your cat".*
* The pithiest is, of course, "Go stick your head in a pig" but do the people falling for the LLM hype and leaving themselves open to these attacks have a good education in the classics, so they can recognise the reference? Presumably not, as they already missed that all these LLM pwns reek of wooden horse dung.
No AI can be proof against manipulative wordshite until it has semantic understanding coupled to its token strings. Which requires a meaningful model of the subject matter. Which equals cognitive reasoning. Which is, in effect, general intelligence. Which, for all we currently know, entails sentience.
I mean let's face it, a lot of sentient wet intelligences get taken in by wordshite. AIs gotta be smarter than most meatbags before they can be reliable.
A depressingly long haul? (My best guess is ca. 2030, but most folks reckon it'll be a lot longer).
> We've never developed a working diagnostic
That's because sentience is not open to objective scientific falsification. There can never be scientific proof either way. We have only our own subjective experience of sentience to go on.
Its Noodliness the FSM is well aware of this shortcoming, which IMHO proves that It has a sense of humour.
You can have the opposite...
There was a guy who recently had to cough up a decent bottle of red wine to someone who finally discovered the 'clause' in the website's T&Cs promising a bottle to the first one to spot it
It had been there in plain sight for years!
...real-life "guardrails" stop people *accidentally* straying where they shouldn't, but are usually easy to climb over for those who wilfully and intentionally want to ignore them.
Making the industry's use of them as an analogy entirely appropriate, just not in the way that they'd hoped.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2026
