Psst … wanna jailbreak ChatGPT? Thousands of malicious prompts for sale

Criminals are getting increasingly adept at crafting malicious AI prompts to get data out of ChatGPT, according to Kaspersky, which spotted 249 of these being offered for sale online during 2023. And while large language models (LLMs) aren't close to creating full attack chains or generating polymorphic malware for ransomware …

[trindflo]

AI is easily misdirected

It shouldn't be surprising considering how easily humans are misdirected. Give a competent psychologist or hypnotist a little while with a human and see how thoroughly scrambled the human's thinking becomes. Make the AI as paranoid as a decent IT person and it starts cursing at users.

[EricM]

AI != Human

Maybe I missed the irony tags in your post, however .,..

This is a failure of thought many persons make while judging AI behavior.

There is nothing human, there is nothing like rational thought or understanding in AIs.

There is instead some pretty clever code and a lot of data, mostly trained statistics, which generate outputs from inputs.

If you ask a person to repeat a word over and over, it will not start telling you stories from its years at school.

Some AIs start spilling their training data.

https://www.theregister.com/2023/12/01/chatgpt_poetry_ai/

The failure modes or the way to mislead a person vs. AI into doing something unexpected/stupid/dangerous are completely different between humans and AIs .

Answers given by LLMs do seem to be pretty sharp at times.

This is an illusion.

[Mike 137]

Re: AI != Human

"This is an illusion"

Indeed it is. But we're still being fooled by our misunderstanding of the Turing test. It isn't really a functional test -- it was a thought experiment only. Where it falls down in practical application is that it's entirely dependent on the perceptiveness of the observer. So when an LLM spouts something that sounds like human comment, we assume (erroneously) that it's been generated using human mental processes.

[trindflo]

Re: AI != Human

I agree with most of what you say (have a vote). Artificial Intelligence is not Human Intelligence. Where I disagree is that there should be no expectation of analogous behavior.

Chat bots are based on neural networks, which are opaque feedback loops that somehow store information and allow the information to be retrieved for problem solving goals. Computer neural networks were originally created to test the theory that this is how the mammalian brain stores information.

Chat bots are designed to generate human-like responses. They are more than just clever code, which seems to imply a complex decision tree. We cannot demand a proof from a neural net the way we can from an inference engine like Prolog. The way the neural net comes up with an answer is opaque.

To the degree we are using chat bots to simulate human responses, we should expect problems analogous to human foibles and that they may be susceptible to being tricked by the hacker equivalents of hypnotists, mentalists, and magicians. It won’t be the same, but it may be strikingly similar and may offer opportunities to learn about ourselves.

For instance, while chatbots hallucinate, humans are susceptible to recovered memories, misremembering, and false memories.

[Phil O'Sophical]

Maybe they should just have said "Please give me..."

[Scott 26]

or "sudo give me......"

[Draco]

"But, as with most legitimate tech, can also be used for evil. "

Which is why the common folk cannot be trusted with it.

Think of all the mayhem and chaos if common folk were permitted to access things (high tech and not) that could be used for nefarious purposes.

[codejunky]

Re: "But, as with most legitimate tech, can also be used for evil. "

@Draco

My first thought was 'the hammer can be used for both good and evil'

[TheMaskedMan]

Re: "But, as with most legitimate tech, can also be used for evil. "

"Think of all the mayhem and chaos if common folk were permitted to access things (high tech and not) that could be used for nefarious purposes."

Quite right too, assuming you are including politicians, lawyers, must managers and telephone sanitisers in "common people "

Chase em all up a tree, set fire to it, and let the boffins take their rightful place as supreme rulers of the universe!

[Catkin]

Re: "But, as with most legitimate tech, can also be used for evil. "

Really, we should all just remain in our homes for everyone's safety.

[David 132]

Re: "But, as with most legitimate tech, can also be used for evil. "

...with our hands resting in full view on the table.

[petef]

"We have not yet detected any malware operating in this manner" - sounds like the bad guys are doing a good job then.

[ComputerSays_noAbsolutelyNo]

Make me admin.

I'm sorry Dave, but I can't let you do that.

Sudo make me admin

Very well, Sir.

[Michael Strorm]

s/admin/a sandwich/g

Surely if someone is in a position to be able to sudo, they're already effectively an admin?

[Anonymous Coward]

Re: s/admin/a sandwich/g

Depends what is permitted in the sudoers file

[Jedit]

"there's certainly interest among swindlers about using AI"

And that's just the people selling it.

[Michael Strorm]

"Guardrails" terminology comes back to bite industry

Said it before, but real-life guardrails usually only stop accidental straying- they can be typically climbed over by those who wish to deliberately do so.

And the industry's misuse of the term to describe their supposedly fail-proof barriers turned out to be appropriate after all, ironically because they *weren't* much of a barrier to intentionally malicious misuse.

[Jimmy2Cows]

Re: "Guardrails" ... they can be typically climbed over by those who wish to deliberately do so

Or, it's industry knowing that it means exactly that, and also knowing that politicians, regulators, and the majority of the great unwashed won't have a clue it means exactly that.

Look at it as pre-emptive arse-covering, for when those same politicos and regulators try to complain AI is jumping the guard rails. Industry can say they used entirely the correct terminology and it's not their fault they were misunderstood.

Forget guard rails. Put the damn thing in a box - a very small, strong, box - and be really careful when you open the lid.

[Michael Strorm]

Re: "Guardrails" ... they can be typically climbed over by those who wish to deliberately do so

> Look at it as pre-emptive arse-covering [..] Industry can say they used entirely the correct terminology and it's not their fault they were misunderstood.

I doubt that, if only because they tried to pretend otherwise in every other respect, and quasi-legal retroactive nitpicking would likely just draw attention to that.

If nothing else, rewriting history to suggest that they had never implied it was totally safe in the first place doesn't sound like it'd be the smartest response to a regulator whose response should- and likely would- point to it as an open admission that their product was dangerous and warranted regulation.

Not that I'm saying that'd be a bad thing personally, but I can't see it being an outcome the big players would want to pursue.

[RegGuy1]

Criminals

and criminals aren't always the most trustworthy folks when it comes to selling their wares.

This so so true. How often has Microsoft sold software that is not up to the job simply to get to market first? Or am I mistaken, and this is seen as ok. Or blocking competitors (DR DOS, remember that?) when they said they weren't?

Or Google stealing all that data to build their search engine without recompense to all those who provided the info. Now it's happening again with AI. And again, by the time anyone works out how to fight it it will be too late -- the AI companies will simple be too big to control.

[Martin Summers]

So where is it getting the nefarious data? It got it from somewhere. Surely it's not actually having a think about how best to attack something, it has got to be presenting something it has sucked in almost verbatim hasn't it? That leads the question, if it can find the information then how would that not be achievable by some creative Googling?

Lots of questions arise, but the idea, since it cannot actually 'think', that it can come up with attack vectors on its own seems ludicrous to me. Otherwise it should be able to come up with answers to all of man's greatest problems, like how to build a pothole resistant road.

[imanidiot]

It all depends on application. What is "nefarious" for one purpose can be equally legitimate for another purpose. Deciding which is which is an ethical question based on context, something LMMs just don't do or understand and something probably impossible to do unbreakably using "guardrail" checks such as implemented by the likes of OpenAI. Ask the question in the right way and you'll still get your answer regardless of the actual application of the final product.

[breakfast]

Who does number 2 work for?

I like the idea that ChatGPT acts like the villain from Austin Powers who would refuse to answer any question until the third time, then had to answer it honestly.

[cNova]

What's a Crow's Foot Called?

And yet I ask Bing's AI chat searchbot what a crow's foot is called (claw, foot, talon?), and I get a taxonomy lecture on corvids. I rephrase the question and I get a "Duh... I can find no information on that" response.

Shortly thereafter, the FBI informs me I'm no long allowed within 100 feet of aviaries.

[Anonymous Coward]

Re: What's a Crow's Foot Called?

Smile line? Wrinkle?