Gullible bots struggle to distinguish between facts and beliefs

Large language models often fail to distinguish between factual knowledge and personal belief, and are especially poor at recognizing when a belief is false. A peer-reviewed study argues that, unless LLMs can more reliably distinguish between facts and beliefs and say whether they are true or false, they will struggle to …

[Paul Herber]

"... in your toaster ... "

Oh no!

[Paul Crawford]

Are you a waffle man?

[Not Yb]

"Hello, do you want some toast this morning?"

"No"

"Why do you not want toast this morning?"

"I just don't"

"Is it something I've done that you dislike?"

"OMG shut UP you stupid toaster!"

"I'm cooking some toast for you now."

...

etc.

[Anonymous Coward]

"I love you WaffleBot!"

[RAMChYLD]

And I laughed at those toasters with the huge letters SMEG embossed on their front. In 10 years those are going to be annoying me by constantly pestering me to eat their toast...

[Nightkiller]

The problem is noy the LLMs. The problem is the moist robot prompt generators that cannot themselves distinguish between fact and opinion.

[Paul Crawford]

moist robot prompt generators

Sounds like a fun trip to Westworld, or maybe Stepford...

[Bebu sa Ware]

"maybe Stepford" ...

Who is not partial to a bit of hot crumpet ... on the side ?

[Anonymous Coward]

Human learning consumes expert moderated content designed to promote learning, critical thinking, scientific method ….

If these bots are scraping the internet it’s hardly a great surprise they are full of shit and can’t categorise truth, falsehood, opinion (biased or otherwise) or uncertainty.

I’d probably either ignore opinion completely or dial down the reliability of data if scraping my local Wetherspoons for my LLM for example.

[Anonymous Coward]

It would help if there was a difference between truth and fiction, but there isn't, as far as a model is concerned.

Remember, prompts aren't questions, and the responses aren't thoughts. We are talking about generating a statistical likely response *as though* our question was posed, based on what people and other bots have filled the internet with... and the internet does not punish clickbait.

[Anonymous Coward]

Duh. Since much of the stuff on the internet is opinion not fact, how is this surprising? How would an LLM know the difference?

[Anonymous Coward]

Through spanking?

[Anonymous Coward]

Computer..." said Zaphod again, who had been trying to think of some subtle piece of reasoning to put the computer down with, and had decided not to bother competing with it on its own ground, "if you don't open that exit hatch this moment I shall zap straight off to your major data banks and reprogram you with a very large axe, got that?

[Anonymous Coward]

… and that’s before you consider stuff that is blatant falsehoods like say InfoWars or just plain old shitposting.

[codejunky]

Like people then?

This being why having the freedom to speak your beliefs is important and allow them to be challenged openly through discussion. Hell who would dictate the official truth? See this recent exposure-

https://www.telegraph.co.uk/news/2025/11/03/bbc-report-reveals-bias-donald-trump/

Slightly longer- https://www.youtube.com/watch?v=EjPlfUt4S9U

People have various beliefs over covid, MMCC co2 theory, green energy and wars. LLM is digging through more of the same varied beliefs people encounter every day

[Gary Stewart]

Re: Like people then?

Gullible codejunky struggles to distinguish between facts and beliefs (and fiction).

[codejunky]

Re: Like people then?

@Gary Stewart

"Gullible codejunky struggles to distinguish between facts and beliefs (and fiction)."

If that is what you read from my comment I guess you are in the quality class of LLM. Or maybe thats an insult to LLM's.

[Dr Dan Holdsworth]

Re: Like people then?

OK, so you've got lots of people all over the world who believe in an invisible, unprovable sky fairy (or fairies) and who are prepared to argue vehemently about these fairies. WE the humans generally know that these beliefs can and should be filed in the round filing cabinet on the floor, but an LLM doesn't really "know" anything. It simply learns responses from the input material it has been fed, and isn't able to make value judgements on the credibility of the training material.

Similarly when fed information on medical science you're pretty likely to inadvertently get a fair old amount of bollocks about homeopathy in the mix. Say what you will about this, but I prefer medical treatments which carry on outperforming the neutral control when the sample size exceeds ten thousand or so. The poor old LLM however doesn't "know" a bloody thing, so if the training material has garbage in it, then out will come garbage from the LLM and there's then pretty much bugger all that can be done about this save screening the training material a bit better.

[codejunky]

Re: Like people then?

@Dr Dan Holdsworth

"OK, so you've got lots of people all over the world who believe in an invisible, unprovable sky fairy (or fairies) and who are prepared to argue vehemently about these fairies. WE the humans generally know that these beliefs can and should be filed in the round filing cabinet on the floor"

Do we? As a species we have a variety of gods which kids are indoctrinated into and many people are killed in the name of such beliefs. Look at ISIS or the attacks in Africa and so on. Then there are the large portion of non-violent yet still believing in their version of a sky fairy. Religion is a good example of this problem.

"Similarly when fed information on medical science you're pretty likely to inadvertently get a fair old amount of bollocks about homeopathy in the mix."

Why does homoeopathy exist? Because there are people who believe in it. How many people fall for pyramid schemes?

"so if the training material has garbage in it, then out will come garbage from the LLM and there's then pretty much bugger all that can be done about this save screening the training material a bit better."

Pretty much my point.

[Anonymous Coward]

Re: Like people then?

As a species we have a variety of gods which kids are indoctrinated into and many people are killed in the name of such beliefs.

Mammon?

[Anonymous Coward]

Re: Like people then?

"Mammon?"

Definitely. Perhaps the biggest god nowadays. Regardless what the people claim their religion is, most are actually serving Mammon.

[Not Yb]

"AI researchers realize AI's are just pattern matching (again)" Film at 11?

I get the feeling that AI researchers may not know much about what they're researching. LLMs ARE pattern matchers, ad infinitum. That's all they do. There's not any sort of "this probability path is less factual" inside.

[Anonymous Coward]

Re: "AI researchers realize AI's are just pattern matching (again)" Film at 11?

Some of them thought if they made them big enough a "miracle" would happen ...

[LionelB]

Re: "AI researchers realize AI's are just pattern matching (again)" Film at 11?

> I get the feeling that AI researchers may not know much about what they're researching.

Oh, the researchers (well, let's say most of them) know pretty damn well what they're researching. It's the marketing - the miss-selling - that's the real issue here.

If LLMs were presented honestly as: "This software does one thing only - it generates plausibly human-like responses to queries based on vast amounts of indiscriminate, uncurated data sucked out of the internet", perhaps more people would pause to think how useful that actually is to them (or to anyone).

[Tron]

Yeah, but...

...lots of people believe that creation and the existence of heaven are absolute facts.

The whole misinformation thing falls over in many communities, who will need a Christian, Muslim or MAGA LLM.

I'm surprised that Alexa hasn't been charged with blasphemy in some countries, with tech execs being hung from trees for offending the righteous locals.

[45RPM]

Re: Yeah, but...

Or that Brexit was a good idea, or that Trump and Farage are anything other than grubby little con men. Some people think the moon landings were faked, that vaccination is a bad thing, that climate change isn’t real and that immigrants are bigger drain on resources than billionaire spongers who won’t pay their taxes.

On the face of this evidence, AI has reached human level intelligence. Sadly, it seems, many humans aren’t very intelligent.

[Philo T Farnsworth]

Re: Yeah, but...

> Sadly, it seems, many humans aren’t very intelligent.

But, meanwhile, you and I. . .

[45RPM]

Re: Yeah, but...

Oh, I make no claims about my own intelligence - in general terms, at least. I have expertise in a few specific areas. But I work with some truly astonishing minds. All of them organic. None of them in favour of denigrating experts and scientists (although… given that they are experts and scientists… wait a minute!)

[Denarius]

Re: Yeah, but...

Still believe in socialism but not Santa Claus.

[45RPM]

Re: Yeah, but...

I think that nearly everyone believes in at least one aspect of socialism. They just might not know that it’s socialism.

* state pension

* public health care

* public education.

* unemployment benefit

… and many more besides. All these good things are Socialist (for the social good). Socialism is not always at odds with capitalism. It is perfectly possible to have public and private education, hospitals, pensions etc.

So yes. I believe in socialism because I’m not an effing eejit. You probably do too, although you may not call it that. Well, you do unless you are an effing eejit ofcourse.

[Anonymous Coward]

Re: Yeah, but...

"Still believe in socialism but not Santa Claus."

You don't have to *believe* into it, it actually exists and it works very well. But of course 99% of people who use that term like that, have no idea what it actually means.

[Anonymous Coward]

Re: Yeah, but...

"Sadly, it seems, many humans aren’t very intelligent."

Hard to say from inside, but I do know that there's plenty of sceptisism and sarcasm.

[ThatOne]

Somebody believes the "Intelligence" part?

> Large language models often fail to distinguish between factual knowledge and personal belief

Even humans struggle with this, so why would a pattern-matching machine be able to distinguish between fact and fiction? It would need to understand the difference between those two to start with, which is technically impossible: For the machine everything is just a pattern with a reward value: "If I say this my handler will be happy, if I say that I'll get scolded". The notions of factuality and reality are light years away...

[Doctor Syntax]

Re: Somebody believes the "Intelligence" part?

The difference between an LLM or any other sort of alleged AI, general or otherwise, and you and I is that they cannot experience reality and we have spent years doing so. We have an internal model of the outside world that we spent a few years getting up to scratch and the rest of our lives keeping that way. It gets regularly checked against what we experience. We interact with that outside world. We know - from experience - that it can hurt us sometimes, that it can please us and that it is essential to our ongoing lives. When we look at things increasingly distant from our everyday experience we are less likely to add them accurately to our internal model although it helps if we have learned critical thinking skills.

There are three words I've used there that are critical to the difference between a human - or animal - intelligence and AI: "reality", "experience" and "life". Their only internal model is a set of relationships between words. A LLM trained in French will have different words between which to establish relationships than an English one. Monoglot English and French speakers have the same understanding of, say, a motor vehicle. A program existing in a computer has never ridden in one, driven one or shelled out hard-earned cash to buy one; it has no understanding beyond linking the outer forms, the words together. It is not further forward - more likely many steps behind - than the cargo cultists in Feynman's lecture who thought that, by clearing runways and imitating air traffic controllers, they could get planes to land and unload cargoes.

[MrAptronym]

I know it is important to publish research on things that seem likely, and I am glad this paper was produced, but this is surely not coming as a surprise, right? These predictive text models were trained on the totality of the text on the internet, a place where anyone can confidently state their opinions and beliefs as fact.

I am not an expert in this field, but I am kind of concerned at how anthropomorphized all this research comes off. These researchers and so many others read more like anthropologists or psychologists than anyone studying a computer system. Just in the abstract we see phrases like "We also find that, while recent models show competence in recursive knowledge tasks, they still rely on inconsistent reasoning strategies, suggesting superficial pattern matching rather than robust epistemic understanding." Which comes off to me as a wild way of saying that these models are not actually capable of understanding... a thing we know?

[Paul Kinsler]

"how anthropomorphized all this research comes off"

Indeed. But perhaps since these models are touted as "AI", it is of some interest exactly to treat them as if they were, and see what an analysis reveals.

And possibly, with an anthro/psych framing like this, results might be more easily taken on board by those who either believe the "AI" claim, or at least who are inclined to treat their interactions with them in that way -- because that is how things *seem* to them.

[Dan 55]

You and I know that LLMs understand and reason as much as a mobile phone keyboard autocorrect, but I suppose research about gullible bots is necessary to try and persuade gullible people.

[Bebu sa Ware]

LLMs understand and reason as much as a mobile phone keyboard autocorrect

A very apt analogy, if I may say so.

Autoconfuse is very much the same technology but on the microscopic end of the LM scale.

[Doctor Syntax]

"It's going to be in your car, in your toaster, and in every streaming service,”

Not while it's me spending my money it isn't. Difficult to avoid some algorithmic crap in modern cars but at my age I've probably bought my last car so I'm unlikely to buy anything worse.

[Flocke Kroes]

A few years ago I could not find a TV without an AI upscaler. Yesterday while watching Insurgent a sparse crowd walked past a complex background. The AI upscaler 'corrected' the exposed segments of background with garish twisting artefacts. I might have been able to forgive the manufacturer if the upscaler had an off switch. I assume TVs of the future will provide such a switch as a premium tier rental option.

[ThatOne]

> I assume TVs of the future will provide such a switch as a premium tier rental option.

Unfortunately no, TV watchers in the future will consider those hallucinations as a normal part of the TV experience. You can only notice something if it stands out from the rest, so if you have seen them on every TV all your life they are just what TV looks like.

[Anonymous Coward]

Just move to vintage cars. Late 80s cars weren't too bad in safety, engine power or actually useful accessories, but no tracking or erroneous 'lane assists' . or almost continuous, loud beeping, for gazillion different reasons. (Case study: A Huyndai Ionic. Literally 5 loud *BEEPS* between opening the door and actually moving. Un-fu**ing-believable.)

*A lot* cheaper to service and the driver is resposible for everything, car doesn't do anything on its own. (Case study: "Emergency brake assist", which decides that snow on the nose is a wall and *BLAM*, emergency brake on, on highway. No way to turn it off.)

Downside of course is that you have to do everything yourself, but if you are actually driving and not just sitting in an ass-hauling thingy, it's actually fun.

[Dwarf]

The hint is in the name..

Remember its an LLM - Large Language Model, the last word being important here.

It doens't matter how much you model something, its a perception of real life and as we all know, models are never 100% accurate since they didn't understand all the detail.

Take the output of a model and apply rigour to it, only use the output after that.

[Anonymous Coward]

Robust epistemic understanding - a cornerstone of human cognition

Right smack on! Epistemology (differentiating knowledge from belief) and ontology (universe of discourse) definitely underpin cognition, including the artificial kind. The open access preprint of this Suzgun et al. Nature Machine Intelligence paper seems to indicate as much (focused on epistemology though):

"Ensuring that models can manage [the interplay between belief, knowledge, and truth] effectively will be key to improving their reliability in high-stakes areas where social and epistemic reasoning is at the forefront."

This gives credence to the notion that nutbag fruitcake stochastic parrots' outputs may be considered intelligent only in the strictest sense and restrained discipline of a looney bin certifiable interpretation of this notion, imho, as adopted by the most enlightened residents of related padded room institutions ...

Invest in straitjackets! ;)

[Anonymous Coward]

LLMs should be great at replacing Gartner then

Finally, a worthy purpose for AI!

[Flocke Kroes]

facts and beliefs

I thought LLMs could not distinguish facts from beliefs by design. This is a high value feature - to scammers.

[Bebu sa Ware]

I imagine professional philosophers ...

shudder in the face of the ludicrous claims of the AI cabal and good many of the counterclaims of the justifiably sceptical critics.

After a few thousand years their discipline is still working on questions of knowledge, belief, existence etc etc (hint: 42 isn't the answer.)

Just a short question like "what precisely is a fact ?" - and how do you distinguish a fact from a non-fact - clearly don't have single, simple knockdown answers. Unless you are a fundamentalist nutjob, facts are clearly not eternal absolute verities.

If in trying to define a fact you grasp at the straw of "truth" you soon discover not only are you out of your depth, you are now actually drowning.

To my mind the question of what constitutes knowledge is even more vexed. Clearly not just a collection of facts - an encyclopedia is hardly knowledgeable.

Asserting AI/LLM is capable of "knowing" or "believing", or even possessing a decision procedure that can discriminate between knowledge and belief — is completely delusional or from a mendacious vendor of oleum serpentis.†

_{† Ne inducas nos in temptationem sed libera nos a malo.}

[Anonymous Coward]

Re: I imagine professional philosophers ...

"what precisely is a fact ?"

The things you see and hear on Fox.

/s

[Flocke Kroes]

Re: I imagine professional philosophers ...

One big difference between people and AI is that people get experience from real life but AI gets experience from the loudest people on the internet. Ignorance outnumbers knowledge so it is amazing thet LLMs get anything right. On the other hand people can usually work out that picking fresh chips out of boiling fat by hand is not a good idea with less than two attempts.

[Anonymous Coward]

Re: I imagine professional philosophers ...

"facts are clearly not eternal absolute verities."

Eternal, no. But absolute (within measurement error), easily. Take a multimeter and measure a voltage. Whatever you get is a fact.

A dead person is an eternal fact: It won't change, no matter some people believe.

Faith of course is the opposite of a fact and therefore they'll never understand facts.

[Anonymous Coward]

"Large language models often fail to distinguish between factual knowledge and personal belief, and are especially poor at recognizing when a belief is false."

There's a major blunder here: An LLM literally does not have a concept of "truth" or "truthness".

When you don't even have a variable (or any equivalent) for it, it obviously can't have any value either.

LLM treats everything as a statistical stream of words and anything else is out of concept.