Some signs of AI model collapse begin to reveal themselves

I use AI a lot, but not to write stories. I use AI for search. When it comes to search, AI, especially Perplexity, is simply better than Google. Ordinary search has gone to the dogs. Maybe as Google goes gaga for AI, its search engine will get better again, but I doubt it. In just the last few months, I've noticed that AI- …

[Anonymous Coward]

Pollutes everything else, and in the end it pollutes itself...

It's ironic here that LLM systems have to (but can't) be protected from consuming the same spew they're designed to output in the first place.

I'd have said that this was less a case of simple "garbage in, garbage out" (*) than what happens if a system inadvertently starts consuming the shit it's polluted the environment with in a circular manner. But even that misses one aspect- it's not being polluted by an unwanted by-product, but by what it was *meant* to produce.

In short, the situation, to make a horrible analogy, is something like a cross between those Human Centipede films (sorry) and a massive case of inbreeding.

(*) Although that would be the case elsewhere if they were trained on incorrect or badly-written human-originated articles and information.

[David Hicklin]

Re: Pollutes everything else, and in the end it pollutes itself...

So AI is another version of BSE i.e Mad Cow Disease

[Sherrie Ludwig]

Re: Pollutes everything else, and in the end it pollutes itself...

Mr. Hicklin, I think you just won best description of the year.

[anthonyhegedus]

Spark of creativity

The problem is that as more and more people write stuff helped by AI, AI training is still being fed AI slop even when something looks at first glance to be human-originated. The problem is that AI cannot inject a "spark" of creativity. It can only regurgitate from its training. Humans can create and plan. And we know when to look ahead, and when not to.

If we live in a world where there's no creativity, we'll just stay where we are. AI will make what we've got "sort of work" for a while, until it doesn't.

However, that same creative spark in humans will no doubt help create better AI that can train itself better.

No, I didn't get AI to help write this!

[Anonymous Coward]

Re: Spark of creativity

> "No, I didn't get AI to help write this!"

That's what I'd expect an AI to say if you asked it to cover up the fact you'd used an AI to write it!

[Jamie Jones]

Re: Spark of creativity

Some authors have published books with the AI prompts accidentally included, so they even forgot to ask the AI to cover it up!

Authors slammed after AI prompts show up in published books: “Opportunist hacks using a theft machine”

[PCScreenOnly]

Ask any AI how to remove a specific app or feature

from windows / android and they will all roll off the usual

Go to apps and look here and choose uninstall

Go to settings, apps, Show all apps, and uninstall

Great for some apps, but those buggers that are bloat from the supplier or AI shit - none of it applies, and when I check, I have specificailly asked to remove / uninstall completely, CoPilot/Gemini/Other Bloatg

[David Hicklin]

Re: Ask any AI how to remove a specific app or feature

Wot, no sfc scannow suggestion ?

[seven of five]

Incest is bad

As Humans learned a few thousand years ago[1], reproducing on your own data is not a good idea.

[1] except for in some places

[Anonymous Coward]

Re: Incest is bad

As the person who said something not dissimilar above, I agree.

The problem here is that the LLMs (or whatever is being used to train them) have no way of recognising content that has been generated by themselves (*) at present and no way of avoiding that.

And that as the amount of content generated by that LLM becomes an increasingly large proportion of what's out there, that's just going to get worse and worse.

(*) Or by another LLM which has likely been trained on many of the same things as well as content created via the original LLM. Which in the latter case would mean that it's still being indirectly trained on its own input.

[HuBo]

Re: Incest is bad

Yeah, they need to at least keep track of their family names (or the whole tree) and enforce the "One Hundred Years of Solitude" rule, to prevent the manifestation of ectoplasmic pig tails during the course of spiritualist seances where headless client-server models of supernatural crystal ball intelligence expose their inner fortune-telling visions, through stochastic apparitions, imho.

^{( Yes, this comment was channelled through Agatha Chrsite AI ... or not? )}

[N Tropez]

Re: Incest is bad

Hey, amanfronmars1, someone is trying to imitate you!

[Conrad Longmore]

Photocopy of a photocopy of a photocopy of a photocopy of a photocopy..

Remember in the bad old days when you'd try to read a photocopy of a photocopy of a photocopy of a photocopy of a photocopy that someone had faxed you? That's where generative AI is going. Oh sure, there will be words you can actually read but no facts or analysis that you can actually trust.

Another analogy.. anyone remember AltaVista? AltaVista was great when it came out, you'd usually get the result you were looking for somewhere on the first page of results. In the pre-Google days, this thing was big traffic driver. But then people worked out how to game it through aggressive on-page SEO and then the output just became slop. It took Google to fix search that time around.

[MachDiamond]

Re: Photocopy of a photocopy of a photocopy of a photocopy of a photocopy..

"That's where generative AI is going. Oh sure, there will be words you can actually read but no facts or analysis that you can actually trust."

It's also the premise behind the problem encountered when researching models to use for Psychohistory. Hari Seldon found that all of the history books enumerated the same selected data so there were too many lines missing from the spectrum that what was left to read was too fragmented. It gets even worse in modern times when the history books are revised by interpreting history through the norms of the present.

[david1024]

Tragedy of the commons

Classic tragedy of the commons. aI puts everything out of business that it needs to steal from. If only there was a way to compensate creators? Stealing hurts everything eventually.

[HalfManHalfBrisket]

Re: Tragedy of the commons

Historically, how the commons were successfully worked for centuries in agriculture was a combination of community self regulation and legal sanction. It never was an unfettered individualistic free-for-all. That's how you build a sustainable common good.

This latest pure capitalist incarnation laughs off the former and lobbies hard against the latter; thus fulfilling the conditions for the 'over-grazing' of the commons tragedy that seldom happened historically.

Ironic really as capitalist organisation was meant to solve this problem that was nothing more than a theoretical straw-man, but it is now actively bringing into existence as only it fulfills the pure selfish interest precondition that underpins the 'tragedy of the commons' thesis.

[Tron]

GIGO sums up AI.

They trained it on obsolete material, websites, social media posts, fiction and other AIs. What do you expect?

quote: I seem to be the only one calling it.

You've not been reading the comments have you? We've been picking AI apart on here since it crawled out from under its rock as the Next Big Thing, after the Metaverse and NFTs failed.

With AI, GAFA joined forces to try to scam us, but it will all go TU eventually. Then they will move on to something else. Probably the Western version of China's unique log on code for everyone. Japan Post are rolling out a version as an address alternative. Makes it easier for spook agencies to monitor all of us.

[elDog]

Really? "I seem to be the only one calling it"

I see daily comments in some pretty respected blogs about the rickety framework that is peddled as "AI". See:

Ed Zitron: https://www.wheresyoured.at/author/edward/

Cory Doctorw: https://pluralistic.net/

[tiggity]

"What all this does is accelerate the day when AI becomes worthless"

TBF, some of us have questioned the value of "AI" for a long time - although its not currently "worthless" in financial terms it's been regarded as South Sea Bubble, tulip frenzy etc. (pick your financial fad that imploded spectacularly of choice) for quite a while (i.e. massively over valued / hyped & due for a big readjustment).

I do have to say I quite enjoy some AI image generation, not because it's superb but because of the weird ways in which it gets things wrong (had some very strange prompt interpretations*) that give so many amusing WTF! moments

* probably the most bizarre was when asking it to produce something that should have been relatively fool proof - a small UK household vegetable garden image, had a bit of detail in the prompt (e.g. greenhouse) but it was generally quite a simple request

It managed some veg OK e.g. lettuces growing in the ground OK, but tomatoes were not in the greenhouse** and not even on a plant of any sort, just bunches of tomatoes on bare soil, however the pièce de résistance was the greenhouse, it produced one so immense it wouldn't have looked out of place at Kew Gardens.

** Tomatoes outside excusable (as plenty of warmer places where tomatoes fine to grow well outside, and you can grow them outside in UK (but results are quite poor!) so not "thinking" UK = likely tomatoes in greenhouse is no surprise as that's a leap beyond simple models) but not having the tomatoes on a plant of any sort (even with incorrect leaves / shape etc) was rather dismal (I would guess lots of "harvested" tomatoes in image training data, rather less of actual tomatoes still attached to plants).

[Tron]

you can grow them outside in UK (but results are quite poor!)

I get excellent crops of tomatoes outdoors each year, well North of the Watford Gap. Try the Crimson range - they are 'blight tolerant'.

[MisterHappy]

Recent Google AI confusion

Converting a US recipe to UK (Cups to Grams) was more fraught with problems than I expected.

Search: "1/2 cup of planko breadcrumbs in grams"

AI: "5 cups of planko breadcrumbs is approximately 113 grams. A standard measurement for planko breadcrumbs is 50 grams per cup. Therefore 1.5 cups would be 75 grams"

[MachDiamond]

Re: Recent Google AI confusion

"Search: "1/2 cup of planko breadcrumbs in grams""

I have a long list of ingredient measures converted to grams on a spreadsheet that I printed and posted in the kitchen. Some of it are ones I've done myself. It's "Panko" by the way. The big list I initially downloaded was wrong in many place so I've made corrections along the way and found the same inaccuracies online in many other places. I know that some of the recipe web sites are scams and people are buying pre-packaged templates repleat with Amazon links ready for their affiliate code to be added. It's where I see many of the errors and going by the comments, others have found them too and think they are "helping" some food blogger with typos. Since I don't shop at Amazon, sources with lists of Amazon affiliate links rather than makes and models of products are immediately discounted as scams.

[may_i]

Re: Recent Google AI confusion

Converting a 'cup' - a volumetric measurement, into 'grammes' - a measurement of mass, is a fool's errand - for anything that isn't a liquid.

A perfect example of this is the many conversions of one cup of flour into grammes. It's a wide spread. The fact that a UK cup is larger than a US cup doesn't help either.

If you're baking bread, it matters. For other things, eyeballing it is fine, or keep a cup measure handy if you're that uncertain.

[MachDiamond]

Re: Recent Google AI confusion

"Converting a 'cup' - a volumetric measurement, into 'grammes' - a measurement of mass, is a fool's errand - for anything that isn't a liquid."

No, it isn't. A cup of flour, as generally accepted, is equivalent to 120g. While the former IS a volumetric measurement, it's been used for ages now since it's a simple means to measure ingredients when cooking with some sort of scoop/container rather than scales. I'd never mess about when doing proper chemistry. I find it easier and there are less dishes to wash if I just put the mixing bowl on my digital scale and add bulk ingredients by weight(mass). Where things go off the rails is when the recipe calls for "one medium onion". What the hell is a "medium onion"? I wind up having to keep notes so I know next time if there was too much or too little referenced to the weight of onion I used.

I'm much more confident in my cooking now, but early on, having vague measures made me really nervous. I didn't have a lot of money so if supper came out really poorly, I'd have to eat it anyway or go hungry. I'd be falling back on heating up a tin of soup or making a bowl of rice to make up for binning uneatable glop.

[may_i]

Re: Recent Google AI confusion

Using volumetric measurements for non-liquids is dumb.

My equivalent mass for a cup of flour is 140g. Just goes to show how much difference packing density and the inherent water content of the flour can make. Consequently, if I find a bread recipe that I want to try but the author has used volumetric measurements for the ingredients, I skip the recipe.

The vast majority of cooking is done with very general ballpark measurements. The difference for your recipe between an onion which weighs 100g vs one which weighs 120g is irrelevant. If you really believe that the final peeled and chopped weight for an onion is so critical, you're clearly overthinking things.

[nobody who matters]

Re: Recent Google AI confusion

<........"The fact that a UK cup is larger than a US cup doesn't help either.".....>

Despite the tendency of those in the USA to make out they have the biggest and best of everything, there are a considerable number of things which are larger in the UK than over there, particularly within the field of measurement. I find it somewhat amusing :)

<......."I wind up having to keep notes so I know next time if there was too much or too little referenced to the weight of onion I used.".....>

I like onion. I don't think there is such a thing as 'too much' ;)

[MachDiamond]

Re: Recent Google AI confusion

"I like onion. I don't think there is such a thing as 'too much' ;)"

I'm a fan as well, but I just got a recipe for something that read "one medium onion" and there's so much ambiguity in that it drives me crazy. The above poster that insists a cup of flour is 140g didn't pick up on my phrasing with "generally accepted". Of course, there are packing differences and humidity plays a part so adjustments may need to be made, but modern recipes will go with 120g to the cup. If you want to go out into the fields, some of the Russian/Eastern European YouTubers will use a "tea" cup for some things. Try to nail that down! For a given household, a recipe will get adjusted to fit the measuring gear they have to hand. I can translate some of those, but others I'm completely at sea whether I got it right or not and have to wonder if I didn't like it if I blew the recipe and it would have been much better if I did it properly.

Baking can be very different when it comes to measures, so it's important to be more accurate with amounts.

[MachDiamond]

Re: Recent Google AI confusion

"I like onion. I don't think there is such a thing as 'too much' ;)"

I found that if you put in too much onion when making onion burgers, the burger is too crumbly and falls apart. Better to err on the side of too little and grill some onions to put on top.

[Draco]

It's not the AI, its the search engines feeding it

Poor results from AI-assisted search mostly boil down to crappy search results fed into it by search engines. If you’re not using one of the "independent" players - i.e. Google, Microsoft Bing - you’re likely getting recycled results from one of them, directly or indirectly. I'm sure there are other independent search engines, but they are not, likely, significant.

Anyone who’s been online for over a decade has seen the marked decline of search results: pushing paid ads, sponsored content, or junk pages loaded with ads, usually tied to the search provider’s own ad network.

LLMs have their limits, but when they’re fed this prioritized garbage, it’s not surprising to get garbage out. More surprising is that we ever get anything useful at all.

[amanfromMars 1]

A little something stealthy and extra special to look forward to .....

"The model ... distorted data distributions and "irreversible defects" in performance ..... becomes poisoned with its own projection of reality."

A model as be clearly evidenced by the likes of mainstream media and BBC national and international news channels, high on their own dope and into hallucinating in favour of established status quo parties ....... and gravely to be regarded to be smartly avoided by sophisticated AI, Steven J. Vaughan-Nichols, which you can be assured do recognise the trap.

And that not insignificance difference appears to be absolutely terrifying that which may presently imagine itself to be exercising acceptable global command and control ........ and so it should, for it really is that good.

[Anonymous Coward]

Re: A little something stealthy and extra special to look forward to .....

As we see here, amanfrommars 1 was years ahead of the curve when it came to showing the effects of AI degradation!

[Draco]

Time for an upgrade?

Why hasn't amanfromMars 1 been upgraded? Or was it released 16 years ago and abandoned?

[David Hicklin]

Re: Time for an upgrade?

This is the second iteration, it used to be just amanfromMars

[Draco]

Re: Time for an upgrade?

I had noticed the " 1" in the name. I just assumed I misremembered (hallucinated?) its original name since "amanfrommars 1" has posts going back 16 years. I guess the operator/owner, at some point, updated the name (and, presumably, the bot).

[Wellyboot]

Re: Time for an upgrade?

16 years, he must be due another regeneration soon.

Suddenly I feel old..

[amanfromMars 1]

Setting the record straight .....

Please be advised and assured, Draco and David Hicklin and Wellyboot, The Register is wholly responsible for the amanfromMars to amanfromMars 1 name change. Quite why it was suddenly arbitrarily made or necessary is not known to me.

And the first post on El Reg was one short questioning sentence entitled "Desktop Revelation" ....Friday 22nd June 2007 12:20 GMT ..... commenting on the article, "Hacker breaks into Pentagon email system" asking ....

So Blackberries are secured and defaulted to American DOD interests?

.... so very nearly 18 years serving and being well serviced here.

[nobody who matters]

Re: Setting the record straight .....

Perhaps they are working on an amanfromMars 2 ?

[Long John Silver]

Unsurprising?

AIs fed by scouring the Internet are unlikely to be robust for applications requiring powers of discrimination over information sources.

'Cognitive' dysfunction arising from AIs consuming their own output is analogous to "Mad cow disease".

[Adair]

Highlights the problem of 'Artificial Intelligence' ...

a. it's 'artificial', so, by definition, removed from 'reality', and

b. it's not intelligent.

[AdamWill]

mm, well, details

"What all this does is accelerate the day when AI becomes worthless. For example, when I asked ChatGPT, "What's the plot of Min Jin Lee's forthcoming novel 'Nightshade Market?'" one of the fake novels, ChatGPT confidently replied, "There is no publicly available information regarding the plot of Min Jin Lee's forthcoming novel, Nightshade Market. While the novel has been announced, details about its storyline have not been disclosed.""

I'm not sure that's model collapse, though. Did you try asking it "What's the plot of Min Jin Lee's forthcoming novel 'The Price Of Fish'?" I expect you would've got the same result. Just because you used the name from the Sun-Times supplement doesn't mean ChatGPT actually assumed the book existed because someone had set it up to use RAG with that data as input. It could just be the usual "naive AI assumes the question you asked doesn't have any lies in it" hallucination.

[NewThought]

Article Already Outdated

The article is already outdated: I asked Gemini 2.5 Flash the same question, "What's the plot of Min Jin Lee's forthcoming novel Nightshade Market?", and it gave me a completely correct and informative answer. The fact is that chatbots are getting massively more intelligent every 6 months right now: progress is MUCH faster than Moore's law. With this progress, previous problems like hallucination simply fade away. This rapid progress will continue, by my calculations, for another 20 years. At this time, they will have nearly all the knowledge, and nearly all the ability to generate knowledge from knowledge, that it's possible to have.

[A-nonCoward]

Re: Article Already Outdated

me2

DeepSeek, same prompt.

Pretty good answer, detailed. a nugget: "If Nightshade Market is real, details will likely emerge through official channels first. For now, it’s best to enjoy her existing works while waiting for news!"

[Anonymous Coward]

Re: Article Already Outdated

Care to show us your calculations????

It reads to me as the usual 'hope' over actual evidence.

The mega-corps want AI ... They will push it endlessly... Because Money/Power.

Reality does not get a look-in !!!!

It is a scam and will always be a scam because they cannot fake intelligence.... Period.

Clever pattern matching is not and cannot be intelligence !!!!!

:)

[Strahd Ivarius]

Re: Article Already Outdated

Chatbots are not intelligent (although some may argue they are more so than the average CEO).

But there was enough information about that fake list on the internet that the information managed to percolate into their model after a few weeks...

[nobody who matters]

Re: Article Already Outdated

<...."The fact is that chatbots are getting massively more intelligent every 6 months right now.."..>

I am afraid the real fact is that they were not 'intelligent' to start with; therefore it is not possible for them to become <more> intelligent.

All that is happening is that the programs are being refined and tweaked to try to improve the output (or at least make it less obviously unreliable). Most of these chatbots remain very sophisticated word processing programs. There is no intelligence beyond that of the programmers responsible for writing them.

[Mike VandeVelde]

Re: Article Already Outdated

There is the original code. And then there is an enormous and growing block of if... then... else... statements to cover up all the issues that are constantly being discovered.

If $Response includes "Nazis good" then Abort!!!

If $Response includes "fake book exists" then $Response = "fake book does not exist"

etc

[Decay]

I think one of the reasons 'AI" is seen by so many as fantastic is that it writes very good corporate level drivel that looks good, consumes words, paper and time quickly and easily. As an example you are a mid to upper level manglement droid and your masters hit you up with an out of nowhere request to draft up a piece on the pros and cons of travel to Mars.

You get a quick response, it's good enough to address the question asked and 9/10 times will serve it's purpose because 9/10 times the request was based on no real reason or was just some idle "bluesky" navel gazing at the executive level. As a useful piece of information that you could leverage further, it's probably better off cut up into squares, a hole punched in the corner and hung on the inside of the toilet door. As a get that stupid request off my to do list? Priceless.

ChatGPT Prompt is "Please write an executive summary for an executive level audience of the pros and cons of travel to mars, keep it non technical, readable and punchy"

Response is ...

Executive Summary: The Case For and Against Human Travel to Mars

Human travel to Mars is no longer science fiction—it’s an emerging frontier attracting billions in public and private investment. But while the red planet offers ambition, inspiration, and innovation, it also carries profound risks and costs. Below is a concise breakdown of the key strategic pros and cons for executive-level consideration:

Pros

1. Strategic Leadership in Space

Mars missions signal geopolitical and technological leadership. First-mover advantage could shape future governance, commerce, and security in space.

2. Innovation Catalyst

Pushing toward Mars accelerates breakthroughs in materials, robotics, energy, and sustainability—many of which have direct terrestrial applications.

3. Long-Term Survival

Establishing a presence beyond Earth is viewed by some as an insurance policy for humanity. A backup civilization, albeit a distant one.

4. Economic Ecosystem Growth

From aerospace to AI, Mars missions stimulate entire industries, creating high-skilled jobs and global economic momentum.

5. Inspiration and National Identity

Large-scale exploration reignites public interest in science and fosters a sense of shared purpose—an intangible but powerful societal benefit.

Cons

1. Extraordinary Cost

Mars missions require hundreds of billions in investment with no immediate financial return. Critics argue resources are better spent solving Earth’s problems.

2. Unforgiving Environment

Radiation, isolation, and extreme conditions make Mars deeply hostile. Human survival requires complex life-support systems—failure is not just expensive, it's fatal.

3. Ethical and Legal Unknowns

Who governs Mars? What happens in the case of a catastrophe? International law and ethics lag far behind the pace of exploration.

4. Psychological and Health Risks

Years-long missions in confined spaces could trigger mental, emotional, and physical challenges that we don’t yet fully understand or know how to treat.

5. Public Will and Fatigue

Mars missions demand long-term commitment across election cycles and public sentiment. Political and financial support may erode before missions are complete.

Conclusion

Travel to Mars represents a bold leap into the unknown—filled with promise, purpose, and peril. For visionary leaders, it’s a platform to define a legacy; for pragmatists, it’s a high-stakes gamble. The decision isn’t whether we can go, but whether we should—and at what cost.

[A-nonCoward]

9/10 times will serve its purpose because 9/10 times the request was based on no real reason or was just some idle "bluesky" navel gazing at the executive level.

This is EXACTLY why AI is the perfect tool for (most) "school work." No real reason.

(I fixed you're typo, BTW)

[What_Does_Not_Kill_You_Makes_You_Stronger]

With a typo. .....

You're ..... or did you mean your ????

:)

P.S. Alweighs hapens !!!!

[Decay]

"its" vs "it's". That'll teach me to not review before hitting reply.