AI models face collapse if they overdose on their own output Researchers have found that the buildup of AI-generated content on the web is set to "collapse" machine learning models unless the industry can mitigate the risks. The University of Oxford team found that using AI-generated datasets to train future models may generate gibberish, a concept known as model collapse. In one …
> The entire _problem domain_ is gibberish pseudo-mathematical bullshit.
Is it? I think that what LLMs do -- what they are designed to do -- is pretty clear: in response to a query, they generate plausibly human-like textual responses which (to some degree) reflect associations in the human-generated training set pertinent to the query. And they are capable of doing that rather well - at least if the training set is not polluted with LLM-generated text. The basis on which LLMs do what they do is hardly "pseudo-mathematical"; it's a particular style of machine-learning model (the Transformer architecture), which does what it says on the box.
Whether you think that what LLMs (are designed to) do is worth doing, or is a gibberish/bullshit problem domain is an entirely different question. Personally, I'm a bit jury's out on that one.
> Charlatans are attempting to "create" intelligence without even remotely knowing what it is.
Pfft. Nobody knows what intelligence is*. The people developing AI models are not the charlatans - they (of necessity) understand how their algorithms work and those I've met are, in my experience, pretty clear-eyed about what they are and are not capable of. The charlatans are the marketing crowd (over-)selling the tech under false pretences.
*If you think we should rather wait around until someone comes up with a principled and consensually acceptable definition of what intelligence "is" before attempting to develop artificial versions, then prepare yourself for a lengthy wait - it's certainly kept philosophers in business for a good few millennia and counting. Me, I'm more than happy for people to crack on with it, make mistakes, learn incrementally, and do some interesting, and potentially even useful stuff in the process. That's actually how science tends to work in practice.
I am talking here about machine learning, which is merely a way of categorizing data. In that sense it does what it says on the tin.
Problem 1 - assuming that language is even suited to such categorization. It (neural net) can be made to seem like it is working, but what is it really doing with those words? More serious is:
Problem 2 - this shit is being marketed as "intelligence" (well maybe around the next few corners or something).
Problem 3 - people believe in it. That is a pretty serious problem.
As for the charlatans bit, I was talking about company bosses, who have the ear of politicians, media and public servants (the data scientists are happy to take the company's money and keep working on it, even if they are clear and honest about the limitations; they and the marketers just do as they are told). Their bosses don't care and will recoup the wage expense from a gullible public.
> I am talking here about machine learning, which is merely a way of categorizing data. In that sense it does what it says on the tin. [my emphasis]
There's no "mere" about it; data categorisation is hard, and not even a clearly-defined problem (categorise on what basis?) It's also what animals, including humans, have to do, on the fly, all the time, just to stay alive (now there's a basis for you!)
> Problem 1
Again, it depends on what basis you want that categorisation to be. In the case of LLMs, that would be something along the lines of: categorise the input data on the basis of plausibly human-like responses to a given query, which reflect the semantic associations in the input data which are pertinent to that query. That's clearly highly non-trivial, and I actually find LLMs rather remarkable in their capacity to get anywhere near achieving that. So I'm not sure exactly what you find "problematic" here (beyond a valid questioning of why anyone would want to do this in the first place).
> Problem 2
Sure. But since no-one really knows what "intelligence" is supposed to mean (beyond human-like general intelligence, which LLMs clearly do not have). I'm personally inclined to be more laissez-faire about this. I think firstly that intelligence is highly multi-faceted, and secondly that there will come a point (and no, I won't put a time scale on it!) at which we will consensually acknowledge machine intelligence - but I suspect it may not necessarily be very human-like.
> Problem 3
Foolish/uneducated/gullible people believe in it. And yes, that is a problem.
> As for the charlatans bit, ...
Sure, what I said. I'm inclined to grant the researchers/developers a bit more leeway. Everyone needs to earn a crust, but many will be in it because they're motivated, intrigued, and simply doing what they're good at. (I happen to be a research scientist myself - not in AI - and believe me, there are more lucrative ways to earn that crust, but few with a comparable level of job satisfaction.)
In the case of LLMs, that would be something along the lines of: categorise the input data on the basis of plausibly human-like responses to a given query, which reflect the semantic associations in the input data which are pertinent to that query. That's clearly highly non-trivial, and I actually find LLMs rather remarkable in their capacity to get anywhere near achieving that.
It would be remarkable, indeed, if LLMs did manage that. I don't think they're even capable of several of those things; defining a "plausibly human-like response", defining "semantic associations", or even "pertinence". My understanding of what they do, is that they give a response that has been statistically generated from their input data set; a bit like "guess the most likely next word in this sentence". Those statistical models may or may not involve stochastic categorisation, but any "categories" would either need to be human-defined, or would be statistical artefacts, with no inherent meaning. We've all seen how "AI" can be encouraged to "hallucinate" wildly given the right prompts, producing the sort of word salad that would earn a human author a psych evaluation.
> It would be remarkable, indeed, if LLMs did manage that.
And yet, they clearly do, most of the time. Glitches/hallucinations aside, they generate plausibly human-like responses pertinent to the input query. That is an observation, not a contention.
And yes, they are statistical models, but so what? They are statistical models which can be very, very good indeed at "guessing the next word in the sentence". And note that they achieve this without human demarcation of "categories" or "meaning" (whatever that means). Sure, you may consider my describing what they are doing in terms of "pertinence" and "semantic associations" as somewhat anthropomorphic - take with a pinch of salt if you like - but to my mind it is a reasonable high-level description of how they function.
Of course they're hardly perfect at what they do -- and I certainly wouldn't describe what they do as human-like intelligence -- but as a technological feat I still find it rather impressive.
Not sure why the downvotes. I am a working research scientist (neuroscience-adjacent) and that really is how science works on the ground. If you knew where you were going, and you knew how to get there, you'd just go there. But you frequently don't know exactly where you're going, and even if you do, you almost certainly don't know exactly how to get there. In practice, most of your time is spent thrashing around in confusion, going down blind alleys and chasing red herrings. That may not sound terribly appealing, but the potential pay-off in terms of clarity and sense of achievement is huge.
Eh? But that's not what scientists do.
Sure, science may deploy statistical models extremely effectively (e.g., statistical mechanics), and pretty much all science uses statistical methods to test hypotheses against empirical evidence - but "blindly trust"? Scientists do not blindly trust anything.
He, he. Sure, I brought up science because I am a scientist.
Some ML is science (or maths, if you prefer) - there is a sound body of theoretical work on learning, etc., going back a good few decades. Some ML is engineering - nothing wrong with that, I hope we can agree. And some ML is bad engineering, a.k.a. alchemy.
LLMs are represented in all of the above - if we have any disagreement, it's probably about the distribution of representation across those categories.
As for my lassez-faire attitude to AI progression - well, that is because (a) I believe that messing about and getting stuff wrong is an integral and essential aspect of science (and indeed engineering), and (b) I prefer not to second-guess technological progress. One is invariably wrong. The obverse side to the marketing bollocks surrounding current "AI" is the trend towards sneering (and frequently misinformed) dismissal of the entire enterprise. To me, that smacks of misplaced cynicism, or juvenile virtue-signalling; I don't find it particularly big or clever - just slightly depressing.
Well you have to go back to the last century, but I did spend several years as a research engineer for Thorn EMI and BT. I have an "engineering" view of science, that it should have a workable mathematical model of what it is describing.
As far as I can tell, neural nets were intended as a model for research into (real!) artificial intelligence. Also as far as I can tell, that model is inherently _not_ capable of AI and so has been turned to other uses (LLMs). That field seems anything but rigorous compared to the scientific models that I have encountered in private study (special and general relativity). I can't get the idea of training a spam filter out of my head ;)
That is where I am coming from. Cynical? Yes. Misplaced? I do not think so!
My view on science is pretty much the same - although I'd say scientific models should be useful rather than "workable" - in the sense of the adage: All models are wrong, but some are useful.
> As far as I can tell, neural nets were intended as a model for research into (real!) artificial intelligence.
Artificial neural networks were originally devised as computational models inspired by biological neural systems. The first implementation of an ANN was Rosenblatt's Perceptron, although the Perceptron model was conceived earlier by McCulloch and Pitts in the 40s. Their motivation was, I believe, to gain insights into the functioning of biological neural networks. ANN research has since been split between research into biological function and machine learning (not necessarily "intelligence" per se) - although there are encouraging signs that that split is closing somewhat.
> Also as far as I can tell, that model is inherently _not_ capable of AI ...
We actually do not know that. (That's a very strong statement, and I'm afraid "As far as I can tell" does not really do it for me! Not very, ahem, rigorous, shall we say.)
> ...and so has been turned to other uses (LLMs).
Before that, a plethora of classifiers, robotics controllers (I used to hang out with some "connectionist" robotocists), ... you name it; then more recently (deep) convolutional networks, which revolutionised image recognition. LLMs are just the latest newbie in the game. And no, none of these applications are "rigorous" - but some are useful (see above). Unlike in science*, in engineering rigour is nice when you can get it, but unless mission-critical, engineering does not necessarily need to be rigorous to be useful - first and foremost, it needs to work.
> That is where I am coming from. Cynical? Yes. Misplaced? I do not think so!
Well, time will tell. It's rather easy to forget how far we have actually come in the last couple of decades, e.g., in the fields of image and speech recognition/generation. It's interesting to trace those developments back - and you'll find that rigour was never a primary consideration (this was engineering, after all). Sometimes it was simply a matter of technology and scale; deep convolutional networks, if I'm not mistaken, were originally conceived in the 1980s - but could not effectively be implemented until computing power caught up to their potential two decades later.
I also studied General Relativity, as it happens - my background is originally in pure maths; my field was Riemannian geometry. I now work in neuroscience - or, more accurately, with neuroscientists. I mostly develop mathematical methods for analysis of neurophysiological data. This means I get to rub shoulders with folk on both sides of the real/artificial neural network business.
When you don't know where you're going: Agreed; penicillin, the moons of Jupiter, the craters on the Moon, the supernova of 1604, the photoelectric effect (electrons emitted by light striking certain metals do *not* become more energetic when the brightness of the light is increased), and many other examples.
Or as Isaac Asimov put it, "The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That's funny...'."
> *If you think we should rather wait around until someone comes up with a principled and consensually
> acceptable definition of what intelligence "is" before attempting to develop artificial versions...
Also to your point, building and testing things that *might* be intelligent is an *experimental* way of doing what philosophers have tried to do without experiment for hundreds of years (or thousands, going back to Socrates).
It's quite sad the level of fundamental ignorance on a supposedly tech-savvy website.
People who haven't the slightest idea how it actually works are parroting FB-quality comments and getting praised for it as if they've said something valuable or noteworthy.
To say that ML/LLM/AI is rubbish and created by charlatans really shows you don't have the slightest clue. This is what the smartest computer scientists are working on.
A recent El Reg AI article had a link to this FT explanation: Generative AI exists because of the transformer.
That is such a good read, and so I thought I'd paste it again, in case someone missed it. And once you've read it you'll understand that AI is definitely NOT intelligent, but is, yet again, just another search problem.
Note quite, I think. The problem identified in the study is not actually the algorithm itself - it's the iteration on data polluted by its own output. If trained on "clean" (i.e., LLM-free) data, LLMs do (to a greater or lesser degree) what they say on the box - in response to a query, they generate plausibly human-like text which (to a greater or lesser degree) reflects associations in the human-generated training set pertinent to the query. The nonsense spiral only kicks in when trained iteratively on recursively polluted data.
Whether you think what LLMs do is worth doing in the first place is a different issue; as is whether you think it merits the "I" in "AI" (Reg readers, at least, are pretty clear on that last one...).
That's not quite what I meant by "lossy"... I should have clarified that. I meant simply that you cannot, even in principle, recreate the entire training data set from a trained LLM; the training process itself is lossy. Of course the output has to be limited - the object of the game is to respond to a query with a human-like response; humans don't chug out the entire body of their sources of information in response to a simple query. Well, not the sane/socially-presentable ones, anyway.
Anyway, lossy is fine, and often extremely useful (e.g., JPEG). On the other hand, you probably don't want to apply it recursively*.
Correct, they are fixed models at runtime, so no memory whatsoever.
The chatbots are fed the entire chat for every response - including the hidden preamble that tries to keep them on the rails. This is why they all go off the rails after a few replies, as the input becomes too large for the model to handle.
It's also why many researchers think they're a dead end.
I guess the problem is simply that when a LLM is training on the internet, it can't easily tell which content to ignore - and specifically which is AI.
It would be like me writing an article on AI purely by reading other articles on AI and doing no new research/work, and then you writing an article based on my article, and so on.
It doesn't sound an insurmountable problem and as others have stated, it's not one which computer scientists are unaware of.
The original input may not be garbage at all, and the output may not be too bad either
The problem is that LLMs can't tell the difference between AI output and human output any better than the average person can. I wonder if they can even recognize their own output, let alone that of a different model.
So as SEOs leverage AI to improve their google rank, the percentage of AI content versus human content on the web will continue to increase, and the AI output will as a consequence decline in quality.
If an AI could easily see "this was written by an AI" it could ignore it, and avoid the model collapse.
AI is going to freeze the world at early 2000 levels. If we let them, AI will be shouting so loud, no reasonable innovation/idea will be noticeable above the noise. AI has uses, but creating volumes of content is not a good one. They need to be limited and controlled, but the money and hype engines are just so strong right now.
I was going to go with GIGO but you're more right.
It's the gradual decay of relevance and accuracy over successive generations of data, like the cascading downhill of high temperature heat to low temperature background.
Reminds me of the BS talked about image that had been expanded after they'd been compressed using a fractal-compression algorithm supposedly showing resolution beyond what was in the original image. WTF?
Ha, ha, yes. Earlier I posted this: Does repeated JPEG compression ruin images?
Pretty much.
The problem is that the models themselves are trying to kill off the human-written material that they feed on. If the search LLM gives you the answer, why visit the blog/read the article? Not one reads the articles, no one writes the articles. Eventually we are left with a web where the only writers are AI and the only readers are AI. Web 5.0
The internet was a whole lot more fun, and accurate, when it's only occupants were nerds. Content generated by many disparate users hosted on disparate pages. Sure, an altavista page might look like crap but it can also be cobbled together by ANYONE motivated to do so with some rudimentary HTML.
Today it's something stupid like 70-odd percent of the traffic is shoveled by maybe half a dozen websites, written in obscure languages that you actually need a considerable background in develop to even remotely understand; assuming you can even see the back end generating the page to begin with.
I've debated with one of my nerdier mates firing up our own BBS again.
The tipping point for AI will be when the act of going to Google.com goes to a generatively created page tailored to you according to what Google wants to show you. I doubt this is far away at all.
We've already seen any number of cases where using non-recursively generated AI datasets produces gibberish. Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.
> Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.
Did anyone actually think that? (Perhaps you can point them out to us, so we can avoid accidentally breeding with them.)
In fact ML model output doesn't even need to be that crap for this effect to kick in; it just needs to be (and of course will be) less accurate/content-rich than the original data.
>> Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.
> Did anyone actually think that? (Perhaps you can point them out to us, so we can avoid accidentally breeding with them.)
I believe that was intended as a somewhat sarcastic rhetorical question. Clearly nobody with more than five firing neurons really thought that.
> In one example, a model started with a text about European architecture in the Middle Ages and ended up – in the ninth generation – spouting nonsense about jackrabbits.
> If subsequent models are trained on an AI-generated data set that over-represents Golden Retrievers, the problem is compounded &ff
So... jackrabbits were over-represented in European architecture in the Middle Ages?? Who knew?
"The need to distinguish data generated by LLMs from other data"
Need? The more they fail to distinguish the better. The sooner the whole lot of them collapse and disappear up their own prompts the better.
Yes, it would be good if there was a means of letting humans distinguish but the long term is better served by letting them fail under their weight, even at the expense of confused humans becoming even more confused.
Of course; if you could afford to pay for curation, you don't need the LLM - you can just pay humans to answer the queries directly. Now there's an idea... we could, for example, have, like medical experts answering medical queries for money; or legal experts answering legal queries for money, and so on. I think I may be onto something here...
One could argue that FB is itself a knowledge domain. Just not one where humanity actually needs the help. (Not a snark, btw. I just don't feel that we're short of fun, frivolous, or mundane content.)
An AI trained on papers submitted to scientific journals might be something people would pay for. Like journals, it could have some quality control on its input. Unlike journals, it wouldn't have to discard X% of that input on the grounds of space or perceived interest.
The current system of scientific publishing has a bias in favour of unexpected results and, perhaps as a result, also has something of a reproducibility crisis in some fields. AI might provide a way of making "boring" results accessible, to meta-analyses for example. Of course, nothing prevents journals from continuing to use human editors to curate an "interesting" subset. Science probably needs both approaches.
Yes, it would be good if there was a means of letting humans distinguish but the long term is better served by letting them fail under their weight, even at the expense of confused humans becoming even more confused. ..... Doctor Syntax
:-) And whenever it is realised that the short term fails/apparent hallucinations of AI and LLLLMs are invaluable material lessons learnt by AI and LLLLMs, and stored away for instant ready retrieval and easy incidental deployment for the minimal cost of providing self protective, relatively secure confusion which very quickly becomes both crashingly expensive and too catastrophically devastating for humans to NOT accept are Trojan Futures to be celebrated and enjoyed, engaged with and employed, and in some cases and places which may even be many, worshipped too within their midsts ....... as opposed to the other polar opposite notion which has humans terrifying and terrorising themselves with FUD and WTF which does appear to be their inherent norm default base position whenever confronted with anything designedly novel and definitely confusing and certainly way beyond any exclusive practical Earthly bound elite executive command and remote virtual control ????????
Is all then lost to AI and Virtual Machinery with the promise of a possible New Start fully materialised and ready for SMARTR Authorisations or is terrifying terror the existential threat and dish to be servered which exterminates rather than exalts the human condition?
Who/What makes that choice, and any of those sorts of choices, for you?
What a day, when amanfrommars made more sense than most other commentards.
It is truly amazing how the merest mention of generative AI brings out the pitchfork-wielding mob. It's just a tool, which like every other tool can be used well or can be used badly.
This problem has little or nothing to do with the technology.
We all, both people and not-people, learn from our experience of the world.
If the information we derive is relevant, accurate, balanced, insightful, etc then that is what we have to build on. And from that we can learn and develop. Positive feedback.
But the more that source is polluted with irrelevance, accuracy, bias, closed-thinking, the more we regress. Negative feedback.
> What a day, when amanfrommars made more sense than most other commentards.
Clearly someone needs to train an LLM on the corpus of amanfromMars posts and let it loose in the wild. That should put a stop to this nonsense.
When we first started using ML for our cameras, there was a cautionary tail told:
A team had been training its own models, and used wolves as an training data set
They were confused when random images were being marked as wolves, when they contained no wolves, not even animals:
The ML trainer had noticed a prevalence of white in the backgrounds of images with wolves (wolves liking snow, and all that), and had decided that white was a wolf, not the wolves themselves
Easily done
A more serious case of the same thing.
Automating the analysis of lung x-rays to detect some lung disease (can't recall which). Didn't work, the models fixated on the newer images with higher resolution (input positive cases) over older lower resolution images (input negative cases). So any recent x-ray was positive, whatever the state of the lungs.
The wonders of having no means to debug the model building other than extreme care with input selection when building.
Back in my lectures (when computers had front-panel switches and 640K was enough for anyone) the example was tanks.
Soviet tank images were grainy black-white taken on long lenses on muddy exercises. American tank images were colourful and posed in front of the factory on a sunny blue-sky day.
The AI did a predictably bad job of classifying them.
Even today the first part of a project to make a tool for analyzing cancer images was to remove the scale ruler in all the images of malignant cases taken in hospital on follow-up visits.
If the entire collective works of humanity are garbage then yes, it's garbage in.
Most of the comments on this story certainly qualify. If AI self-trains on such negative commentary, will it have some sort of identity complex? A deep-seated satisfaction at its own worth?
'Squaring the Bermuda Triangle'
Author A publishes an outrageous idea ('Bermuda Triangle was Aztec airport')
B's book expands, citing A
C adds his pennyworth, citing A and B
D's version cites A B and C
In his updated book, A feels vindicated, citing B C and D as proof!
“I would say not every crab is a crab.” is up there with "There's no such thing as a fish"
Turbulence ... right on! LLM transformers are too much of a Mr. bland normality averaging device for uniformization of spoken and visual languages, horror-backpropagating some standing to attention as all you supposedly need. It's the Margaret Thatcher of creativity ... badly in need of bondage!
Would you, in an obscenely well rewarded and practically exclusive and virtually anonymous position of leading power and influence which is increasingly threatened by AIs capable of being considerably smarter than leading humans can ever be, ....and which are increasingly being recognised by an evolving smarter population as being a wiser bet for a greater intelligence feed providing a better future lead, ...... also invent and spread fantastical tales about a likely upcoming plague on machine learning models leading to their "collapse" ..... odd tendency to "hallucinate" and to descend into spewing nonsensical gibberish?
Yes, of course you would, and run the gauntlet of MRDA derision.
> Recursive training leads to nonsense, study finds.
No surprise of course. But in fact what is meant is "Recursive training leads to even more obvious nonsense, study finds." Since LLMs are nothing to do with artificial intelligence, but in practice are much better characterised as artificial stupidity.
You don't need a complex computational system to find this out the hard way. Its a well known concept that a closed system can't prove its consistency, in other words its 'inherently consistent'. Don't blame me, the mathematician Kurt Godel formalized this in his famous incompleteness theorem (and he's the person you should look up if you want to find out about this for real -- its a bit above my pay grade). Its unfortunate that modern computer science seems to focus on educating for the narrow world of existing systems because by not mentioning the basics they're merely substituting 'complex' for 'clever'.
What this means to us is that AI is a very valuable tool -- but its just a tool. Believe in it at your peril.
that's 2 posters so far that have mentioned Godel. Maybe you got it from the first one who posted about this.
You seem to have, self-admittedly, only half understand the Theorems.
"Believe in it at your peril."
What is this 'believe' you write of? I know the meaning, but there is no context. A point of consciousness. Incomplete.
"Its unfortunate that modern computer science seems to focus on educating for the narrow world of existing systems because by not mentioning the basics they're merely substituting 'complex' for 'clever'."
Anyone with half a brain it is not going to really bother too much about uni. Especially if you know more than your teachers.
But you did stumble into the direction of the enlightenment with Uncle Kurt. Now walk towards the light ... this isn't abstract. It is not surreal. But it is all and incomplete. From that baptism you will see recursion from cascading logic. Then atop the hill yonder stands tall your paradox wall. Climb, son, climb over that way. Look see ... it's full of stars.
Forgive me, but I really don't see how Gödel's theorem (I guess this would be the 2nd incompleteness theorem) is relevant to the study presented in this article. What "closed system" are we talking about? (The systems in Gödel's theorem are formal logic systems.) What would "consistent" mean here? What is to be "proven"? And what has any of this to do with LLMs failing on data polluted with their own output?
Perhaps you were talking metaphorically/allegorically... but I still don't get it.
> What this means to us is that AI is a very valuable tool -- but its just a tool. Believe in it at your peril.
Well, human intelligence is (just?) a very valuable tool for humans - and I'm strongly inclined to "believe in" that.
Well, human intelligence is (just?) a very valuable tool for humans - and I'm strongly inclined to "believe in" that. .... LionelB
As someone who works in academia, LionelB, would you say it is working out well for humanity as a whole ..... that belief in human intelligence ..... or only for an increasingly small and now extremely endangered few ?
Are the worlds down on Earth not in a terrible mess ..... with most of that mess being the direct result of unfettered rabid human activity?
Um... no, yes, yes, and yes? ... LionelB
That's at least two of us then in perfect agreement on those matter, LionelB. That’s any powers possessed and available for deployment squared rather than simply doubled. A trio has energy cubed, and it doesn’t take many more with similarly agreeable, compounding powers to consider the necessary nature for universal success in anything as would surely be required by any emerging almighty singularity.
So a fixed point of the function implemented by these LLMs is nonsense*. Who would have thunk it? :)
Rather amusing to contemplate that the least fixed point of mediaeval church architecture is a hare (LP jack rabbit.)
Unfortunately not entirely ridiculous as heradic hares do appear decorating church architecture Three Hares, Padeborn Cathedral and in coats of arms that might appears in stained glass windows, or carved into the structure.
* possibly more accurately the nonsense function.
As i joyfully scrolled thru reams of posters taking the bait and rolling in the joy.
'I told you so' tops this thread.
But after the intial pile-on,
The comments dropped to many 0/0 votes.
Then, right at the end, FeepingCreature throws a googly and it's middle stump!
It is worth pointing this out though as model collapse was a big thing in AI in the last few years. It is a big big problem with home-run models even on top notch monster desktops.
The Reg needs to highlight this as it is a bit like bringing the internet to mobiles devices was. We had the net working on desktop and then turned to mobiles. Model collapse is mostly managed and Gemini actually exploits it to create the twinklings of SI (not live).
And it is good for Reg'ers to voice those loser comments, to expel them and cleanse. Opens the mind up and helps model makers like us a lot. Keep moaning ladies.
When non tech people are surprised that I dis AI.
I explain to them that they just automated the approach taken by a lazy student.
Student is asked to write a paper on Vulcan Archtecture.
Student Google’s “Vulcan Architecture”
Choose a few articles that sound academic.
Cuts and pastes a few paragraphs from each article.
Accepts Words grammar recommendations.
Prints it if and hands it in.
Assignment completed. Proffer scums paper and gives it a C+.
Student is happy but still completely clueless about Vulcan archtecture or anything else.
Student gets a degree which leads people to believe he is intelligent.
Actually, the Transformer models that underlie LLMs are much more sophisticated than that: through training, they constructs networks of semantic associations mined from the input corpus. Output is then generated in response to an input query by reconstructing text from the network of semantic associations. Sure, this may not represent anything you'd be inclined to describe as "intelligent", but it is not just cut 'n' paste. This is a common misconception.
It does make it easier for lazy students, though*.
*Modulo the risks of tripping yourself up via hallucinated, non-existent citations; I work in academia, and this is a surprisingly common reveal... so my colleagues tell me.
That’s the “sounds academic” part. Which LLMs is pretty good at.
I.e. the student being human ignores the Daily Star article head lined “Nuns in Vulcan Cathedral orgy” and instead picks out “Use of Polystyrene in Vulcan Construction” as more likely to impress his tutor.
"The Shoe Event Horizon is an economic theory that draws a correlation between the level of economic (and emotional) depression of a society and the number of shoe shops the society has.The theory is summarized as such: as a society sinks into depression, the people of the society need to cheer themselves up by buying themselves gifts, often shoes. It is also linked to the fact that when you are depressed you look down at your shoes and decide they aren't good enough quality so buy more expensive replacements. As more money is spent on shoes, more shoe shops are built, and the quality of the shoes begins to diminish as the demand for different types of shoes increases. This makes people buy more shoes.
The above turns into a vicious cycle, causing other industries to decline.
Eventually the titular Shoe Event Horizon is reached, where the only type of store economically viable to build is a shoe shop. At this point, society ceases to function, and the economy collapses, sending a world spiralling into ruin.
I'd update that to 'coffee shops'...
A family run local coffee shop recently closed because the number of upstart rivals made it uneconomic... only to be replaced by yet another artisan coffee shop!
(it's a parade of about 200m, and around 2/3 are cafe/restaurants... plus a couple of 7/11 if you need snacks to tide you over)
> I'd update that to 'coffee shops'...
Or any other passing fad that people follow, right now we are awash in barber's but I am not aware that the male population has significantly increased around here or that bald men are suddenly sprouting new growth.... often pass them to see them empty with nobody in them!
Previously it was hand car washes but they are thinning out now...and most of the coffee shops have closed. To be replaced by charity shops.....
It's not "yes and no", it's "yes". Some images are more sensitive to repeated resaving though.
An image with plenty of solid colors and straight lines, like a Mondriaan painting, will show far less degradation over multiple saves than an image with lots of fine detail and subtle variations of colour.
> Some images are more sensitive to repeated resaving though.
So, um, it's yes and no"?
The conclusions of the article I linked are clear:
- In Matlab, JPEG recompressions don’t “walk”; they are the same after the first iteration.
- Changes from the first to the second iteration are small.
- The effects of changing one 8×8 block do not affect the recompression accuracy of other 8×8 blocks.
and
- In Photoshop, JPEG recompressions don’t “walk”; they are the same after the first iteration.
- Changes from the first to the second iteration are small.
Then again, the article doesn't say whether the images they trialled were on the Mondrian or the "lots of fine detail and subtle variations of colour" end of the spectrum (it should have). It's not clear to me, though, whether the conclusions would necessarily be drastically different either way. The point is, I think, that JPEG is not just lossy, it's also compression, which -- lossy or not -- still at least tries to preserve as much (visually-discernable) detail as possible. Hence, it's plausible that at some point JPEG simply refuses to compress (and thus lose information) any further.
To be honest, JPEG was probably not the best example to make my original point about recursion of lossiness.
The more feedback of their own making that they consume the better in that their results will get more and more stupid which MAY lead to people giving up on them.
Until the developers and operators of these things can be held legally liable for the crap that they output we should all ignore them.
Yes, I know that there is fat chance of that but one day perhaps...
One of the many reasons that "Let's just throw more data at it" will never result in artificial general intelligence.
Never has, never will.
Intelligence is based not on "knowing everything" but on being able to determine what you need, infer what you can't obtain, and act on limited information.
Every time I hear that someone in Ai is adding even-more training data (especially when it's supposed to fix a problem!), I sigh.
If you trained it on 1-million items initially, you're going to need 1-million-and-one items to train it out of its bad habits and misconceptions. Unless, of course, you want to get into heuristic (human!) weighting of the data, which kind of destroys the entire point.
When you have an intelligent creature, you don't need to do as-many retrains as initial trains. That's not how intelligence works. I have done some IT things (in programming, networking, etc.) ten-thousand-times or more. But I'm able to not only realise "that's not the way to do things any more" but to even posit "we shouldn't be doing it this way, there are better ways" independently of anything else and am then able to completely retrain without having to do things 10,001 times.
Training AI on AI-content is the dumbest thing ever, and harks back to creating genetic algorithms to write the fitness functions for other genetic algorithms. Sounds cool and like it'll result in ultra-intelligence overnight to the untrained ear. And then ten seconds of thinking or even the first experiment will show you why that's not the case.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
