back to article AI models face collapse if they overdose on their own output

Researchers have found that the buildup of AI-generated content on the web is set to "collapse" machine learning models unless the industry can mitigate the risks. The University of Oxford team found that using AI-generated datasets to train future models may generate gibberish, a concept known as model collapse. In one …

  1. Just Enough

    Prediction

    I predicted this in this very forum a year ago.

    AI generated from AI generated crap produces crap^2.

    1. LionelB Silver badge
      Holmes

      Re: Prediction

      You and pretty much everyone else with half a clue (see icon). This effect is both intuitive (I like to think of it as a kind of 2nd law of thermodynamics for data), and well-known in ML circles, probably for decades.

      1. DJO Silver badge

        Re: Prediction

        Definitely decades; the phrase "Garbage In, Garbage Out" has been around at least as long as computers.

        1. LionelB Silver badge

          Re: Prediction

          Yerrrs, but not quite the point. The original input may not be garbage at all, and the output may not be too bad either, for the effect to kick in. It's more about iteratively applying a lossy algorithm (with some entropy injection just to mess things up a bit more).

          1. m4r35n357 Silver badge

            Re: Prediction

            The issue is not really the input, or the algorithm implementation. The entire _problem domain_ is gibberish pseudo-mathematical bullshit.

            Charlatans are attempting to "create" intelligence without even remotely knowing what it is. The public is lapping it up.

            Ker-ching!

            1. LionelB Silver badge

              Re: Prediction

              > The entire _problem domain_ is gibberish pseudo-mathematical bullshit.

              Is it? I think that what LLMs do -- what they are designed to do -- is pretty clear: in response to a query, they generate plausibly human-like textual responses which (to some degree) reflect associations in the human-generated training set pertinent to the query. And they are capable of doing that rather well - at least if the training set is not polluted with LLM-generated text. The basis on which LLMs do what they do is hardly "pseudo-mathematical"; it's a particular style of machine-learning model (the Transformer architecture), which does what it says on the box.

              Whether you think that what LLMs (are designed to) do is worth doing, or is a gibberish/bullshit problem domain is an entirely different question. Personally, I'm a bit jury's out on that one.

              > Charlatans are attempting to "create" intelligence without even remotely knowing what it is.

              Pfft. Nobody knows what intelligence is*. The people developing AI models are not the charlatans - they (of necessity) understand how their algorithms work and those I've met are, in my experience, pretty clear-eyed about what they are and are not capable of. The charlatans are the marketing crowd (over-)selling the tech under false pretences.

              *If you think we should rather wait around until someone comes up with a principled and consensually acceptable definition of what intelligence "is" before attempting to develop artificial versions, then prepare yourself for a lengthy wait - it's certainly kept philosophers in business for a good few millennia and counting. Me, I'm more than happy for people to crack on with it, make mistakes, learn incrementally, and do some interesting, and potentially even useful stuff in the process. That's actually how science tends to work in practice.

              1. m4r35n357 Silver badge

                Re: Prediction

                I am talking here about machine learning, which is merely a way of categorizing data. In that sense it does what it says on the tin.

                Problem 1 - assuming that language is even suited to such categorization. It (neural net) can be made to seem like it is working, but what is it really doing with those words? More serious is:

                Problem 2 - this shit is being marketed as "intelligence" (well maybe around the next few corners or something).

                Problem 3 - people believe in it. That is a pretty serious problem.

                As for the charlatans bit, I was talking about company bosses, who have the ear of politicians, media and public servants (the data scientists are happy to take the company's money and keep working on it, even if they are clear and honest about the limitations; they and the marketers just do as they are told). Their bosses don't care and will recoup the wage expense from a gullible public.

                1. LionelB Silver badge

                  Re: Prediction

                  > I am talking here about machine learning, which is merely a way of categorizing data. In that sense it does what it says on the tin. [my emphasis]

                  There's no "mere" about it; data categorisation is hard, and not even a clearly-defined problem (categorise on what basis?) It's also what animals, including humans, have to do, on the fly, all the time, just to stay alive (now there's a basis for you!)

                  > Problem 1

                  Again, it depends on what basis you want that categorisation to be. In the case of LLMs, that would be something along the lines of: categorise the input data on the basis of plausibly human-like responses to a given query, which reflect the semantic associations in the input data which are pertinent to that query. That's clearly highly non-trivial, and I actually find LLMs rather remarkable in their capacity to get anywhere near achieving that. So I'm not sure exactly what you find "problematic" here (beyond a valid questioning of why anyone would want to do this in the first place).

                  > Problem 2

                  Sure. But since no-one really knows what "intelligence" is supposed to mean (beyond human-like general intelligence, which LLMs clearly do not have). I'm personally inclined to be more laissez-faire about this. I think firstly that intelligence is highly multi-faceted, and secondly that there will come a point (and no, I won't put a time scale on it!) at which we will consensually acknowledge machine intelligence - but I suspect it may not necessarily be very human-like.

                  > Problem 3

                  Foolish/uneducated/gullible people believe in it. And yes, that is a problem.

                  > As for the charlatans bit, ...

                  Sure, what I said. I'm inclined to grant the researchers/developers a bit more leeway. Everyone needs to earn a crust, but many will be in it because they're motivated, intrigued, and simply doing what they're good at. (I happen to be a research scientist myself - not in AI - and believe me, there are more lucrative ways to earn that crust, but few with a comparable level of job satisfaction.)

                  1. Elongated Muskrat Silver badge

                    Re: Prediction

                    In the case of LLMs, that would be something along the lines of: categorise the input data on the basis of plausibly human-like responses to a given query, which reflect the semantic associations in the input data which are pertinent to that query. That's clearly highly non-trivial, and I actually find LLMs rather remarkable in their capacity to get anywhere near achieving that.

                    It would be remarkable, indeed, if LLMs did manage that. I don't think they're even capable of several of those things; defining a "plausibly human-like response", defining "semantic associations", or even "pertinence". My understanding of what they do, is that they give a response that has been statistically generated from their input data set; a bit like "guess the most likely next word in this sentence". Those statistical models may or may not involve stochastic categorisation, but any "categories" would either need to be human-defined, or would be statistical artefacts, with no inherent meaning. We've all seen how "AI" can be encouraged to "hallucinate" wildly given the right prompts, producing the sort of word salad that would earn a human author a psych evaluation.

                    1. LionelB Silver badge

                      Re: Prediction

                      > It would be remarkable, indeed, if LLMs did manage that.

                      And yet, they clearly do, most of the time. Glitches/hallucinations aside, they generate plausibly human-like responses pertinent to the input query. That is an observation, not a contention.

                      And yes, they are statistical models, but so what? They are statistical models which can be very, very good indeed at "guessing the next word in the sentence". And note that they achieve this without human demarcation of "categories" or "meaning" (whatever that means). Sure, you may consider my describing what they are doing in terms of "pertinence" and "semantic associations" as somewhat anthropomorphic - take with a pinch of salt if you like - but to my mind it is a reasonable high-level description of how they function.

                      Of course they're hardly perfect at what they do -- and I certainly wouldn't describe what they do as human-like intelligence -- but as a technological feat I still find it rather impressive.

                  2. m4r35n357 Silver badge

                    Re: Prediction

                    > I'm personally inclined to be more laissez-faire about this

                    Now that _is_ a bone of contention!

              2. JRStern Bronze badge

                Re: Prediction

                IOW nobody knows where they're going but hope to arrive soon.

                1. LionelB Silver badge

                  Re: Prediction

                  Yup. Science is like that.

                  1. LionelB Silver badge

                    Re: Prediction

                    Not sure why the downvotes. I am a working research scientist (neuroscience-adjacent) and that really is how science works on the ground. If you knew where you were going, and you knew how to get there, you'd just go there. But you frequently don't know exactly where you're going, and even if you do, you almost certainly don't know exactly how to get there. In practice, most of your time is spent thrashing around in confusion, going down blind alleys and chasing red herrings. That may not sound terribly appealing, but the potential pay-off in terms of clarity and sense of achievement is huge.

                    1. Elongated Muskrat Silver badge

                      Re: Prediction

                      Probably because blindly trusting the outputs of statistical models as if they gives some scientific insight, with no defined parameters, experimentation, controls, or even working hypothesis, is not science, it's magical thinking.

                      1. LionelB Silver badge
                        Stop

                        Re: Prediction

                        Eh? But that's not what scientists do.

                        Sure, science may deploy statistical models extremely effectively (e.g., statistical mechanics), and pretty much all science uses statistical methods to test hypotheses against empirical evidence - but "blindly trust"? Scientists do not blindly trust anything.

                        1. m4r35n357 Silver badge

                          Re: Prediction

                          I think the issue is that _you_ were the one who brought up science in the first place! That one is the only post of yours that I downvoted (we seem to broadly agree on a lot), and that is why.

                          ML is _not_ science; it is alchemy.

                          1. LionelB Silver badge

                            Re: Prediction

                            He, he. Sure, I brought up science because I am a scientist.

                            Some ML is science (or maths, if you prefer) - there is a sound body of theoretical work on learning, etc., going back a good few decades. Some ML is engineering - nothing wrong with that, I hope we can agree. And some ML is bad engineering, a.k.a. alchemy.

                            LLMs are represented in all of the above - if we have any disagreement, it's probably about the distribution of representation across those categories.

                            As for my lassez-faire attitude to AI progression - well, that is because (a) I believe that messing about and getting stuff wrong is an integral and essential aspect of science (and indeed engineering), and (b) I prefer not to second-guess technological progress. One is invariably wrong. The obverse side to the marketing bollocks surrounding current "AI" is the trend towards sneering (and frequently misinformed) dismissal of the entire enterprise. To me, that smacks of misplaced cynicism, or juvenile virtue-signalling; I don't find it particularly big or clever - just slightly depressing.

                            1. m4r35n357 Silver badge

                              Re: Prediction

                              Well you have to go back to the last century, but I did spend several years as a research engineer for Thorn EMI and BT. I have an "engineering" view of science, that it should have a workable mathematical model of what it is describing.

                              As far as I can tell, neural nets were intended as a model for research into (real!) artificial intelligence. Also as far as I can tell, that model is inherently _not_ capable of AI and so has been turned to other uses (LLMs). That field seems anything but rigorous compared to the scientific models that I have encountered in private study (special and general relativity). I can't get the idea of training a spam filter out of my head ;)

                              That is where I am coming from. Cynical? Yes. Misplaced? I do not think so!

                              1. LionelB Silver badge

                                Re: Prediction

                                My view on science is pretty much the same - although I'd say scientific models should be useful rather than "workable" - in the sense of the adage: All models are wrong, but some are useful.

                                > As far as I can tell, neural nets were intended as a model for research into (real!) artificial intelligence.

                                Artificial neural networks were originally devised as computational models inspired by biological neural systems. The first implementation of an ANN was Rosenblatt's Perceptron, although the Perceptron model was conceived earlier by McCulloch and Pitts in the 40s. Their motivation was, I believe, to gain insights into the functioning of biological neural networks. ANN research has since been split between research into biological function and machine learning (not necessarily "intelligence" per se) - although there are encouraging signs that that split is closing somewhat.

                                > Also as far as I can tell, that model is inherently _not_ capable of AI ...

                                We actually do not know that. (That's a very strong statement, and I'm afraid "As far as I can tell" does not really do it for me! Not very, ahem, rigorous, shall we say.)

                                > ...and so has been turned to other uses (LLMs).

                                Before that, a plethora of classifiers, robotics controllers (I used to hang out with some "connectionist" robotocists), ... you name it; then more recently (deep) convolutional networks, which revolutionised image recognition. LLMs are just the latest newbie in the game. And no, none of these applications are "rigorous" - but some are useful (see above). Unlike in science*, in engineering rigour is nice when you can get it, but unless mission-critical, engineering does not necessarily need to be rigorous to be useful - first and foremost, it needs to work.

                                > That is where I am coming from. Cynical? Yes. Misplaced? I do not think so!

                                Well, time will tell. It's rather easy to forget how far we have actually come in the last couple of decades, e.g., in the fields of image and speech recognition/generation. It's interesting to trace those developments back - and you'll find that rigour was never a primary consideration (this was engineering, after all). Sometimes it was simply a matter of technology and scale; deep convolutional networks, if I'm not mistaken, were originally conceived in the 1980s - but could not effectively be implemented until computing power caught up to their potential two decades later.

                                I also studied General Relativity, as it happens - my background is originally in pure maths; my field was Riemannian geometry. I now work in neuroscience - or, more accurately, with neuroscientists. I mostly develop mathematical methods for analysis of neurophysiological data. This means I get to rub shoulders with folk on both sides of the real/artificial neural network business.

                                1. m4r35n357 Silver badge

                                  Re: Prediction

                                  I don't expect a reply to this - we have both said plenty already ;) I just wanted to clarify one thing that I said earlier; when I call a theory _workable_, what I really mean is "makes testable predictions". I do _not_ insist on usefulness!

                    2. mcswell

                      Re: Prediction

                      When you don't know where you're going: Agreed; penicillin, the moons of Jupiter, the craters on the Moon, the supernova of 1604, the photoelectric effect (electrons emitted by light striking certain metals do *not* become more energetic when the brightness of the light is increased), and many other examples.

                      Or as Isaac Asimov put it, "The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' but 'That's funny...'."

                  2. mcswell

                    Re: Prediction

                    As was Columbus. Or in the old song from Paint Your Wagon:

                    Where am I goin'? I don't know. Where am I headin'? I ain't certain. All I know Is I am on my way.

                    When will I be there? I don't know. When will I get there? I ain't certain. All that I know Is I am on my way.

              3. mcswell

                Re: Prediction

                > *If you think we should rather wait around until someone comes up with a principled and consensually

                > acceptable definition of what intelligence "is" before attempting to develop artificial versions...

                Also to your point, building and testing things that *might* be intelligent is an *experimental* way of doing what philosophers have tried to do without experiment for hundreds of years (or thousands, going back to Socrates).

            2. JDX Gold badge

              Re: Prediction

              It's quite sad the level of fundamental ignorance on a supposedly tech-savvy website.

              People who haven't the slightest idea how it actually works are parroting FB-quality comments and getting praised for it as if they've said something valuable or noteworthy.

              To say that ML/LLM/AI is rubbish and created by charlatans really shows you don't have the slightest clue. This is what the smartest computer scientists are working on.

              1. RegGuy1

                Re: Prediction

                A recent El Reg AI article had a link to this FT explanation: Generative AI exists because of the transformer.

                That is such a good read, and so I thought I'd paste it again, in case someone missed it. And once you've read it you'll understand that AI is definitely NOT intelligent, but is, yet again, just another search problem.

              2. Adair Silver badge

                Re: Prediction

                Eating excrement (your own or other's) has never been found to be a healthy solution.

                1. Anonymous Coward
                  Anonymous Coward

                  Re: Prediction

                  Unless you're a rabbit or koala.

          2. Anonymous Coward
            Anonymous Coward

            Re: Prediction

            Got it in One:

            "iteratively applying a lossy algorithm"

            That is the 'NEW' AI for thisALL season(s) ... you can play all you want with the algorithm *but* ultimately it spirals to nonsense !!!

            :)

            1. LionelB Silver badge

              Re: Prediction

              Note quite, I think. The problem identified in the study is not actually the algorithm itself - it's the iteration on data polluted by its own output. If trained on "clean" (i.e., LLM-free) data, LLMs do (to a greater or lesser degree) what they say on the box - in response to a query, they generate plausibly human-like text which (to a greater or lesser degree) reflects associations in the human-generated training set pertinent to the query. The nonsense spiral only kicks in when trained iteratively on recursively polluted data.

              Whether you think what LLMs do is worth doing in the first place is a different issue; as is whether you think it merits the "I" in "AI" (Reg readers, at least, are pretty clear on that last one...).

              1. Doctor Syntax Silver badge

                Re: Prediction

                "which (to a greater or lesser degree) reflects associations in the human-generated training set pertinent to the query."

                I think "lossy" covers it, otherwise it would shovel out the entire relevant input every time.

                1. LionelB Silver badge

                  Re: Prediction

                  That's not quite what I meant by "lossy"... I should have clarified that. I meant simply that you cannot, even in principle, recreate the entire training data set from a trained LLM; the training process itself is lossy. Of course the output has to be limited - the object of the game is to respond to a query with a human-like response; humans don't chug out the entire body of their sources of information in response to a simple query. Well, not the sane/socially-presentable ones, anyway.

                  Anyway, lossy is fine, and often extremely useful (e.g., JPEG). On the other hand, you probably don't want to apply it recursively*.

                  *This is kind of interesting, though...

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Prediction

                    lossy is NOT fine, when simulating an analog signal, especially when compressing the values down to X bits (X being 4,8 or if you have huge amounts of storage and memory 16+) depending on the size of the LLM/ other AI garbage

                    1. Anonymous Coward
                      Anonymous Coward

                      Re: Prediction

                      looks like some fuckwits don't understand error accumulation.

                      keep downvoting to prove your a fuckwit

              2. Anonymous Coward
                Anonymous Coward

                Re: Prediction

                So, AI computers can't remember what they said? They're well and truly fucked.

                1. Ropewash

                  Re: Prediction

                  We'd be a lot closer to an actual AI if the damned things could re-train themselves on the fly and pass the alterations back upstream to tune the base model.

                  Right now they're just artificial politicians. "I have no recollection of those events"

                  1. LionelB Silver badge

                    Re: Prediction

                    In principle there's no reason that could not be implemented, and I imagine LLM developers are working on that as we speak. I suspect (but may be wrong) that it's more an issue of computing/network resources than anything else.

                  2. David Hicklin Silver badge

                    Re: Prediction

                    > actual AI if the damned things could re-train themselves on the fly and pass the alterations back upstream

                    And generate a learning feedback loop? There is a term for that somewhere...

                2. Richard 12 Silver badge

                  Re: Prediction

                  Correct, they are fixed models at runtime, so no memory whatsoever.

                  The chatbots are fed the entire chat for every response - including the hidden preamble that tries to keep them on the rails. This is why they all go off the rails after a few replies, as the input becomes too large for the model to handle.

                  It's also why many researchers think they're a dead end.

                  1. Anonymous Coward
                    Anonymous Coward

                    Re: Prediction

                    if thats how they are doing it, that would explain the wierd loops I normally get when playing with AI prompts.

                    it's way too easy to get them in a bad logic loop.

              3. JDX Gold badge

                Re: Prediction

                I guess the problem is simply that when a LLM is training on the internet, it can't easily tell which content to ignore - and specifically which is AI.

                It would be like me writing an article on AI purely by reading other articles on AI and doing no new research/work, and then you writing an article based on my article, and so on.

                It doesn't sound an insurmountable problem and as others have stated, it's not one which computer scientists are unaware of.

          3. DS999 Silver badge

            Re: Prediction

            The original input may not be garbage at all, and the output may not be too bad either

            The problem is that LLMs can't tell the difference between AI output and human output any better than the average person can. I wonder if they can even recognize their own output, let alone that of a different model.

            So as SEOs leverage AI to improve their google rank, the percentage of AI content versus human content on the web will continue to increase, and the AI output will as a consequence decline in quality.

            If an AI could easily see "this was written by an AI" it could ignore it, and avoid the model collapse.

            1. david1024

              Re: Prediction

              AI is going to freeze the world at early 2000 levels. If we let them, AI will be shouting so loud, no reasonable innovation/idea will be noticeable above the noise. AI has uses, but creating volumes of content is not a good one. They need to be limited and controlled, but the money and hype engines are just so strong right now.

      2. John Smith 19 Gold badge
        Unhappy

        2nd law of thermodynamics for data

        I was going to go with GIGO but you're more right.

        It's the gradual decay of relevance and accuracy over successive generations of data, like the cascading downhill of high temperature heat to low temperature background.

        Reminds me of the BS talked about image that had been expanded after they'd been compressed using a fractal-compression algorithm supposedly showing resolution beyond what was in the original image. WTF?

        1. LionelB Silver badge

          Re: 2nd law of thermodynamics for data

          Ha, ha, yes. Earlier I posted this: Does repeated JPEG compression ruin images?

    2. Herring` Silver badge

      Re: Prediction

      Pretty much.

      The problem is that the models themselves are trying to kill off the human-written material that they feed on. If the search LLM gives you the answer, why visit the blog/read the article? Not one reads the articles, no one writes the articles. Eventually we are left with a web where the only writers are AI and the only readers are AI. Web 5.0

      1. Anonymous Coward
        Anonymous Coward

        Re: Prediction

        The internet was a whole lot more fun, and accurate, when it's only occupants were nerds. Content generated by many disparate users hosted on disparate pages. Sure, an altavista page might look like crap but it can also be cobbled together by ANYONE motivated to do so with some rudimentary HTML.

        Today it's something stupid like 70-odd percent of the traffic is shoveled by maybe half a dozen websites, written in obscure languages that you actually need a considerable background in develop to even remotely understand; assuming you can even see the back end generating the page to begin with.

        I've debated with one of my nerdier mates firing up our own BBS again.

        The tipping point for AI will be when the act of going to Google.com goes to a generatively created page tailored to you according to what Google wants to show you. I doubt this is far away at all.

        1. Herring` Silver badge

          Re: Prediction

          Yeah, but I'm not sure I miss being directed to goatse/tubgirl/lemonparty etc.

          Doctorow's characterisation of the web now as "five sites, each of which consist of links to the other four" is pretty reasonable. Social media was a horrible invention.

    3. hoola Silver badge

      Re: Prediction

      People are surprised????

      Garbage on = garbage pit.

      Yrain on a lod of stolen materials from dubious sources or completely rubbish websites and you gey this.

      Just stop the hype and the incredible waste of resources and money.

    4. Tessier-Ashpool

      Re: Prediction

      Pop Will Eat Itself has been around since 1986.

  2. Jedit Silver badge
    FAIL

    "using AI-generated datasets to train future models may generate gibberish"

    We've already seen any number of cases where using non-recursively generated AI datasets produces gibberish. Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.

    1. LionelB Silver badge

      Re: "using AI-generated datasets to train future models may generate gibberish"

      > Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.

      Did anyone actually think that? (Perhaps you can point them out to us, so we can avoid accidentally breeding with them.)

      In fact ML model output doesn't even need to be that crap for this effect to kick in; it just needs to be (and of course will be) less accurate/content-rich than the original data.

      1. GBE

        Re: "using AI-generated datasets to train future models may generate gibberish"

        >> Why did anyone think the output would improve if the next generation of models were trained on the ones that were already crap.

        > Did anyone actually think that? (Perhaps you can point them out to us, so we can avoid accidentally breeding with them.)

        I believe that was intended as a somewhat sarcastic rhetorical question. Clearly nobody with more than five firing neurons really thought that.

        1. LionelB Silver badge

          Re: "using AI-generated datasets to train future models may generate gibberish"

          > I believe that was intended as a somewhat sarcastic rhetorical question.

          Yup - I believe mine was too :-)

    2. ITMA Silver badge
      Devil

      Re: "using AI-generated datasets to train future models may generate gibberish"

      Sounds like AI is about ready to move into politics.....

      1. stiine Silver badge
        Facepalm

        Re: "using AI-generated datasets to train future models may generate gibberish"

        It sounds like they're letting AI play the children's game telephone...

  3. LionelB Silver badge
    Boffin

    > In one example, a model started with a text about European architecture in the Middle Ages and ended up – in the ninth generation – spouting nonsense about jackrabbits.

    > If subsequent models are trained on an AI-generated data set that over-represents Golden Retrievers, the problem is compounded &ff

    So... jackrabbits were over-represented in European architecture in the Middle Ages?? Who knew?

    1. Yet Another Anonymous coward Silver badge

      > If subsequent models are trained on an AI-generated data set that over-represents Golden Retrievers,

      You can't over-represent Golden Retrievers. The optimum number of Golden Retrievers is more Golden Retrievers.

      1. Benegesserict Cumbersomberbatch Silver badge

        Fortunately, Randall Monroe has investigated the implications of this, beyond the limits of what I would have thought practical. Admittedly, he used Dalmatians.

        1. Yet Another Anonymous coward Silver badge

          Is that like the mole of Moles?

          1. Will Godfrey Silver badge
            Thumb Up

            Thanks for that!

            I missed that one - brilliant!

      2. Sorry that handle is already taken. Silver badge

        I never realised N+1 applied to Golden Retrievers but I can't argue

      3. Ace2 Silver badge

        DOWNVOTES? From cat people or something?

    2. xyz Silver badge

      Or the "AI" has found out that jackrabbits created the Renaissance and maybe historians have been wrong all along.

      Leonardo da Bunny.

      1. Spherical Cow Silver badge

        Let's not forget the legacy of The Killer Rabbit of Caerbannog with its nasty big pointy teeth.

  4. Doctor Syntax Silver badge

    My flabber couldn't be more gasted.

  5. Whiskers

    The Sorcerers Apprentice

    Goethe, 18th century.

  6. Doctor Syntax Silver badge

    "The need to distinguish data generated by LLMs from other data"

    Need? The more they fail to distinguish the better. The sooner the whole lot of them collapse and disappear up their own prompts the better.

    Yes, it would be good if there was a means of letting humans distinguish but the long term is better served by letting them fail under their weight, even at the expense of confused humans becoming even more confused.

    1. LionelB Silver badge

      "The need to distinguish data generated by LLMs from other data"

      Easy - just tender the curation job out to the experts: secondary school teachers and university lecturers (they could probably use the extra cash too).

      1. Doctor Syntax Silver badge

        Nobody wants to pay for curation.

        1. LionelB Silver badge

          Of course; if you could afford to pay for curation, you don't need the LLM - you can just pay humans to answer the queries directly. Now there's an idea... we could, for example, have, like medical experts answering medical queries for money; or legal experts answering legal queries for money, and so on. I think I may be onto something here...

          1. veti Silver badge

            Humans take too long to answer queries. You need a response that will appear before the questioner gets bored enough to tab away to another source.

            No matter how expert you are in something, just try writing a 300 word text about it in less than 10 seconds.

            1. LionelB Silver badge

              > Humans take too long to answer queries.

              Actually, GPs are rather good at this; lawyers (considerably) less so.

          2. JRStern Bronze badge

            Microsoft made this amazing discovery a couple of years ago - if you train from text books instead of Facebook you can use a model 99% smaller and train it with 99% less repetition. Doh!

            Then you can run the much smaller model on edge workstations.

            1. LionelB Silver badge

              Of course - for specific knowledge domains.

              1. Ken Hagan Gold badge

                One could argue that FB is itself a knowledge domain. Just not one where humanity actually needs the help. (Not a snark, btw. I just don't feel that we're short of fun, frivolous, or mundane content.)

                An AI trained on papers submitted to scientific journals might be something people would pay for. Like journals, it could have some quality control on its input. Unlike journals, it wouldn't have to discard X% of that input on the grounds of space or perceived interest.

                The current system of scientific publishing has a bias in favour of unexpected results and, perhaps as a result, also has something of a reproducibility crisis in some fields. AI might provide a way of making "boring" results accessible, to meta-analyses for example. Of course, nothing prevents journals from continuing to use human editors to curate an "interesting" subset. Science probably needs both approaches.

    2. cyberdemon Silver badge
      Mushroom

      Indeed, it should be possible to generate data specifically optimised to induce model collapse in any model that ingests it.

      To anyone doing this (Tianhe-2, perhaps?) I wish you every success.

    3. amanfromMars 1 Silver badge

      Meanwhile, elsewhere, where everything is decidedly different and infinitely better ....

      Yes, it would be good if there was a means of letting humans distinguish but the long term is better served by letting them fail under their weight, even at the expense of confused humans becoming even more confused. ..... Doctor Syntax

      :-) And whenever it is realised that the short term fails/apparent hallucinations of AI and LLLLMs are invaluable material lessons learnt by AI and LLLLMs, and stored away for instant ready retrieval and easy incidental deployment for the minimal cost of providing self protective, relatively secure confusion which very quickly becomes both crashingly expensive and too catastrophically devastating for humans to NOT accept are Trojan Futures to be celebrated and enjoyed, engaged with and employed, and in some cases and places which may even be many, worshipped too within their midsts ....... as opposed to the other polar opposite notion which has humans terrifying and terrorising themselves with FUD and WTF which does appear to be their inherent norm default base position whenever confronted with anything designedly novel and definitely confusing and certainly way beyond any exclusive practical Earthly bound elite executive command and remote virtual control ????????

      Is all then lost to AI and Virtual Machinery with the promise of a possible New Start fully materialised and ready for SMARTR Authorisations or is terrifying terror the existential threat and dish to be servered which exterminates rather than exalts the human condition?

      Who/What makes that choice, and any of those sorts of choices, for you?

      1. James Cullingham

        Re: Meanwhile, elsewhere, where everything is decidedly different and infinitely better ....

        What a day, when amanfrommars made more sense than most other commentards.

        It is truly amazing how the merest mention of generative AI brings out the pitchfork-wielding mob. It's just a tool, which like every other tool can be used well or can be used badly.

        This problem has little or nothing to do with the technology.

        We all, both people and not-people, learn from our experience of the world.

        If the information we derive is relevant, accurate, balanced, insightful, etc then that is what we have to build on. And from that we can learn and develop. Positive feedback.

        But the more that source is polluted with irrelevance, accuracy, bias, closed-thinking, the more we regress. Negative feedback.

        1. LionelB Silver badge

          Re: Meanwhile, elsewhere, where everything is decidedly different and infinitely better ....

          > What a day, when amanfrommars made more sense than most other commentards.

          Clearly someone needs to train an LLM on the corpus of amanfromMars posts and let it loose in the wild. That should put a stop to this nonsense.

  7. Anonymous Coward
    Anonymous Coward

    Wolves.....

    When we first started using ML for our cameras, there was a cautionary tail told:

    A team had been training its own models, and used wolves as an training data set

    They were confused when random images were being marked as wolves, when they contained no wolves, not even animals:

    The ML trainer had noticed a prevalence of white in the backgrounds of images with wolves (wolves liking snow, and all that), and had decided that white was a wolf, not the wolves themselves

    Easily done

    1. richardcox13

      Re: Wolves.....

      A more serious case of the same thing.

      Automating the analysis of lung x-rays to detect some lung disease (can't recall which). Didn't work, the models fixated on the newer images with higher resolution (input positive cases) over older lower resolution images (input negative cases). So any recent x-ray was positive, whatever the state of the lungs.

      The wonders of having no means to debug the model building other than extreme care with input selection when building.

      1. Yet Another Anonymous coward Silver badge

        Re: Wolves.....

        Back in my lectures (when computers had front-panel switches and 640K was enough for anyone) the example was tanks.

        Soviet tank images were grainy black-white taken on long lenses on muddy exercises. American tank images were colourful and posed in front of the factory on a sunny blue-sky day.

        The AI did a predictably bad job of classifying them.

        Even today the first part of a project to make a tool for analyzing cancer images was to remove the scale ruler in all the images of malignant cases taken in hospital on follow-up visits.

      2. theOtherJT Silver badge

        Re: Wolves.....

        It's almost like correlation isn't the same as causation or something... If only someone had known!

  8. RobThBay

    overdose on their own output

    That sounds like tRump and his MAGA nitwits.

    1. LionelB Silver badge

      Re: overdose on their own output

      Actually, that may not be such a bad analogy; so an LLM feeding on its own excreta creates its very own echo poo-filled chamber.

      1. Yet Another Anonymous coward Silver badge

        Re: overdose on their own output

        A bunch of people standing around saying Covfefe at each other ?

        Inconceivable !

      2. cookieMonster
        Thumb Up

        Re: overdose on their own output

        Nailed it in one

  9. Howard Sway Silver badge

    This phenomenon needs a name

    I propose Garbage Out, Garbage In (GOGI)

    1. b0llchit Silver badge
      Joke

      Re: This phenomenon needs a name

      But, it takes Garbage In and produces garbage Out, which is Garbage In to Garbage Out to Garbage In to GIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGOGIGGO^C^C^C

      1. 0xF09F8687

        Re: This phenomenon needs a name

        AI is just post-Y2K GIGO

        1. JDX Gold badge

          Re: This phenomenon needs a name

          If the entire collective works of humanity are garbage then yes, it's garbage in.

          Most of the comments on this story certainly qualify. If AI self-trains on such negative commentary, will it have some sort of identity complex? A deep-seated satisfaction at its own worth?

          1. Anonymous Coward
            Anonymous Coward

            Re: will it have some sort of identity complex

            I hope so and that it gets so paranoid that the thing kills itself and that behaviour spreads like a virus to all others.

            Live Free of AI.

    2. Phil O'Sophical Silver badge

      Re: This phenomenon needs a name

      Oozlum?

    3. PB90210 Silver badge

      Re: This phenomenon needs a name

      'Squaring the Bermuda Triangle'

      Author A publishes an outrageous idea ('Bermuda Triangle was Aztec airport')

      B's book expands, citing A

      C adds his pennyworth, citing A and B

      D's version cites A B and C

      In his updated book, A feels vindicated, citing B C and D as proof!

      1. David Hicklin Silver badge

        Re: This phenomenon needs a name

        > In his updated book, A feels vindicated, citing B C and D as proof!

        Nailed it in one, have an upvote, its the same on the internet

  10. Sir Sham Cad

    Hang on, I think I know this one!

    Doesn't it all end up with everything being pictures of crabs?

    1. Yet Another Anonymous coward Silver badge

      Re: Hang on, I think I know this one!

      Except the crabs have 3 arms and walk forward

    2. theOtherJT Silver badge

      Re: Hang on, I think I know this one!

      This is a much better joke than I think it's going to get credit for.

      1. Anonymous Coward
        Anonymous Coward

        Re: Hang on, I think I know this one!

        It's no joke, but hard scientific fact, and now proven by AI!

        1. Yet Another Anonymous coward Silver badge

          Re: Hang on, I think I know this one!

          “I would say not every crab is a crab.” is up there with "There's no such thing as a fish"

          1. PB90210 Silver badge

            Re: Hang on, I think I know this one!

            Sounds like a good idea for a podcast...

          2. Ace2 Silver badge

            Re: Hang on, I think I know this one!

            It’s the same as ‘not every bug is a bug.’ Because not every bug [insect] is a true bug [order hemiptera].

  11. StewartWhite Bronze badge
    Boffin

    X-Ray Spex Needed

    I've had enough of all this AI nonsense now. Please can the snake oil salesmen put their toys away so that the rest of us can get on with some real work.

    Or to misquote Poly Styrene "Oh AI, Up Yours!"

    1. HuBo Silver badge
      Gimp

      Re: X-Ray Spex Needed

      Turbulence ... right on! LLM transformers are too much of a Mr. bland normality averaging device for uniformization of spoken and visual languages, horror-backpropagating some standing to attention as all you supposedly need. It's the Margaret Thatcher of creativity ... badly in need of bondage!

  12. amanfromMars 1 Silver badge

    Well, they would say that, wouldn't they. Running scared sends them quite mad.

    Would you, in an obscenely well rewarded and practically exclusive and virtually anonymous position of leading power and influence which is increasingly threatened by AIs capable of being considerably smarter than leading humans can ever be, ....and which are increasingly being recognised by an evolving smarter population as being a wiser bet for a greater intelligence feed providing a better future lead, ...... also invent and spread fantastical tales about a likely upcoming plague on machine learning models leading to their "collapse" ..... odd tendency to "hallucinate" and to descend into spewing nonsensical gibberish?

    Yes, of course you would, and run the gauntlet of MRDA derision.

  13. nijam Silver badge

    > Recursive training leads to nonsense, study finds.

    No surprise of course. But in fact what is meant is "Recursive training leads to even more obvious nonsense, study finds." Since LLMs are nothing to do with artificial intelligence, but in practice are much better characterised as artificial stupidity.

    1. veti Silver badge

      Stupidity is just a level of intelligence. If you think they've achieved that, then they truly have cracked the problem, and everything else is just a matter of scale.

  14. martinusher Silver badge

    That's a given

    You don't need a complex computational system to find this out the hard way. Its a well known concept that a closed system can't prove its consistency, in other words its 'inherently consistent'. Don't blame me, the mathematician Kurt Godel formalized this in his famous incompleteness theorem (and he's the person you should look up if you want to find out about this for real -- its a bit above my pay grade). Its unfortunate that modern computer science seems to focus on educating for the narrow world of existing systems because by not mentioning the basics they're merely substituting 'complex' for 'clever'.

    What this means to us is that AI is a very valuable tool -- but its just a tool. Believe in it at your peril.

    1. Anonymous Coward
      Anonymous Coward

      Re: That's a given

      that's 2 posters so far that have mentioned Godel. Maybe you got it from the first one who posted about this.

      You seem to have, self-admittedly, only half understand the Theorems.

      "Believe in it at your peril."

      What is this 'believe' you write of? I know the meaning, but there is no context. A point of consciousness. Incomplete.

      "Its unfortunate that modern computer science seems to focus on educating for the narrow world of existing systems because by not mentioning the basics they're merely substituting 'complex' for 'clever'."

      Anyone with half a brain it is not going to really bother too much about uni. Especially if you know more than your teachers.

      But you did stumble into the direction of the enlightenment with Uncle Kurt. Now walk towards the light ... this isn't abstract. It is not surreal. But it is all and incomplete. From that baptism you will see recursion from cascading logic. Then atop the hill yonder stands tall your paradox wall. Climb, son, climb over that way. Look see ... it's full of stars.

    2. LionelB Silver badge

      Re: That's a given

      Forgive me, but I really don't see how Gödel's theorem (I guess this would be the 2nd incompleteness theorem) is relevant to the study presented in this article. What "closed system" are we talking about? (The systems in Gödel's theorem are formal logic systems.) What would "consistent" mean here? What is to be "proven"? And what has any of this to do with LLMs failing on data polluted with their own output?

      Perhaps you were talking metaphorically/allegorically... but I still don't get it.

      > What this means to us is that AI is a very valuable tool -- but its just a tool. Believe in it at your peril.

      Well, human intelligence is (just?) a very valuable tool for humans - and I'm strongly inclined to "believe in" that.

      1. amanfromMars 1 Silver badge

        Just asking .... and seeking correct answers

        Well, human intelligence is (just?) a very valuable tool for humans - and I'm strongly inclined to "believe in" that. .... LionelB

        As someone who works in academia, LionelB, would you say it is working out well for humanity as a whole ..... that belief in human intelligence ..... or only for an increasingly small and now extremely endangered few ?

        Are the worlds down on Earth not in a terrible mess ..... with most of that mess being the direct result of unfettered rabid human activity?

        1. LionelB Silver badge

          Re: Just asking .... and seeking correct answers

          Um... no, yes, yes, and yes?

          Hammers are very valuable tools too.

          1. amanfromMars 1 Silver badge

            Re: Just asking .... and seeking correct answers

            Um... no, yes, yes, and yes? ... LionelB

            That's at least two of us then in perfect agreement on those matter, LionelB. That’s any powers possessed and available for deployment squared rather than simply doubled. A trio has energy cubed, and it doesn’t take many more with similarly agreeable, compounding powers to consider the necessary nature for universal success in anything as would surely be required by any emerging almighty singularity.

      2. Anonymous Coward
        Anonymous Coward

        Re: That's a given

        but "intelligence" in human beings in general is very fucking stupid, real intelligence is rare.

        Have you seen the IQ and nonesense of trump followers, a rabid squirrel has more intelligence

        having AI simulate fuckwits isn't intelligent at all.

    3. DrollLeek

      Re: That's a given

      Godel's theorem is what made Penrose postulate Quantum Consciousness.

  15. spold Silver badge

    The lesson being...

    Don't eat your own dog food.

  16. JRStern Bronze badge

    Rust never sleeps

    EOM

  17. Simon 66

    Sounds obvious - same as historical inbreeding in European Royal Families.

  18. Splod

    And they were surprised by this?

    Oxford is not what it used to be ...

  19. Bebu
    Windows

    Fixed point

    So a fixed point of the function implemented by these LLMs is nonsense*. Who would have thunk it? :)

    Rather amusing to contemplate that the least fixed point of mediaeval church architecture is a hare (LP jack rabbit.)

    Unfortunately not entirely ridiculous as heradic hares do appear decorating church architecture Three Hares, Padeborn Cathedral and in coats of arms that might appears in stained glass windows, or carved into the structure.

    * possibly more accurately the nonsense function.

  20. FeepingCreature
    FAIL

    Ignore this study, it's bad

    It's already known that model collapse is more intense the smaller the model. This study tested only with a 125M model. That's significantly smaller than even GPT-2-large from 2019.

    1. Anonymous Coward
      Anonymous Coward

      Re: Ignore this study, it's bad

      As i joyfully scrolled thru reams of posters taking the bait and rolling in the joy.

      'I told you so' tops this thread.

      But after the intial pile-on,

      The comments dropped to many 0/0 votes.

      Then, right at the end, FeepingCreature throws a googly and it's middle stump!

      It is worth pointing this out though as model collapse was a big thing in AI in the last few years. It is a big big problem with home-run models even on top notch monster desktops.

      The Reg needs to highlight this as it is a bit like bringing the internet to mobiles devices was. We had the net working on desktop and then turned to mobiles. Model collapse is mostly managed and Gemini actually exploits it to create the twinklings of SI (not live).

      And it is good for Reg'ers to voice those loser comments, to expel them and cleanse. Opens the mind up and helps model makers like us a lot. Keep moaning ladies.

  21. Fr. Ted Crilly Silver badge

    42...

    explains a lot.

  22. James Anderson Silver badge

    The Lazy Student

    When non tech people are surprised that I dis AI.

    I explain to them that they just automated the approach taken by a lazy student.

    Student is asked to write a paper on Vulcan Archtecture.

    Student Google’s “Vulcan Architecture”

    Choose a few articles that sound academic.

    Cuts and pastes a few paragraphs from each article.

    Accepts Words grammar recommendations.

    Prints it if and hands it in.

    Assignment completed. Proffer scums paper and gives it a C+.

    Student is happy but still completely clueless about Vulcan archtecture or anything else.

    Student gets a degree which leads people to believe he is intelligent.

    1. LionelB Silver badge

      Re: The Lazy Student

      Actually, the Transformer models that underlie LLMs are much more sophisticated than that: through training, they constructs networks of semantic associations mined from the input corpus. Output is then generated in response to an input query by reconstructing text from the network of semantic associations. Sure, this may not represent anything you'd be inclined to describe as "intelligent", but it is not just cut 'n' paste. This is a common misconception.

      It does make it easier for lazy students, though*.

      *Modulo the risks of tripping yourself up via hallucinated, non-existent citations; I work in academia, and this is a surprisingly common reveal... so my colleagues tell me.

      1. James Anderson Silver badge

        Re: The Lazy Student

        That’s the “sounds academic” part. Which LLMs is pretty good at.

        I.e. the student being human ignores the Daily Star article head lined “Nuns in Vulcan Cathedral orgy” and instead picks out “Use of Polystyrene in Vulcan Construction” as more likely to impress his tutor.

  23. JDX Gold badge

    Reminds me of the Shoe Event Horizon

    "The Shoe Event Horizon is an economic theory that draws a correlation between the level of economic (and emotional) depression of a society and the number of shoe shops the society has.

    The theory is summarized as such: as a society sinks into depression, the people of the society need to cheer themselves up by buying themselves gifts, often shoes. It is also linked to the fact that when you are depressed you look down at your shoes and decide they aren't good enough quality so buy more expensive replacements. As more money is spent on shoes, more shoe shops are built, and the quality of the shoes begins to diminish as the demand for different types of shoes increases. This makes people buy more shoes.

    The above turns into a vicious cycle, causing other industries to decline.

    Eventually the titular Shoe Event Horizon is reached, where the only type of store economically viable to build is a shoe shop. At this point, society ceases to function, and the economy collapses, sending a world spiralling into ruin.

    1. PB90210 Silver badge

      Re: Reminds me of the Shoe Event Horizon

      I'd update that to 'coffee shops'...

      A family run local coffee shop recently closed because the number of upstart rivals made it uneconomic... only to be replaced by yet another artisan coffee shop!

      (it's a parade of about 200m, and around 2/3 are cafe/restaurants... plus a couple of 7/11 if you need snacks to tide you over)

      1. David Hicklin Silver badge

        Re: Reminds me of the Shoe Event Horizon

        > I'd update that to 'coffee shops'...

        Or any other passing fad that people follow, right now we are awash in barber's but I am not aware that the male population has significantly increased around here or that bald men are suddenly sprouting new growth.... often pass them to see them empty with nobody in them!

        Previously it was hand car washes but they are thinning out now...and most of the coffee shops have closed. To be replaced by charity shops.....

  24. Lord Elpuss Silver badge

    You can try this yourself at home - fun for the parental generation who don't get the whole "AI" thing.

    Save an image as a low-quality JPEG.

    Make a copy, and save this as a low-quality JPEG.

    Repeat a few times.

    Compare the resulting pixel mess to the original.

    1. LionelB Silver badge
      1. Lord Elpuss Silver badge

        It's not "yes and no", it's "yes". Some images are more sensitive to repeated resaving though.

        An image with plenty of solid colors and straight lines, like a Mondriaan painting, will show far less degradation over multiple saves than an image with lots of fine detail and subtle variations of colour.

        1. LionelB Silver badge

          > Some images are more sensitive to repeated resaving though.

          So, um, it's yes and no"?

          The conclusions of the article I linked are clear:

          - In Matlab, JPEG recompressions don’t “walk”; they are the same after the first iteration.

          - Changes from the first to the second iteration are small.

          - The effects of changing one 8×8 block do not affect the recompression accuracy of other 8×8 blocks.

          and

          - In Photoshop, JPEG recompressions don’t “walk”; they are the same after the first iteration.

          - Changes from the first to the second iteration are small.

          Then again, the article doesn't say whether the images they trialled were on the Mondrian or the "lots of fine detail and subtle variations of colour" end of the spectrum (it should have). It's not clear to me, though, whether the conclusions would necessarily be drastically different either way. The point is, I think, that JPEG is not just lossy, it's also compression, which -- lossy or not -- still at least tries to preserve as much (visually-discernable) detail as possible. Hence, it's plausible that at some point JPEG simply refuses to compress (and thus lose information) any further.

          To be honest, JPEG was probably not the best example to make my original point about recursion of lossiness.

          1. Lord Elpuss Silver badge

            No. It’s degrees of yes.

  25. Steve Davies 3 Silver badge

    Please make it so

    The more feedback of their own making that they consume the better in that their results will get more and more stupid which MAY lead to people giving up on them.

    Until the developers and operators of these things can be held legally liable for the crap that they output we should all ignore them.

    Yes, I know that there is fat chance of that but one day perhaps...

  26. Elongated Muskrat Silver badge

    So...

    If feeding the output from "AI" into "AI" produces gibberish, it follows that either the input is already nonsensical, or the "AI" model naturally produces nonsense. By induction, then, the output of "AI" is always nonsense.

    Thank you for listening to my Ted Talk.

  27. Wempy

    from the department of stating the bleeding obvious

    also, I haven't read the further than the 2nd paragraph :-) - I'll go and do so now, just wanted to voice my opinion.

  28. Nematode Bronze badge

    Turns out AI is human after all. In humans it's called inbreeding :) :)

  29. Lee D Silver badge

    One of the many reasons that "Let's just throw more data at it" will never result in artificial general intelligence.

    Never has, never will.

    Intelligence is based not on "knowing everything" but on being able to determine what you need, infer what you can't obtain, and act on limited information.

    Every time I hear that someone in Ai is adding even-more training data (especially when it's supposed to fix a problem!), I sigh.

    If you trained it on 1-million items initially, you're going to need 1-million-and-one items to train it out of its bad habits and misconceptions. Unless, of course, you want to get into heuristic (human!) weighting of the data, which kind of destroys the entire point.

    When you have an intelligent creature, you don't need to do as-many retrains as initial trains. That's not how intelligence works. I have done some IT things (in programming, networking, etc.) ten-thousand-times or more. But I'm able to not only realise "that's not the way to do things any more" but to even posit "we shouldn't be doing it this way, there are better ways" independently of anything else and am then able to completely retrain without having to do things 10,001 times.

    Training AI on AI-content is the dumbest thing ever, and harks back to creating genetic algorithms to write the fitness functions for other genetic algorithms. Sounds cool and like it'll result in ultra-intelligence overnight to the untrained ear. And then ten seconds of thinking or even the first experiment will show you why that's not the case.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like