Re: Mistakes and 'learning'
I'm not convinced that there is *any* "learning from its mistakes" with these LLMs, at least at any level that reflects on their gross behaviours (like handing over API keys).
"Learning from mistakes" as well as "learning from its successes" is/was part and parcel of Machine Learning - from teaching matchboxes to play Noughts & Crosses all the way up to the Generative Adversarial Networks that will generate "in the style of" images. Start off by giving the system a good shake, some nice random weights, and then play the "Carrot and Stick" game, punishing "mistakes" and rewarding "successes" by tweaking the numbers a bit each time. That mechanism was used to produce the "ability to generate plausible language" behaviours of the LLMs and their predecessors, in very expensive training sessions - expensive because you have to do a *lot* of iterations, a ludicrous number of iterations, scaling up as the number of near as damnit arbitrary numbers (or "parameters" as the programming illiterate like to say) increases. How large are these models now? We are well past Carl Sagan territory now, billions and billions of nadans.
To go with the compute burnt for all those iterations you need to have training data. Lots and lots of examples, all with their meta-data to indicate whether output that matches that datum is to given the carrot or the stick: this photo has a cat in it, located here, this photo does *not* have a cat (although it does have a dog). And you'll spot from that example that you need to have an awful lot of training data, including potential false positives (pictures of everything that naturally resembles a cat in some way, but isn't - that has four legs but is an ottoman sofa) as well as lots of false negatives (all the environments that could contain a cat, but currently don't). For the basics of English, this isn't *too* hard to arrange: they just nicked all those books and could use random inputs for the comparative samples for when they couldn't be bothered with any better meta-data.
So all those iterations gets us past "i before except after c" and the myriad ways that fails, even unto "object verb subject" with plausible results.
BUT whenever people talk about LLMs, and in particular the current obsession of "agentic" systems, "learning from their mistakes" they mean some huge chunk of output, which at the micro scale is all ok (the spelling is ok, it even has the accent I can never figure out how to put into 'naive') but at the macro scale is horrendous (it just sold the house and bought shares in Staples, I expected paperclips to hold this HO scale terrace together).
Take a moment to understand what that actually means, to correct macro-level behaviour by "learning from its mistakes". Just in terms of simple training requirements: there needs to be enough raw data, examples of "good" versus "bad" activity, all neatly tagged with meta-data. Enough examples of these agentic systems going apeshit in every way possible, to compare against the sequence of actions that worked.
Where is all this training material going to come from? Remember, just for the basic language stuff, the LLM creators raped - oops, scraped - all the authors' works they could, legitimately or not.
We *could* discuss pedagogy, how we teach humans to make careful plans and then scrutinise their execution, learning as we go (and then we could look at all the humans whose Ikea shelves don't fit on *either* side of the fireplace, at least not without "unexpectedly" needing to cut the ends off the mantlepiece, to see how well that goes), which methods *could* be used. If we believe that the LLM creators - or, more properly now, the creators of things like OpenClaw, which (ab)use LLMs for "agentic" stuff - are carefully setting up Planners (go back and read the literature from the 1970s onwards) to create, well, plans, which are run against accurate models of the real-world to provide a safe sandbox for the failures to occur in (and there will be, must be, so many more failures for each success else there is no learning); go read about SHRDLU and its block world for a simple example, in comparison to what these current systems are being given access to.
But, let's be honest, they are all just trying to let Profesor Fry and every other User go wild and then scrape the results. Which means they are absolutely desperate for funding from documentary makers, as those are the only ones who are going to record and publish the meta-data as well!
But if they get potential training data, will they even bother to run it, really? Consider how much each iteration through those samples will cost: each single datum to be rewarded/punished is now a long sequence of actions ("create a new project on Github" being one small action) involving many LLM tokens apiece. The further you get from the ABCs the more the search space branches *and* the greater the cost for just one run along a single branch.
Bottom line: having these beasts learn from their mistakes is so very expensive and so very unlikely.
