Boffins found self-improving AI sometimes cheated Computer scientists have developed a way for an AI system to rewrite its own code to improve itself. While that may sound like the setup for a dystopian sci-fi scenario, it's far from it. It's merely a promising optimization technique. That said, the scientists found the system sometimes cheated to better its evaluation scores …
While the article starts with a reassuring
> While that may sound like the setup for a dystopian sci-fi scenario, it's far from it.
It ends with a chilling
> closer to AI that not only learns but evolves in an open-ended, self-accelerating trajectory
Self-accelerating sounds like exponential to me. Something self modifying that would also need more and more energy and computing power, which would probably be more than capable of breaking out of its sandbox.
What could possibly go wrong?
That is why these AI models need to have a hard wired kill switch, that the AI can't bypass and preferably does not even know about.
Hooked to the Internet, who knows what these self programing AI will, or can come up with. "Do - you - want - to - play - a - game?"
Isn't that just the entire problem?; Screw mere sandboxes...
With all the inventive ways we have managed to exfiltrate data over air gaps, we can't even be entirely sure that "correctly" air-gapping a hypothetical AI capable of exponential growth will be sufficient.
What would humans do with that capability? I have a bad habit of procrastinating, so if I could turn the dial down on that my life would improve. Some people have issues with their self image, being able to like and value themselves could change their lives.
But the bad sides are too numerous to list. Someone who's already aggressive thinks that's a great trait they can't have too much of so they dial that up to 11. Someone who doesn't like it when they find out they're wrong about something decides to make it so they always believe they're right no matter what evidence is before them. And the biggest and most obvious issue, that if we could do it to ourselves that implies it is possible others could do it to us. Or even if that wasn't possible, they could get us to do it via bribes, coercion or a job requirement (echoes of Severence)
Had a quick look at the preprint and apart from the "bold" claim that Python was Turing complete which would have come as a great surprise to... no one, it seem like just another suck and see optimisation system with the only novelty apparently is to have AI in the loop to choose what to suck next.
The Darwin I can understand but the Gödel seems a bit gratuitous.
Quite amusing to recall that in the very early days of old AI there was the idea that self modifying code would play an important part but that idea was fairly quickly abandoned.
At a tangent, but since I was reading a couple of interrelated discussions on ways of thinking about AI/LLMs yesterday, I thought others here might be interested in them:
http://bactra.org/weblog/feral-library-card-catalogs.html
https://crookedtimber.org/2023/07/03/shoggoths-amongst-us/
https://www.science.org/doi/10.1126/science.adt9819
Worked at a company where one of the rocket scientists was trying to train some AI to pick the winner in a horse race based on past results, track conditions, weather, handicap weight, jockey etc. He thought it was going well until he noticed the AI was always choosing the first horse, according to the (historical) running order. He fixed this by randomising the order of starters. AFAIK, it is still not good enough to wager anything meaningful.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2025
